ml-quality-guard
Skill2 starsChanged 12 days agoMITView on GitHub
ML Quality Guard. Rigor harness for data science and machine learning work: scan notebooks, scripts, APIs and prompts for leakage, statistical errors, imbalance mistakes, reproducibility gaps, production and LLM explainer slop, then fix and score. Use when asked to check, verify, lint, review, validate, score or "is this right" for a notebook, model, pipeline, analysis, FastAPI service or explainer, and before declaring any ML deliverable done.
- Reads credentials
In this repository16 other files it gives its agents.
- ai-prediction-explainerSkill
- data-mining-machine-learningSkill
- data-miningSkill
- data-preparationSkill
- data-profilingSkill
- data-storytellingSkill
- feature-engineeringSkill
- hyperparameter-tuningSkill
- ml-web-applicationSkill
- model-deploymentSkill
- model-evaluationSkill
- model-explainabilitySkill
- model-trainingSkill
- notebook-auditSkill
- problem-framingSkill
- statistical-analysisSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 5 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 23
Statistical and Leakage Self-Correction: Run the scanner. Then check by hand what it cannot see:
- Distinctive: line 25
All fitting inside pipelines on training folds.
- Distinctive: line 26
Every test states question, estimand, assumptions.
- Distinctive: line 28
Every widget callback runs once without error (call the update function directly in a cell).
- Distinctive: line 29
Refinement: Remove redundant code and repeated output, recompute the RIGOR score, print the status line.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
ML Quality Guard
Nothing reaches the user until it passes this loop.
Run the scanner
python "${CLAUDE_PLUGIN_ROOT}/scripts/ml_scan.py" <paths> [--format text|json] [--imbalanced] [--fail-on warning]Scans .py (including # %% percent files), .ipynb, .R, .sql, .md prompts, Dockerfile, .env*, .ts, .tsx, .js. Output per finding: rule ID, severity, location (line N or cell C line N), message. Ends with Rigor score 0-100. Exit code 1 on any error. The catalog lives in references/slop-catalog.md; read a rule there only when a finding needs explaining.
Escape hatch with a reason: # ml-ignore M04 threshold fixed by regulator at 0.5. No reason, no ignore.
Think, Make, Check (max 3 fix cycles)
- Compile and Run. Execute from a clean kernel:
python "${CLAUDE_PLUGIN_ROOT}/scripts/notebook_run.py" <notebook>(nbconvert or a plain script run). Confirm shapes and row counts at every join and split. Confirm the notebook finishes. - Statistical and Leakage Self-Correction. Run the scanner. Then check by hand what it cannot see:
- Split logic matches the data (stratified, group-aware, forward-chaining). Test set touched once.
- All fitting inside pipelines on training folds; resampling in-fold only.
- Every test states question, estimand, assumptions; effect size and CI next to p-values.
- Baseline comparison, intervals on headline metrics, threshold chosen on validation, calibration checked after resampling.
- Every widget callback runs once without error (call the update function directly in a cell).
- Refinement. Remove redundant code and repeated output, recompute the RIGOR score, print the status line.
Fix errors at the cause, not by suppressing. After 3 cycles, report what remains.
RIGOR gates (0 or 1 each)
| Gate | Passes when |
| Reproducibility | Seed constant, pinned versions, relative paths, Restart and Run All succeeds |
| Integrity | No leakage findings, fitting in pipelines, labels verified, resampling in-fold |
| Generalization | Honest validation, baseline beaten, intervals, slices, threshold from validation |
| Explainability | Global and local explanations on held-out data, feature dictionary, caveats |
| Observability | Logging, data checks, monitoring plan, model card |
| Communication | Decision-first narrative, labeled charts, limitations, what-if with static fallback |
| Deployability | Reload parity, schema-validated API, health checks and tests pass, secrets out of code and browser, explainer grounded |
Score each gate from evidence, not intent. Gates that do not apply to the deliverable (for example Deployability for a one-off A/B readout) are reported as n/a and excluded from the denominator.
Status line
Rigor: <errors> errors, <warnings> warnings, score <0-100>, RIGOR <passed>/<applicable>, run: passed|failed|not run
Checks beyond the scanner
- Notebook state: execution counts in order, no outputs over 100 KB, no full data frames printed.
- Leakage: list each feature with its availability time; anything after prediction time is removed.
- Imbalance: positive count stated next to every rare-class metric; PR-AUC shown; accuracy never alone.
- Production:
bin/data-mining-ml verify-model models/<version>passes; API tests pass. - Explainer: grounding test rejects a number absent from the context.
Details of each gate: references/rigor-gates.md.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.