notebook-audit
Skill2 starsChanged 12 days agoMITView on GitHub
Notebook Audit. Score and fix an existing Jupyter notebook, Colab notebook, Python script, ML repository or model service - leakage, statistics, imbalance, reproducibility, explainability, communication and production readiness - with a ranked findings list, RIGOR score and minimal fixes. Use for review my notebook, audit this analysis, check this model, code review for data science, is this production ready, or before sharing or deploying someone else's work.
In this repository16 other files it gives its agents.
- ai-prediction-explainerSkill
- data-mining-machine-learningSkill
- data-miningSkill
- data-preparationSkill
- data-profilingSkill
- data-storytellingSkill
- feature-engineeringSkill
- hyperparameter-tuningSkill
- ml-quality-guardSkill
- ml-web-applicationSkill
- model-deploymentSkill
- model-evaluationSkill
- model-explainabilitySkill
- model-trainingSkill
- problem-framingSkill
- statistical-analysisSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 29
Do not rewrite working code to match taste.
- Distinctive: line 30
If the notebook cannot run (missing data or library), say which and score what can be scored.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Notebook Audit
Procedure
- Inventory. List notebooks, scripts, model files, API code, prompts and data files. Note the environment (Jupyter, JupyterLab, Colab, VS Code) and Python version.
- Scan.
python "${CLAUDE_PLUGIN_ROOT}/scripts/ml_scan.py" <paths> --format text. Add--imbalancedwhen the target is rare. - Run.
python "${CLAUDE_PLUGIN_ROOT}/scripts/notebook_run.py" <notebook>from a clean kernel. A notebook that does not run scores 0 on Reproducibility. - Read for what the scanner cannot see (
references/audit-rubric.md): target definition and timing, split versus data structure, metric versus cost, threshold source, test-set reuse, causal language, chart honesty, limitations. - Score the seven RIGOR gates with evidence (cell or line).
- Report (at most 30 lines): status line, RIGOR table, findings ranked P0 (wrong results), P1 (misleading or unsafe), P2 (quality), each with location, rule ID, why it matters and the exact fix.
- Fix on request: P0 first with minimal diffs, rerun the notebook and the scanner, report before and after. At most 3 cycles.
Priorities
| Priority | Examples |
| P0 | Leakage (L01 to L10), test set used for tuning or threshold, fabricated metrics (C01), unpickling untrusted input (P01) |
| P1 | Accuracy alone on imbalance (M02), ROC-AUC alone (M03), no baseline, no intervals, causal claims, secrets in code, LLM without grounding |
| P2 | Missing feature dictionary, unlabeled charts, oversized outputs, unpinned installs |
Rules
- Evidence for every finding: file, cell and line, or the scanner rule.
- Do not rewrite working code to match taste; change only what a finding requires.
- If the notebook cannot run (missing data or library), say which and score what can be scored.
- For a second opinion without builder bias, run the
ml-criticagent.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.