auto-experiment
Skill8 starsChanged 8 months agoMITView on GitHub
Run autonomous YOLO training experiments — reads training-plan.md, assesses bottlenecks, acts strategically, and delegates HP optimization to model.tune().
2 other projects use a version of this.
Adopted by
Most recently taken up by
- zjunlp/Mechanisttheir own version, first seen Jul 11, 2026
- zxl696/smallmodel-auto-experiment-skilltheir own version, first seen Mar 21, 2026
Other versions2 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 2
- zjunlp/Mechanist★ 75same name
- zxl696/smallmodel-auto-experiment-skill★ 5same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository15 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- active-learningSkill
- analyze-resultsSkill
- annotateSkill
- benchmarkSkill
- compare-modelsSkill
- cvat-deploySkill
- cvat-pullSkill
- cvat-pushSkill
- explain-resultsSkill
- monitor-trainingSkill
- review-annotationsSkill
- review-datasetSkill
- setup-projectSkill
- training-loopSkill
The file
We split this skill into the rules it gives an agent
12 rules. About 0 are common, 6 are distinctive, and 6 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 16
If not found, check for program.md (deprecated name) and warn user to rename it
- Distinctive: line 23
Read in order before every session:
- Very specific: line 32
If no baseline exists: yolo-experiment baseline --budget 5 --patience 3 After baseline, update training-plan.md Current Performance section.
- Distinctive: line 33
After baseline, update training-plan.md Current Performance section.
- Very specific: line 37
Write a session marker in summary.md:
- Distinctive: line 41
If a session marker exists with today's date and no end marker, this is a resumed session — count existing entries toward budget.
- Distinctive: line 49
Read dataset profile, per-class AP, training dynamics from last run.
- Distinctive: line 56
Write a hypothesis: "I expect X because Y, so I'll do Z"
- Very specific: line 90
Read experiment report from experiments/exp_NNN_*/report.md
- Very specific: line 97
Write session end marker: ## Session End — N experiments run
- Very specific: line 98
Write experiments/session_YYYY-MM-DD.md with before/after + key learnings
- Distinctive: line 99
Update training-plan.md Model Lineage with current best path, metrics, source experiment
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Auto-Experiment — Strategic Decision Loop
Autonomous experimentation via hypothesis-driven reasoning.
The agent makes strategic decisions (architecture, data, augmentation strategy).
HP optimization is delegated to model.tune() — don't compete with it.
Pre-Flight
yolo-project.yamlexists — run/setupif nottraining-plan.mdexists with goals + boundaries — run/setupif not- If not found, check for
program.md(deprecated name) and warn user to rename it
- If not found, check for
- Dataset is valid —
yolo-validate - GPU available —
python -c "import torch; print(torch.cuda.is_available())" - Read
experiments/dataset_profile.yaml— run/review-datasetif missing
Context Loading
Read in order before every session:
yolo-project.yaml— classes, dataset path, model defaultstraining-plan.md— goals, hard constraints, soft preferences, allowed actions, domain knowledgeexperiments/summary.md— all prior journal entries (if exists)experiments/dataset_profile.yaml— dataset characteristics for architecture reasoningexperiments/analysis.md— latest analysis (if exists)
Baseline
If no baseline exists: yolo-experiment baseline --budget 5 --patience 3
After baseline, update training-plan.md Current Performance section.
Session Start
Write a session marker in summary.md:
## Session YYYY-MM-DD HH:MM — Budget: N experimentsIf a session marker exists with today's date and no end marker, this is a resumed session — count existing entries toward budget.
Reasoning Loop
For each decision, follow all 3 steps in order:
1. ASSESS — What's the bottleneck?
Read dataset profile, per-class AP, training dynamics from last run.
Classify the bottleneck:
- Data quality → active learning (
/review-dataset,/cvat-push,/autolabel) - Architecture mismatch → swap config (
yolo-experiment run --override "model=configs/architectures/...") - HP not optimized →
yolo-experiment tune --space <preset>
Write a hypothesis: "I expect X because Y, so I'll do Z"
Cold-start (first post-baseline): Focus on dataset profile characteristics, per-class AP spread, class imbalance. Training dynamics analysis starts from experiment 2 onward.
2. ACT — Do the right thing
Architecture change:
yolo-experiment run --override "model=configs/architectures/yolo11-p2.yaml" --budget <epochs>Requires dataset profile justification — cite ≥2 numbers. See resources/architecture-guide.md.
HP optimization (delegate to model.tune):
# Use presets when diagnosis maps cleanly:
yolo-experiment tune --space lr --iterations 20 --epochs 10
yolo-experiment tune --space augmentation --iterations 20 --epochs 10
yolo-experiment tune --space loss --iterations 15 --epochs 10
yolo-experiment tune --space optimizer --iterations 20 --epochs 10
# Use custom when diagnosis is specific:
yolo-experiment tune --space "lr0=0.001:0.01 momentum=0.85:0.98" --iterations 20 --epochs 10Strategic experiment (resolution, freeze depth, augmentation strategy):
yolo-experiment run --override "<param>=<value>" --budget <epochs>Data action (when data quality is the bottleneck):
/review-dataset, /cvat-push, /autolabel
3. LOG — Record what happened
- Read experiment report from
experiments/exp_NNN_*/report.md - Append simplified journal entry to
summary.mdmatching schema inresources/journal-schema.yaml - Every field marked REQUIRED must be present.
- Decide: continue this direction, try different lever, or stop
Session End
- Write session end marker:
## Session End — N experiments run - Write
experiments/session_YYYY-MM-DD.mdwith before/after + key learnings - Update training-plan.md Model Lineage with current best path, metrics, source experiment
Guardrails
- Checkpoint backup — hook handles automatically before every run/tune
- Immutable data — NEVER modify original dataset files
- Architecture change gate — 3+ experiments on current architecture with <0.5% improvement before switching. Exception: >50% small objects with no P2 head (see
resources/architecture-guide.md) - Architecture justification — must cite ≥2 dataset profile numbers in journal
- Log everything — write journal entry even if experiment fails
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

