experiment-registry
Skill43 starsChanged 7 months agoMITView on GitHub
Manage ML experiment lifecycle via structured YAML registry — register, record results, compare runs, track status. Triggers: 'register experiment', 'compare experiments', 'experiment results', 'exp init', 'exp list', '注册实验', '对比实验', '查实验', '哪个实验最好', '记录结果'. Not for training monitoring, checkpoints, or launching jobs.
- Installs packages
In this repository17 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- bibtex-fetchSkill
- context-auditSkill
- daily-summarySkill
- handoffSkill
- init-projectSkill
- notion-lifeosSkill
- paper-analyzerSkill
- paper-self-reviewSkill
- project-reviewSkill
- remote-cluster-agentSkill
- research-leadSkill
- session-teamSkill
- sync-docsSkill
- team-digestSkill
- web-fetcherSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 5 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 18
If exp is not found, install it first.
- Distinctive: line 28
When the user mentions experiments, follow this flow:
- Distinctive: line 39
Be proactive with context: After any operation, offer useful next steps:
- Distinctive: line 40
After registering → "Ready to add benchmarks when results come in"
- Distinctive: line 41
After adding a benchmark → suggest comparing with related experiments if they exist
- Distinctive: line 42
After comparing → highlight the best performer and surface any findings
- Very specific: line 69
All list/show/compare commands support --json for machine-readable output.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Experiment Registry
Structured YAML experiment registry for ML research. YAML was chosen over databases or JSON because experiment files should be human-readable, git-diffable, and hand-editable — researchers often need to inspect or tweak entries directly.
Prerequisites
The exp CLI must be installed:
pip install exp-registryIf exp is not found, install it first. If the project has no exp.config.yaml, run exp init before any other command.
How to Think About Experiment Registry
This tool manages the metadata layer of experiments — what was run, with what config, and what results came out. It does NOT manage training code, checkpoints, or logs.
The mental model: each experiment gets a YAML file that serves as its "identity card." Over time, benchmark results accumulate in that file as the experiment progresses through training steps. When it's time to report or decide next steps, you compare across experiments.
Decision Flow
When the user mentions experiments, follow this flow:
- Check environment first. Does
exp.config.yamlexist? If not, ask if they want to initialize (exp init). Don't silently init — the user should confirm the project root. - Understand intent. Map the user's request to the right action:
- "New experiment" / "register" →
exp register(but firstexp listto check for ID conflicts) - "What's running" / "experiment status" →
exp list(suggest filters if there are many) - "Record results" / "add benchmark" →
exp add-benchmark(ask for dataset and step if not provided) - "Compare" / "which is better" →
exp compare(auto-discover shared datasets fromexp show) - "What happened with exp07" →
exp showfirst, then summarize findings
- "New experiment" / "register" →
- Be proactive with context. After any operation, offer useful next steps:
- After registering → "Ready to add benchmarks when results come in"
- After adding a benchmark → suggest comparing with related experiments if they exist
- After comparing → highlight the best performer and surface any findings
Smart Comparison
When the user asks "compare experiments" or "which is better," don't just dump the table. First run exp show on each experiment to discover which datasets and eval_modes they share, then run exp compare on those shared dimensions. If experiments have no common benchmarks, tell the user — don't produce an empty comparison silently.
Series Grouping
Experiment IDs like exp07a, exp07b, exp07c automatically group into series exp07. This lets you filter by series to see all variants of one experimental idea. When a user discusses "the exp07 experiments," use exp list --series exp07 to get the full picture.
Command Reference
| Task | Command |
| Initialize | exp init |
| List all | exp list |
| Filter by status | exp list --status completed |
| Filter by type | exp list --type rl |
| Filter by series | exp list --series exp07 |
| Show details | exp show <id> |
| JSON output | exp show <id> --json |
| Register new | exp register <id> --type rl --model Qwen3-VL-8B (auto-records git HEAD; --commit <sha> to override) |
| Add benchmark | exp add-benchmark <id> --dataset mmlu --eval-mode cot --samples 100 --step 50 --extra acc=0.72 |
| Compare | exp compare <id1> <id2> --dataset mmlu |
| Update status | exp update <id> --status completed |
| Add finding | exp update <id> --finding "key insight" |
All list/show/compare commands support --json for machine-readable output.
Typical Workflows
Experiment Lifecycle
register → (train) → add-benchmark at step N → add-benchmark at step M → update status + findingExample:
exp register exp01 --type rl --model Qwen3-VL-8B --reward reward_tool_strict- (user trains the model)
exp add-benchmark exp01 --dataset zebra-cot --eval-mode agent --samples 50 --step 50 --extra text_only=0.42 with_tools=0.52exp add-benchmark exp01 --dataset zebra-cot --eval-mode agent --samples 50 --step 90 --extra text_only=0.44 with_tools=0.55exp update exp01 --status completed --finding "tool use improves reasoning"
Benchmarks are organized by step number within each dataset — this tracks how performance evolves during training, which is critical for deciding when to stop or which checkpoint to use.
Compare for Meeting/Report
exp compare exp07a exp07b exp07c --dataset zebra-cotOutputs a Markdown table — paste directly into docs or slides.
Project Configuration
exp.config.yaml at project root:
registry_dir: experiments/ # where YAML files live
paths_template:
local: outputs/{id}/ # {id} is replaced with experiment ID
defaults:
type: rl # default for `exp register`
types:
rl:
fields: [model, config, script, reward]
sft:
fields: [model, config, script, base_model]Config is discovered by walking up from CWD. No config = sensible defaults (registry_dir: experiments/).
YAML Schema
Each experiment is one file in <registry_dir>/<exp_id>.yaml:
Required fields: id, name, type, series, date, status
Auto-generated: series (inferred from ID prefix), paths (from template), date (today)
Structured benchmarks:
benchmarks:
- dataset: string
eval_mode: string
samples: int
steps:
<step_number>: { <metric>: <value>, ... }Error Recovery
| Error | Cause | What to Do |
| "experiment already exists" | Duplicate ID | exp show <id> to check, use a different ID |
| "experiment not found" | Wrong ID | exp list to see available IDs |
| "No benchmark data found" | Wrong dataset in compare | exp show <id> to check available datasets |
| "Missing required fields" | Corrupted YAML | Inspect and fix the YAML file directly — it's designed to be human-editable |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

