agentleFS
jiahao-shao1jiahao-shao1/sjh-skills/skills/experiment-registry/SKILL.md

experiment-registry

Skill43 starsChanged 7 months agoMITView on GitHub

Manage ML experiment lifecycle via structured YAML registry — register, record results, compare runs, track status. Triggers: 'register experiment', 'compare experiments', 'experiment results', 'exp init', 'exp list', '注册实验', '对比实验', '查实验', '哪个实验最好', '记录结果'. Not for training monitoring, checkpoints, or launching jobs.

  • Installs packages

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 5 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If exp is not found, install it first.

    line 18
  2. Distinctive:

    When the user mentions experiments, follow this flow:

    line 28
  3. Distinctive:

    Be proactive with context: After any operation, offer useful next steps:

    line 39
  4. Distinctive:

    After registering → "Ready to add benchmarks when results come in"

    line 40
  5. Distinctive:

    After adding a benchmark → suggest comparing with related experiments if they exist

    line 41
  6. Distinctive:

    After comparing → highlight the best performer and surface any findings

    line 42
  7. Very specific:

    All list/show/compare commands support --json for machine-readable output.

    line 69

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Experiment Registry

Structured YAML experiment registry for ML research. YAML was chosen over databases or JSON because experiment files should be human-readable, git-diffable, and hand-editable — researchers often need to inspect or tweak entries directly.

Prerequisites

The exp CLI must be installed:

pip install exp-registry

If exp is not found, install it first. If the project has no exp.config.yaml, run exp init before any other command.

How to Think About Experiment Registry

This tool manages the metadata layer of experiments — what was run, with what config, and what results came out. It does NOT manage training code, checkpoints, or logs.

The mental model: each experiment gets a YAML file that serves as its "identity card." Over time, benchmark results accumulate in that file as the experiment progresses through training steps. When it's time to report or decide next steps, you compare across experiments.

Decision Flow

When the user mentions experiments, follow this flow:

  1. Check environment first. Does exp.config.yaml exist? If not, ask if they want to initialize (exp init). Don't silently init — the user should confirm the project root.
  2. Understand intent. Map the user's request to the right action:
    • "New experiment" / "register" → exp register (but first exp list to check for ID conflicts)
    • "What's running" / "experiment status" → exp list (suggest filters if there are many)
    • "Record results" / "add benchmark" → exp add-benchmark (ask for dataset and step if not provided)
    • "Compare" / "which is better" → exp compare (auto-discover shared datasets from exp show)
    • "What happened with exp07" → exp show first, then summarize findings
  3. Be proactive with context. After any operation, offer useful next steps:
    • After registering → "Ready to add benchmarks when results come in"
    • After adding a benchmark → suggest comparing with related experiments if they exist
    • After comparing → highlight the best performer and surface any findings

Smart Comparison

When the user asks "compare experiments" or "which is better," don't just dump the table. First run exp show on each experiment to discover which datasets and eval_modes they share, then run exp compare on those shared dimensions. If experiments have no common benchmarks, tell the user — don't produce an empty comparison silently.

Series Grouping

Experiment IDs like exp07a, exp07b, exp07c automatically group into series exp07. This lets you filter by series to see all variants of one experimental idea. When a user discusses "the exp07 experiments," use exp list --series exp07 to get the full picture.

Command Reference

TaskCommand
Initializeexp init
List allexp list
Filter by statusexp list --status completed
Filter by typeexp list --type rl
Filter by seriesexp list --series exp07
Show detailsexp show <id>
JSON outputexp show <id> --json
Register newexp register <id> --type rl --model Qwen3-VL-8B (auto-records git HEAD; --commit <sha> to override)
Add benchmarkexp add-benchmark <id> --dataset mmlu --eval-mode cot --samples 100 --step 50 --extra acc=0.72
Compareexp compare <id1> <id2> --dataset mmlu
Update statusexp update <id> --status completed
Add findingexp update <id> --finding "key insight"

All list/show/compare commands support --json for machine-readable output.

Typical Workflows

Experiment Lifecycle

register → (train) → add-benchmark at step N → add-benchmark at step M → update status + finding

Example:

  1. exp register exp01 --type rl --model Qwen3-VL-8B --reward reward_tool_strict
  2. (user trains the model)
  3. exp add-benchmark exp01 --dataset zebra-cot --eval-mode agent --samples 50 --step 50 --extra text_only=0.42 with_tools=0.52
  4. exp add-benchmark exp01 --dataset zebra-cot --eval-mode agent --samples 50 --step 90 --extra text_only=0.44 with_tools=0.55
  5. exp update exp01 --status completed --finding "tool use improves reasoning"

Benchmarks are organized by step number within each dataset — this tracks how performance evolves during training, which is critical for deciding when to stop or which checkpoint to use.

Compare for Meeting/Report

exp compare exp07a exp07b exp07c --dataset zebra-cot

Outputs a Markdown table — paste directly into docs or slides.

Project Configuration

exp.config.yaml at project root:

registry_dir: experiments/           # where YAML files live
paths_template:
  local: outputs/{id}/               # {id} is replaced with experiment ID
defaults:
  type: rl                           # default for `exp register`
types:
  rl:
    fields: [model, config, script, reward]
  sft:
    fields: [model, config, script, base_model]

Config is discovered by walking up from CWD. No config = sensible defaults (registry_dir: experiments/).

YAML Schema

Each experiment is one file in <registry_dir>/<exp_id>.yaml:

Required fields: id, name, type, series, date, status

Auto-generated: series (inferred from ID prefix), paths (from template), date (today)

Structured benchmarks:

benchmarks:
  - dataset: string
    eval_mode: string
    samples: int
    steps:
      <step_number>: { <metric>: <value>, ... }

Error Recovery

ErrorCauseWhat to Do
"experiment already exists"Duplicate IDexp show <id> to check, use a different ID
"experiment not found"Wrong IDexp list to see available IDs
"No benchmark data found"Wrong dataset in compareexp show <id> to check available datasets
"Missing required fields"Corrupted YAMLInspect and fix the YAML file directly — it's designed to be human-editable

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.