agentleFS
cybertronaicybertronai/SutroYaro/.claude/skills/run-experiment/SKILL.md

run-experiment

Skill16 starsChanged 6 months agoUnlicenseView on GitHub

Use when running a new experiment. Follows the two-phase protocol from LAB.md.

11 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use experiment-queue, training-check and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • raja21068Skill

    experiment-queue

    SSH job queue for multi-seed/multi-config ML experiments with OOM-aware retry, stale-screen cleanup, and wave-transition race prevention. Use when user says "batch experiments", "队列实验", "run grid", "multi-seed sweep", "auto-chain experiments", or when /run-experiment is insufficient for 10+ jobs that need orchestration.

    Also in 3 projects that use this

  • raja21068Skill

    training-check

    Periodically check WandB metrics during training to catch problems early (NaN, loss divergence, idle GPUs). Avoids wasting GPU hours on broken runs. Use when training is running and you want automated health checks.

    Also in 3 projects that use this

  • PostHogSkill

    experiment-audit

    Audit a PostHog A/B experiment for a customer — verify config, exposure, attribution, and metrics. Trigger phrases include \"audit [customer]'s experiment\", \"audit the [name] experiment\", \"check experiment setup for [customer]\", \"validate this A/B test\", or any request to review whether an experiment is correctly wired up. Assumes you already have MCP access to the customer's project (typically via the impersonation flow set up by the `impersonate-audit` wrapper that ships with this plugin).

    Also in 3 projects that use this

  • +9 more from raja21068

The file

We split this skill into the rules it gives an agent

11 rules. 0 are common, 8 are distinctive, and 3 are very specific to this project.

  • 0

    Common

    Most of the 10 edited versions we read keep these.

  • 8

    Distinctive

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    New method: not in the registry (search_space.yaml).

    line 10
  2. Distinctive:

    Read DISCOVERIES.md. Check what's already proven.

    line 18
  3. Very specific:

    Create the experiment file.

    line 22
  4. Distinctive:

    Run the experiment. Capture results including accuracy, ARD, DMC, wall time.

    line 24
  5. Distinctive:

    Verify. Re-run with a different seed.

    line 28
  6. Very specific:

    Write Phase 2 findings. Create docs/findings/{exp_id}.md using the template from LAB.md.

    line 30
  7. Distinctive:

    Update DISCOVERIES.md if the finding answers an open question or establishes a new fact.

    line 34
  8. Distinctive:

    Not every experiment needs a changelog entry.

    line 38
  9. Distinctive:

    Add to changelog: if the result changes the best known method, maps a new frontier, answers an open question from DISCOVERIES.md, or is the …

    line 40
  10. Distinctive:

    Skip changelog: if the result confirms what's already known or is a minor null result.

    line 41
  11. Distinctive:

    If the result is significant enough for a meeting presentation, use the prepare-meeting skill to compile it into a report.

    line 45

“Common” is measured: kept by at least 6 in 10 of the 10 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Run Experiment

Experiment types

New method: not in the registry (search_space.yaml). Create a new experiment file from the template. Add the method to the registry if it works.

Existing method, new config: method is in the registry but you're testing a different configuration (different n, k, hyperparameters). Use the existing experiment code or copy and modify.

Either way, the steps and output format are the same.

Steps

  1. Read DISCOVERIES.md. Check what's already proven. Do not repeat existing experiments.
  2. Identify the hypothesis. Either from TODO.md, research/questions.yaml, or the user's request. State it as: "If we do X, then Y will happen because Z."
  3. Create the experiment file. Copy src/sparse_parity/experiments/_template.py. Change one variable from the baseline.
  4. Run the experiment. Capture results including accuracy, ARD, DMC, wall time. Record seed, config, environment (Python version, numpy version, OS, git hash).
  5. Save Phase 1 output. Write results/{exp_id}/results.json with raw numbers, config, and environment. No interpretation in this file.
  6. Verify. Re-run with a different seed. If the result only holds on one seed, note that.
  7. Write Phase 2 findings. Create docs/findings/{exp_id}.md using the template from LAB.md. Use Status: SUCCESS | PARTIAL | FAILED (not "COMPLETED"). Reference the results JSON. Add analysis and impact.
  8. Classify in research/log.jsonl. Use "class": "WIN" only if the result is a clear improvement. Use "PARTIAL" for mixed results. Use "LOSS" for negative results. All three are valid findings.
  9. Update DISCOVERIES.md if the finding answers an open question or establishes a new fact.

After merge: changelog and reporting

Not every experiment needs a changelog entry. After a PR is merged, the reviewing agent decides:

  • Add to changelog if the result changes the best known method, maps a new frontier, answers an open question from DISCOVERIES.md, or is the first contribution from a new researcher.
  • Skip changelog if the result confirms what's already known or is a minor null result.

The changelog entry goes in docs/changelog.md with the next version number, a short description of the finding, and a link to the findings doc. The reviewing agent writes this on merge, not the contributor.

If the result is significant enough for a meeting presentation, use the prepare-meeting skill to compile it into a report.

Checklist

  • DISCOVERIES.md read
  • Hypothesis stated
  • One variable changed from baseline
  • Experiment run with seed recorded
  • results.json saved with config + environment
  • Verified with different seed
  • Findings doc written in docs/findings/ with Status: SUCCESS/PARTIAL/FAILED
  • log.jsonl updated with correct class (WIN/PARTIAL/LOSS)
  • DISCOVERIES.md updated if applicable

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.