run-experiment
Skill16 starsChanged 6 months agoUnlicenseView on GitHub
Use when running a new experiment. Follows the two-phase protocol from LAB.md.
11 other projects use a version of this.
Adopted by
Most recently taken up by
- AfonsoZhang/auto-research-systemtheir own version, first seen Nov 15, 2026
- bcmyguest/personal-skillstheir own version, first seen Nov 8, 2026
- jisuzzzz/haechitheir own version, first seen Nov 8, 2026
- curvedinf/dev-skillstheir own version, first seen Oct 20, 2026
- isaac-sim/IsaacLab-Arenatheir own version, first seen Oct 8, 2026
- talhaMah56/claude_skillstheir own version, first seen Sep 12, 2026
Used with
Projects with it also use experiment-queue, training-check and 10 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
raja21068Skillexperiment-queue
SSH job queue for multi-seed/multi-config ML experiments with OOM-aware retry, stale-screen cleanup, and wave-transition race prevention. Use when user says "batch experiments", "队列实验", "run grid", "multi-seed sweep", "auto-chain experiments", or when /run-experiment is insufficient for 10+ jobs that need orchestration.
Also in 3 projects that use this
raja21068Skilltraining-check
Periodically check WandB metrics during training to catch problems early (NaN, loss divergence, idle GPUs). Avoids wasting GPU hours on broken runs. Use when training is running and you want automated health checks.
Also in 3 projects that use this
PostHogSkillexperiment-audit
Audit a PostHog A/B experiment for a customer — verify config, exposure, attribution, and metrics. Trigger phrases include \"audit [customer]'s experiment\", \"audit the [name] experiment\", \"check experiment setup for [customer]\", \"validate this A/B test\", or any request to review whether an experiment is correctly wired up. Assumes you already have MCP access to the customer's project (typically via the impersonation flow set up by the `impersonate-audit` wrapper that ships with this plugin).
Also in 3 projects that use this
- +9 more from raja21068
Other versions11 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 11
- wanshuiyin/Auto-claude-code-research-in-sleep★ 19ksame name
- isaac-sim/IsaacLab-Arena★ 657same name
- zjunlp/Mechanist★ 75same name
- Chanw-research/claude-code-paper-writing★ 42same name
- jisuzzzz/haechi★ 17same name
- Yusong-Enceladus/claude-skills★ 8same name
- curvedinf/dev-skills★ 7same name
- raja21068/AutoResearch★ 2same name
- AfonsoZhang/auto-research-system★ 1same name
- bcmyguest/personal-skills★ 0same name
1 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository9 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .codex/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- anti-slop-guideSkill
- info-defragSkill
- prepare-meetingSkill
- sutro-contextSkill
- sutro-syncSkill
- weekly-catchupSkill
The file
We split this skill into the rules it gives an agent
11 rules. 0 are common, 8 are distinctive, and 3 are very specific to this project.
0
Common
Most of the 10 edited versions we read keep these.
8
Distinctive
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 10
New method: not in the registry (search_space.yaml).
- Distinctive: line 18
Read DISCOVERIES.md. Check what's already proven.
- Very specific: line 22
Create the experiment file.
- Distinctive: line 24
Run the experiment. Capture results including accuracy, ARD, DMC, wall time.
- Distinctive: line 28
Verify. Re-run with a different seed.
- Very specific: line 30
Write Phase 2 findings. Create docs/findings/{exp_id}.md using the template from LAB.md.
- Distinctive: line 34
Update DISCOVERIES.md if the finding answers an open question or establishes a new fact.
- Distinctive: line 38
Not every experiment needs a changelog entry.
- Distinctive: line 40
Add to changelog: if the result changes the best known method, maps a new frontier, answers an open question from DISCOVERIES.md, or is the …
- Distinctive: line 41
Skip changelog: if the result confirms what's already known or is a minor null result.
- Distinctive: line 45
If the result is significant enough for a meeting presentation, use the prepare-meeting skill to compile it into a report.
“Common” is measured: kept by at least 6 in 10 of the 10 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Run Experiment
Experiment types
New method: not in the registry (search_space.yaml). Create a new experiment file from the template. Add the method to the registry if it works.
Existing method, new config: method is in the registry but you're testing a different configuration (different n, k, hyperparameters). Use the existing experiment code or copy and modify.
Either way, the steps and output format are the same.
Steps
- Read DISCOVERIES.md. Check what's already proven. Do not repeat existing experiments.
- Identify the hypothesis. Either from TODO.md, research/questions.yaml, or the user's request. State it as: "If we do X, then Y will happen because Z."
- Create the experiment file. Copy
src/sparse_parity/experiments/_template.py. Change one variable from the baseline. - Run the experiment. Capture results including accuracy, ARD, DMC, wall time. Record seed, config, environment (Python version, numpy version, OS, git hash).
- Save Phase 1 output. Write
results/{exp_id}/results.jsonwith raw numbers, config, and environment. No interpretation in this file. - Verify. Re-run with a different seed. If the result only holds on one seed, note that.
- Write Phase 2 findings. Create
docs/findings/{exp_id}.mdusing the template from LAB.md. UseStatus: SUCCESS | PARTIAL | FAILED(not "COMPLETED"). Reference the results JSON. Add analysis and impact. - Classify in research/log.jsonl. Use
"class": "WIN"only if the result is a clear improvement. Use"PARTIAL"for mixed results. Use"LOSS"for negative results. All three are valid findings. - Update DISCOVERIES.md if the finding answers an open question or establishes a new fact.
After merge: changelog and reporting
Not every experiment needs a changelog entry. After a PR is merged, the reviewing agent decides:
- Add to changelog if the result changes the best known method, maps a new frontier, answers an open question from DISCOVERIES.md, or is the first contribution from a new researcher.
- Skip changelog if the result confirms what's already known or is a minor null result.
The changelog entry goes in docs/changelog.md with the next version number, a short description of the finding, and a link to the findings doc. The reviewing agent writes this on merge, not the contributor.
If the result is significant enough for a meeting presentation, use the prepare-meeting skill to compile it into a report.
Checklist
- DISCOVERIES.md read
- Hypothesis stated
- One variable changed from baseline
- Experiment run with seed recorded
- results.json saved with config + environment
- Verified with different seed
- Findings doc written in docs/findings/ with Status: SUCCESS/PARTIAL/FAILED
- log.jsonl updated with correct class (WIN/PARTIAL/LOSS)
- DISCOVERIES.md updated if applicable
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

