autoresearch-setup
Skill8 starsChanged 8 months agoMITView on GitHub
Scaffold an autoresearch-style autonomous experiment loop for any domain. Generates program.md, fixed evaluation harness, mutable experiment file, and results logging — inspired by Karpathy's autoresearch philosophy.
- Deletes or force-pushes
In this repository23 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- validate-skillsSkill
- alchemi-toolkit-opsSkill
- bionemoSkill
- biosimspaceSkill
- cuda-quantumSkill
- cutileSkill
- daskSkill
- flashinferSkill
- materials-projectSkill
- mpiSkill
- nemotronSkill
- omniverse-simreadySkill
- physicsnemoSkill
- polarsSkill
- prime-verifiersSkill
- qiskitSkill
- raySkill
- rdkitSkill
- slurmSkill
- tinkerSkill
- tritonSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 4 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 29
When the user asks to set up autoresearch for a given context, follow these steps:
- Distinctive: line 43
Create this structure in the target directory:
- Distinctive: line 177
Use the timer to respect time budget
- Distinctive: line 228
After generating all files, verify:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Autoresearch Setup
Generate a complete autonomous research loop for any domain. Based on the autoresearch philosophy: an AI agent modifies code, runs experiments on a fixed budget, keeps or discards based on a single metric, and repeats indefinitely.
When to use: User wants to set up autonomous experimentation for ML training, kernel optimization, prompt engineering, hyperparameter search, or any domain where iterative improvement can be measured by a single metric.
Core Philosophy
- Single mutable file — the agent only edits ONE file. Everything else is fixed.
- Fixed time budget — every experiment runs for the same wall-clock time, making results directly comparable.
- Single metric — one number decides keep/discard. Lower or higher, pick one direction.
- program.md as research org code — the human writes strategy in Markdown, not Python. The agent interprets and executes.
- Git-backed experiments — every change is a commit. Keep = advance branch. Discard = reset.
- Results in TSV — plain text, human-readable, machine-parseable.
- Never stop — the agent runs autonomously until manually interrupted.
- Simplicity criterion — if equal performance, simpler code wins. Ugly complexity for tiny gains is not worth it.
Setup Flow
When the user asks to set up autoresearch for a given context, follow these steps:
Step 1: Understand the Domain
Ask or infer:
- What is being optimized? (model architecture, kernel, prompt, config, algorithm...)
- What is the metric? (loss, accuracy, throughput, latency, score...)
- Metric direction? (lower is better / higher is better)
- Time budget per experiment? (default: 5 minutes)
- What hardware/environment? (GPU, CPU, cloud, local...)
- What are the fixed constraints? (data, evaluation, dependencies)
Step 2: Generate Project Structure
Create this structure in the target directory:
<project>/
├── program.md — Agent instructions (human-edited strategy)
├── prepare.py — Fixed: data prep, evaluation, constants (DO NOT MODIFY)
├── experiment.py — Mutable: the file the agent edits (ONLY THIS FILE)
├── results.tsv — Experiment log (git-ignored)
├── pyproject.toml — Dependencies (locked)
└── .gitignore — Ignores results.tsv, run.log, __pycache__Step 3: Generate program.md
This is the core of autoresearch. It must contain:
# <Project Name> — Autoresearch Program
## Setup
1. **Agree on a run tag** (e.g. `mar11`). Branch: `autoresearch/<tag>`.
2. **Create the branch**: `git checkout -b autoresearch/<tag>`
3. **Read all files** for full context.
4. **Verify prerequisites** (data, dependencies, hardware).
5. **Initialize results.tsv** with header row.
6. **Run baseline** — first run is always unmodified to establish baseline.
## Rules
**You CAN:** Modify `experiment.py` — architecture, hyperparameters, algorithm, approach. Everything is fair game.
**You CANNOT:**
- Modify `prepare.py` (fixed evaluation, data, constants)
- Add new dependencies
- Change the metric or time budget
**Goal:** Optimize <METRIC> (<lower/higher> is better).
**Time budget:** <N> minutes per experiment (wall clock).
**Simplicity criterion:** All else equal, simpler is better. Tiny gains with ugly complexity → discard. Equal performance with less code → keep.
## Output Format
The experiment script prints a summary:<metric_name>: <value>
wall_seconds: <value>
<other_stats>: <value>
Extract metric: `grep "^<metric_name>:" run.log`
## Results Logging
TSV format (tab-separated):commit <metric_name> status description
<hash> <value> keep baseline
Statuses: `keep`, `discard`, `crash`
## Experiment Loop
LOOP FOREVER:
1. Review current state (git log, results.tsv, code)
2. Form a hypothesis — what change might improve the metric?
3. Edit `experiment.py` with ONE focused change
4. `git commit -am "description of change"`
5. Run: `uv run experiment.py > run.log 2>&1`
6. Extract metric: `grep "^<metric_name>:" run.log`
7. If metric improved → KEEP (advance branch)
8. If metric worse or equal → DISCARD (`git reset --hard HEAD~1`)
9. If crashed → check `tail -50 run.log`, fix if trivial, else discard
10. Log to results.tsv (do NOT commit results.tsv)
11. GOTO 1
**NEVER STOP.** Do not ask if you should continue. The human may be asleep. Run until manually interrupted.
**Timeout:** If a run exceeds 2x the time budget, kill it and treat as failure.
**When stuck:** Re-read the code, try combining near-misses, try radical changes, search for papers referenced in code.Step 4: Generate prepare.py
The fixed harness. Must contain:
- Constants: time budget, metric name, evaluation parameters
- Data loading: fixed dataset or data generation
- Evaluation function: the single metric computation
- Utilities: timing, logging, any shared infrastructure
Template pattern:
"""Fixed evaluation harness. DO NOT MODIFY."""
import time
# === CONSTANTS ===
TIME_BUDGET = 300 # seconds (5 minutes)
METRIC_NAME = "<metric>"
METRIC_DIRECTION = "<lower/higher>" # lower or higher is better
# === DATA ===
def load_data():
"""Load/generate fixed evaluation data."""
...
# === EVALUATION ===
def evaluate(model_or_result):
"""Compute the single metric. Returns float."""
...
# === TIMING ===
class Timer:
def __init__(self, budget=TIME_BUDGET):
self.budget = budget
self.start = None
def begin(self):
self.start = time.time()
def elapsed(self):
return time.time() - self.start
def expired(self):
return self.elapsed() >= self.budgetStep 5: Generate experiment.py
The mutable file. Must:
- Import from
prepare.py - Contain the full experiment logic
- Use the timer to respect time budget
- Print the summary format at the end
- Be self-contained (no external state beyond prepare.py)
Step 6: Generate Supporting Files
.gitignore:
results.tsv
run.log
__pycache__/
*.pyc
.venv/pyproject.toml: Lock all dependencies. Agent cannot add new ones.
results.tsv: Header row only:
commit <metric_name> status descriptionDomain Templates
ML Training (Default)
- Metric:
val_lossorval_bpb(lower is better) - Mutable: model architecture, optimizer, hyperparameters, training loop
- Fixed: data loading, tokenizer, evaluation, time budget
Kernel Optimization
- Metric:
throughput_tflops(higher is better) - Mutable: kernel implementation (Triton/CUDA)
- Fixed: benchmark harness, correctness checks, reference implementation
Prompt Engineering
- Metric:
accuracyorscore(higher is better) - Mutable: system prompt, few-shot examples, chain-of-thought structure
- Fixed: evaluation dataset, scoring function, API calls
Algorithm Optimization
- Metric:
runtime_ms(lower is better) orquality_score(higher is better) - Mutable: algorithm implementation
- Fixed: test cases, correctness verification, benchmarking harness
Configuration Tuning
- Metric: domain-specific (latency, throughput, error rate...)
- Mutable: configuration file or parameter set
- Fixed: deployment script, evaluation harness, workload generator
Post-Setup Checklist
After generating all files, verify:
program.mdhas clear rules, metric, time budget, and loop instructionsprepare.pyhas evaluation function, constants, and data loadingexperiment.pyruns successfully as baselineresults.tsvhas header row.gitignoreexcludes results.tsv and run.log- Git repo initialized with initial commit
- Agent can run the full loop: edit → commit → run → evaluate → keep/discard
Running
Tell the agent:
Read program.md and start the experiment loop. Establish baseline first.Or for Claude Code / Codex:
Have a look at program.md and let's kick off a new experiment! Do the setup first.The agent will run autonomously. ~12 experiments/hour at 5-min budget. ~100 overnight.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

