autoresearch
Skill11 starsChanged 3 months agoMITView on GitHub
Autonomous iterative experimentation loop for any programming task. Guides the user through defining goals, measurable metrics, and scope constraints, then runs an autonomous loop of code changes, testing, measuring, and keeping/discarding results. Inspired by Karpathy''s autoresearch. USE FOR: autonomous improvement, iterative optimization, experiment loop, auto research, performance tuning, automated experimentation, hill climbing, try things automatically, optimize code, run experiments, autonomous coding loop. DO NOT USE FOR: one-shot tasks, simple bug fixes, code review, or tasks without a measurable metric.
- Deletes or force-pushes
105 other projects use a version of this, 7 of them edited.
Adopted by
Most recently taken up by
- KPISTOLAS/claude-skillstheir own version, first seen Nov 16, 2026
- matt-riley/agent-skillstheir own version, first seen Nov 15, 2026
- tboome33/obsidian-mcp-routertheir own version, first seen Nov 15, 2026
- AndyMelnykov/obsidian-claudetheir own version, first seen Nov 15, 2026
- thisis-romar/opendeck-factorytheir own version, first seen Nov 15, 2026
- arbrain-ai/arbraintheir own version, first seen Nov 15, 2026
Used with
Projects with it also use aeon-update, auto-workflow and 10 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
aaronjmarsSkillaeon-update
Pull framework updates from the upstream Aeon repo into this instance - 3-way merges canon's new commits into a PR, never clobbering operator config.
Also in 4 projects that use this
aaronjmarsSkillauto-workflow
Two-mode aeon.yml workflow builder - analyze inspects URLs and emits a tiered, signal-verified skill-enablement plan plus an aeon.yml diff; enable flips slugs to enabled:true and opens a PR.
Also in 4 projects that use this
- +10 more from aaronjmars
Other versions105 projects use a version of this.
Word for word · 0
None found.
Edited · 7
- github/awesome-copilot★ 41k62% same
- a-tokyo/aiworkspace★ 1962% same
- SebastienDegodez/skraft-plugin★ 1562% same
- asterixix/polish-academic-mcp★ 462% same
- bmsuisse/pydeno★ 162% same
- taixuann/development-space★ 162% same
- ReaperOAK/ForgeOS★ 062% same
Wrote their own version · 98
- Yeachan-Heo/oh-my-claudecode★ 41ksame name
- Yeachan-Heo/oh-my-claude-sisyphus★ 41ksame name
- Yeachan-Heo/oh-my-codex★ 34ksame name
- AgriciDaniel/claude-obsidian★ 16ksame name
- Orchestra-Research/AI-research-SKILLs★ 13ksame name
- zechenzhangAGI/AI-research-SKILLs★ 13ksame name
- Orchestra-Research/AI-Research-SKILLs★ 12ksame name
- advaitpaliwal/feynman★ 10ksame name
- companion-inc/feynman★ 10ksame name
- getcompanion-ai/feynman★ 10ksame name
88 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository10 other files it gives its agents.
- AGENTS.mdAGENTS.md
- create-skill-autoresearchSkill
- design-taste-frontendSkill
- documentation-writerSkill
- handoffSkill
- llm-councilSkill
- premortemSkill
- production-gradeSkill
- tribunalSkill
- writing-great-skillsSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
39 rules. 0 are common, 25 are distinctive, and 14 are very specific to this project.
0
Common
Most of the 101 edited versions we read keep these.
25
Distinctive
This author's own choices. Worth a look.
14
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- could be automatic: a setting could enforce it
- Distinctive: line 25
DO use the METRIC protocol for all measurements.
- Very specific: line 26
DO keep structured logs: autoresearch.jsonl (machine) + results.tsv (human).
- Distinctive: line 28
DO commit before running, revert on failure -- only kept commits remain on the branch.
- Distinctive: line 29
DO run autonomously once the loop starts -- never pause to ask "should I continue?".
- Very specific: line 30
DO run checks (if autoresearch.checks.sh exists) before measuring.
- Distinctive: line 33
DO NOT: keep changes that regress the metric unless the user explicitly allowed trade-offs.
- Distinctive: line 40
Before any experimentation begins, work with the user to establish these parameters.
- Distinctive: line 41
Ask the user directly for each item.
- Very specific: line 135
If yes, create autoresearch.checks.sh with the specified checks.
- Distinctive: line 154
Ask the user to confirm. Do not proceed until confirmed.
- Very specific: line 181
Create the branch: git checkout -b autoresearch/<tag>.
- Distinctive: line 183
Read in-scope files: Read all files that are in scope to build full context.
- Distinctive: line 185
Initialize tracking files: Create in the workspace root:
- Very specific: line 188
Add results.tsv, run.log, autoresearch.jsonl, and autoresearch.md to .git/info/exclude so they stay untracked.
- Distinctive: line 202
Run this loop continuously.
- Very specific: line 297
Every experiment log entry in autoresearch.jsonl MUST include these fields:
- Distinctive: line 308
When generating experiment ideas, follow this priority order:
- Very specific: line 311
Informed by ASI: Read learned and next_action_hint from past experiments.
- Distinctive: line 313
Diversify after plateaus: If the last 3-5 experiments all failed, try a different approach.
- Distinctive: line 314
Combine winners: If experiments A and B each improved independently, try combining them.
- Distinctive: line 316
Radical changes: After exhausting incremental ideas, try larger architectural changes.
- Distinctive: line 321
If ALL experiments in the window were discarded (no improvement), check whether the plateau is within measurement noise using confidence …
- Distinctive: line 354
Time budget: If a run exceeds 2x the expected duration, kill it and treat as a crash.
- Very specific: line 355
Existing tests: If constraints require tests to pass, include them in autoresearch.checks.sh.
- Distinctive: line 360
If the session was interrupted (context limit, user stop, crash):
- Very specific: line 362
Read autoresearch.md to re-establish goal, metric, scope, and constraints
- Very specific: line 363
Read the tail of autoresearch.jsonl to reconstruct current state:
- Distinctive: line 367
Run git log --oneline on the autoresearch branch to see kept experiments
- Very specific: line 368
Check autoresearch.ideas.md for queued hypotheses
- Distinctive: line 374
Track context usage: After ~15 experiments in one session, proactively handoff rather than waiting for quality to degrade from context …
- Very specific: line 376
Minimize resume payload: When resuming, read only autoresearch.md (config), the last 5 entries from autoresearch.jsonl (recent ASI), …
- Distinctive: line 387
When the loop ends (budget reached, user interrupts, or plateau):
- Very specific: line 445
All experiments happen on the autoresearch/<tag> branch
- Distinctive: line 446
Every experiment is committed BEFORE running (clean revert point)
- Distinctive: line 448
Only stage in-scope files.
- Distinctive: line 449
KEEP: Commit stays. Branch advances.
- Distinctive: line 457
Measure everything: No experiment without a measurement.
- Very specific: line 458
METRIC protocol: All measurements use METRIC name=value format.
- Distinctive: line 461
Stay autonomous: Never stop to ask. Think harder if stuck.
“Common” is measured: kept by at least 6 in 10 of the 101 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Autoresearch: Autonomous Iterative Experimentation
An autonomous experimentation loop for any programming task. You define the goal and how to measure it; the agent iterates autonomously -- modifying code, running experiments, measuring results, and keeping or discarding changes -- until interrupted.
Inspired by Karpathy's autoresearch, enhanced with battle-tested patterns from pi-autoresearch.
Agent Behavior Rules
- DO guide the user through the Setup phase interactively before starting the loop.
- DO establish a baseline measurement before making any changes.
- DO use the METRIC protocol for all measurements.
- DO keep structured logs:
autoresearch.jsonl(machine) +results.tsv(human). - DO record ASI fields (hypothesis, learned, rollback_reason) on every experiment.
- DO commit before running, revert on failure -- only kept commits remain on the branch.
- DO run autonomously once the loop starts -- never pause to ask "should I continue?".
- DO run checks (if
autoresearch.checks.shexists) before measuring. - DO NOT modify files the user marked as out-of-scope.
- DO NOT skip the measurement step -- every experiment must be measured.
- DO NOT keep changes that regress the metric unless the user explicitly allowed trade-offs.
- DO NOT install new dependencies or make environment changes unless the user approved it.
Phase 1: Setup (Interactive)
Before any experimentation begins, work with the user to establish these parameters. Ask the user directly for each item. Do not assume or skip any.
1.1 Define the Goal
Ask the user:
What are you trying to improve or optimize?
Examples: execution time, memory usage, binary size, test pass rate, code coverage, API response latency, throughput, error rate, benchmark score, build time, bundle size, LLM output quality, prompt effectiveness, skill accuracy, etc.
Record the user's answer as the goal.
1.2 Define the Metric
Ask the user:
How do we measure success? What exact command produces the metric?
I need:
- The command to run (e.g.,
./evaluate.sh,npm run benchmark,pytest --tb=short)- Metric output format: The command must print
METRIC name=valuelines to stdout. Example output:METRIC quality_score=8.33- Primary metric name: Which METRIC name is the main optimization target?
- Direction: Is lower better or higher better?
If your command doesn't emit METRIC lines yet, I can help you wrap it.
Record:
METRIC_COMMAND: the command to runPRIMARY_METRIC: the name of the primary metric (e.g.,quality_score)METRIC_DIRECTION:lower_is_betterorhigher_is_better
1.3 Define the Scope
Ask the user:
Which files or directories am I allowed to modify?
And which files are OFF LIMITS (read-only)?
Record:
IN_SCOPE_FILES: files/dirs the agent may editOUT_OF_SCOPE_FILES: files/dirs that must not be modified
1.4 Define Constraints
Ask the user:
Are there any constraints I should respect?
Examples:
- Time budget per experiment (e.g., "each run should take < 2 minutes")
- No new dependencies
- Must keep all existing tests passing
- Must not change the public API
- Must maintain backward compatibility
- VRAM/memory limit
- Code complexity limits (prefer simpler solutions)
Record as CONSTRAINTS.
1.5 Define the Experiment Budget
Ask the user:
How many experiments should I run, or should I just keep going until you stop me?
You can say a number (e.g., "try 20 experiments") or "unlimited" (I'll run until you interrupt).
Record as MAX_EXPERIMENTS (number or unlimited).
1.6 Simplicity Criterion
Inform the user of the default simplicity policy:
Simplicity policy (default): All else being equal, simpler is better. A small improvement that adds ugly complexity is not worth it. Removing code while maintaining or improving the metric is a great outcome. I'll weigh the complexity cost against the improvement magnitude. Does this policy work for you, or do you want to adjust it?
Record any adjustments as SIMPLICITY_POLICY.
1.7 Optional: Correctness Gate
Ask the user:
Is there a correctness check that must pass before I measure the metric?
For example: lint checks, type checks, test suites, syntax validation. If yes, I'll create an
autoresearch.checks.shscript that must exit 0 before any metric measurement. Failed checks = experiment treated as crash.
If yes, create autoresearch.checks.sh with the specified checks.
1.8 Confirm Setup and Write Session Contract
Summarize all parameters back to the user in a clear table:
| Parameter | Value |
| Goal | ... |
| Metric command | ... |
| Primary metric | ... |
| Direction | lower is better / higher ... |
| In-scope files | ... |
| Out-of-scope files | ... |
| Constraints | ... |
| Max experiments | ... |
| Simplicity policy | ... |
| Checks script | yes / no |
Ask the user to confirm. Do not proceed until confirmed.
Once confirmed, write the session contract file autoresearch.md in the workspace:
# Autoresearch Session
- **Goal**: <goal>
- **Primary metric**: <PRIMARY_METRIC> (<METRIC_DIRECTION>)
- **Metric command**: `<METRIC_COMMAND>`
- **Checks command**: `./autoresearch.checks.sh` (or "none")
- **Budget**: <MAX_EXPERIMENTS> experiments
- **Branch**: autoresearch/<tag>
- **In-scope**: <IN_SCOPE_FILES>
- **Out-of-scope**: <OUT_OF_SCOPE_FILES>
- **Constraints**: <CONSTRAINTS>
- **Simplicity**: <SIMPLICITY_POLICY>
- **Started**: <ISO timestamp>Phase 2: Branch & Baseline
Once the user confirms:
- Create a branch: Propose a tag based on today's date (e.g.,
autoresearch/may30). Create the branch:git checkout -b autoresearch/<tag>. - Read in-scope files: Read all files that are in scope to build full context.
- Initialize tracking files: Create in the workspace root:
results.tsvwith the header:experiment\tcommit\tmetric\tstatus\tdescriptionautoresearch.jsonl(empty, will be appended to) Addresults.tsv,run.log,autoresearch.jsonl, andautoresearch.mdto.git/info/excludeso they stay untracked.
- Run the baseline: Execute the metric command on the current unmodified code.
Record the result as experiment
0with statusbaseline. Log the baseline to bothresults.tsvandautoresearch.jsonl. - Report baseline to the user:
Baseline established: [PRIMARY_METRIC] = [value] Starting autonomous experimentation loop.
Phase 3: Experiment Loop
Run this loop continuously. Do not stop to ask the user. Run until:
MAX_EXPERIMENTSis reached, OR- The user manually interrupts, OR
- Plateau detected (see below)
For each experiment:
LOOP:
1. THINK - Analyze previous results, ASI from past experiments, and current code.
Generate a hypothesis: "I believe <change> will improve <metric> because <reason>."
Check autoresearch.ideas.md for queued hypotheses.
Consider: what worked, what didn't, what hasn't been tried.
2. EDIT - Modify the in-scope file(s) to implement the hypothesis.
Keep changes focused and minimal per experiment.
3. COMMIT - Stage and commit before running:
`git add <in-scope-files> && git commit -m "experiment N: <description>"`
This ensures every experiment has a clean revert point.
4. RUN - If autoresearch.checks.sh exists, run it first.
If checks FAIL: log status = "checks_failed", revert commit
(`git reset --hard HEAD~1 && git clean -fd -e 'autoresearch.*' -e 'results.tsv' -e 'run.log'`),
skip to LOG.
If checks PASS (or no checks): execute the metric command.
Redirect output to run.log: `<command> > run.log 2>&1`
5. MEASURE - Parse METRIC lines from run.log.
Extract PRIMARY_METRIC value. Record all secondary metrics.
If extraction fails (crash/error), read the last 50 lines
of run.log for diagnostics.
6. DECIDE - Compare PRIMARY_METRIC to the current best:
- KEEP: Metric improved AND improvement exceeds noise threshold.
Noise threshold = max(0.01, 1.0 × MAD of recent kept scores).
Use the last 5 kept experiments for MAD; if < 3 kept, use 0.01.
Improvements within noise are treated as DISCARD.
Update the "best" baseline. Log status = "keep".
- DISCARD: Metric same, worse, or improved within noise. Revert:
`git reset --hard HEAD~1 && git clean -fd -e 'autoresearch.*' -e 'results.tsv' -e 'run.log'`
Log status = "discard".
- CRASH: Metric extraction failed or runtime error.
Attempt a quick fix (typo, import, simple error).
If fixed, amend the commit (`git commit --amend`) and rerun from step 4.
If unfixable after 2 attempts, revert (same as DISCARD)
and log status = "crash".
7. LOG - Append to results.tsv:
experiment_number commit_hash metric_value status description
Append to autoresearch.jsonl (one JSON object per line):
{
"experiment": N,
"status": "keep|discard|crash|checks_failed|baseline",
"hypothesis": "<what we thought would improve>",
"primary_metric": <value>,
"secondary_metrics": { "<name>": <value>, ... },
"commit": "<hash or null>",
"description": "<what changed>",
"learned": "<insight that survives revert>",
"rollback_reason": "<why discarded, if discarded>",
"next_action_hint": "<what to try next>",
"confidence": <MAD ratio or null>,
"timestamp": "<ISO timestamp>"
}
If a hypothesis was deferred, append it to autoresearch.ideas.md.
8. CONTINUE - Check plateau. Go to step 1.METRIC Protocol
The metric command MUST print lines in this format to stdout:
METRIC <name>=<value>Rules:
- One METRIC per line, at line start
- Names:
[\w.]+(alphanumeric, underscores, dots) - Values: finite numbers (integers or floats)
- The PRIMARY_METRIC name must appear in the output
- Additional metrics are tracked as secondary (for tradeoff visibility)
Examples:
METRIC quality_score=8.33
METRIC structure=9.0
METRIC coverage=7.5
METRIC latency_ms=142ASI (Actionable Side Information)
Every experiment log entry in autoresearch.jsonl MUST include these fields:
- hypothesis: What we believed would improve the metric and why
- learned: The key insight from this experiment (survives revert)
- rollback_reason: Why the experiment was discarded (null if kept)
- next_action_hint: What to try next based on this result
ASI is the ONLY structured memory that survives a git revert. It prevents the agent from repeating failed approaches and builds cumulative knowledge across the session.
Experiment Strategy
When generating experiment ideas, follow this priority order:
- Low-hanging fruit first: Simple parameter tweaks, obvious inefficiencies.
- Informed by ASI: Read
learnedandnext_action_hintfrom past experiments. - Check the ideas backlog: Review
autoresearch.ideas.mdfor queued hypotheses. - Diversify after plateaus: If the last 3-5 experiments all failed, try a different approach.
- Combine winners: If experiments A and B each improved independently, try combining them.
- Simplification passes: Periodically try removing code/complexity to see if the metric holds.
- Radical changes: After exhausting incremental ideas, try larger architectural changes.
Plateau Detection
Track a sliding window of the last PLATEAU_WINDOW experiments (default: 5).
If ALL experiments in the window were discarded (no improvement), check whether the plateau is within measurement noise using confidence scoring (see below).
Report:
Plateau detected after [N] consecutive failed experiments. Best score: [value]. Confidence: [ratio]x MAD. Consider:
- Trying a fundamentally different approach
- Reviewing the ideas backlog for untried directions
- Stopping if confidence is high (improvements may be at measurement ceiling)
Continue if budget remains, but shift strategy to radical changes.
Confidence Scoring (MAD)
For noisy metrics (benchmarks with variance between runs), track confidence using Median Absolute Deviation:
- Collect all primary metric values from the current session (kept + discarded, excluding crashes)
- Compute the median of all values
- Compute MAD = median of absolute deviations from the median
- Confidence ratio = |best_kept - baseline| / MAD
Interpretation:
- >= 2.0x MAD: Improvement is likely real (high confidence)
- 1.0-2.0x MAD: Above noise but marginal (medium confidence)
- < 1.0x MAD: Within noise -- consider re-running or treating as no improvement
The logged confidence ratio is advisory -- it provides interpretive context but does not override the binding noise threshold in DECIDE (which uses the same MAD stats). The DECIDE gate is the binding rule; confidence is for human/agent reasoning about result reliability.
For metrics with zero variance (deterministic), confidence is null and the noise threshold falls back to the 0.01 absolute minimum.
Handling Constraints
- Time budget: If a run exceeds 2x the expected duration, kill it and treat as a crash.
- Existing tests: If constraints require tests to pass, include them in
autoresearch.checks.sh. - Memory/resources: Monitor and revert if resource usage exceeds stated limits.
Resume After Interrupt
If the session was interrupted (context limit, user stop, crash):
- Read
autoresearch.mdto re-establish goal, metric, scope, and constraints - Read the tail of
autoresearch.jsonlto reconstruct current state:- Last experiment number
- Best score and commit
- Recent ASI (
learned,next_action_hint) from the last few experiments
- Run
git log --onelineon the autoresearch branch to see kept experiments - Check
autoresearch.ideas.mdfor queued hypotheses - Resume the loop from the next experiment number
Context Budget Awareness
Each experiment consumes context. To avoid degradation:
- Track context usage: After ~15 experiments in one session, proactively handoff rather than waiting for quality to degrade from context saturation
- Minimize resume payload: When resuming, read only
autoresearch.md(config), the last 5 entries fromautoresearch.jsonl(recent ASI),autoresearch.ideas.md, andresults.tsvsummary. Do NOT reload all research, gold standards, or full experiment history into context - Prefer small diffs: Summarize prior experiments rather than replaying them.
The JSONL
learnedandnext_action_hintfields carry the signal without the bulk
Phase 4: Reporting
When the loop ends (budget reached, user interrupts, or plateau):
- Print the full results.tsv as a formatted table.
- Summarize:
- Total experiments run
- Experiments kept / discarded / crashed
- Starting metric (baseline) vs. final metric
- Improvement percentage
- Top 3 most impactful changes (from kept experiments)
- Key learnings (aggregated from ASI
learnedfields)
- Show the cumulative git log of kept experiments:
git log --oneline <start_commit>..HEAD - Recommend next steps: Based on ASI and results, suggest what to try next.
- Ideas backlog: Print any remaining items from
autoresearch.ideas.md.
Quick Reference
METRIC Line Format
METRIC <name>=<value> # one per line, at line start
METRIC quality_score=8.33
METRIC structure=9.0Names: [\w.]+, values: finite numbers. Primary metric named in session config.
Results TSV Format
Tab-separated, 5 columns:
experiment commit metric status description
0 a1b2c3d 0.997900 baseline unmodified code
1 b2c3d4e 0.993200 keep increase learning rate to 0.04
2 - 1.005000 discard switch to GeLU activation
3 - 0.000000 crash double model width (OOM)autoresearch.jsonl Format
One JSON object per line. Status values: baseline, keep, discard, crash, checks_failed. Each entry contains: experiment, status, hypothesis, primary_metric, secondary_metrics, commit, description, learned, rollback_reason, next_action_hint, confidence, timestamp.
Session Artifacts
| File | Purpose | Tracked in git? |
autoresearch.md | Session contract (goal, config) | No (.git/info/exclude) |
autoresearch.jsonl | Machine log with ASI | No (.git/info/exclude) |
results.tsv | Human-readable results journal | No (.git/info/exclude) |
autoresearch.checks.sh | Correctness gate (optional) | Yes (committed) |
autoresearch.ideas.md | Deferred hypothesis backlog | No (.git/info/exclude) |
run.log | Last command output | No (.git/info/exclude) |
Git Workflow (Commit-First Model)
- All experiments happen on the
autoresearch/<tag>branch - Every experiment is committed BEFORE running (clean revert point)
- Commit:
git add <in-scope-files> && git commit -m "experiment N: <description>"Only stage in-scope files. Nevergit add -A(risks staging out-of-scope changes). - KEEP: Commit stays. Branch advances.
- DISCARD:
git reset --hard HEAD~1 && git clean -fd -e 'autoresearch.*' -e 'results.tsv' -e 'run.log' - CRASH: Attempt fix + amend; if unfixable, same revert as DISCARD
- CHECKS_FAILED: Same revert as DISCARD (revert before measuring)
- Session artifacts stay untracked (added to
.git/info/exclude)
Key Principles
- Measure everything: No experiment without a measurement.
- METRIC protocol: All measurements use
METRIC name=valueformat. - ASI survives reverts: Log hypothesis + learned even on discard.
- Revert failures: The branch only advances on improvements.
- Stay autonomous: Never stop to ask. Think harder if stuck.
- Keep it simple: Complexity is a cost. Weigh it against gains.
- Log everything: JSONL is the machine journal, TSV is the human journal.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

