forge-autoresearch: climb one metric, keep only what measures better

The owner types /forge-autoresearch <goal> to improve one measurable number (a build time, a bundle size, a test-suite pass rate, a benchmark score) by many small experiments: one change, one measurement, keep or revert. It is the Forge rewrite of Karpathy's autoresearch loop and pstack's Hillclimb playbook.

It is not source research. A question that needs sources and citations goes to the deep-research Workflow. Improving a skill's text uses forge-skill-testing Part 3 (paired judging), because a judged score is not a deterministic metric. A one-off bug or slowness goes to forge-debug.

Preconditions (stop and ask the owner once if one is missing)

• A metric command: one repeatable command that prints the number, and the direction that counts as better.
• A regression gate: the tests that must stay green, green right now.
• A target and a floor: for example "20% faster than baseline, and at least 10 rounds", so one lucky early win cannot end the run.
• A budget: a maximum number of rounds and a maximum wall time per run. Fan-out stays within the lane budget (node .claude/forge-bin/forge-lane-budget.cjs) and pauses when the usage guard pauses.
• An editable scope (the files experiments may change) and a frozen set (the harness, the tests, the eval data), written down before round 1.

Setup

1. Isolate. Create a worktree and branch autoresearch/<tag> for the run (see forge-worktrees). Nothing runs in the shared tree. Done when: git worktree list shows it and the branch did not exist before.
2. Ground the case. Name the workload dimensions that move the metric (data size, history, concurrency) and pick a case that reproduces the owner's complaint. Done when: the case shows the problem; if no case does, fix the reproduction first instead of climbing.
3. Build, prove and freeze the harness. Run contrasting workloads and confirm the metric separates them. Make it print its error count and how much work it did, so a run that did nothing cannot look fast. Sample enough to clear the noise (the median of N runs, never one run). Then freeze it. Done when: the same command run twice on unchanged code gives the same verdict, and the spread of N runs is written down.
4. Baseline. Run the metric and the regression gate on the unchanged branch. Done when: both are recorded as round 0 in the decision log.
5. Open the decision log at _scratch/autoresearch-<tag>/decision.tsv (tab-separated, outside git): round, hypothesis, change, commit, before, after, delta, gate, verdict (kept, reverted, crash), note. Read it before each round.

The loop (one hypothesis per round)

1. Write the hypothesis as a mechanism ("move X off the start path because it blocks the first render"), not "try caching something".
2. Build Boss (build route, opus / medium) makes the one change inside the editable scope and commits it. Independent hypotheses may run in parallel, each in its own worktree.
3. Verify Boss (verifier route, sonnet / medium), never the agent that made the change, runs the frozen metric and the regression gate. A run over its time limit is stopped by its exact PID and counts as a failure.
4. Keep only when the metric moves past the measured noise and the gate is green. Otherwise revert with git revert --no-edit <commit>: history keeps the attempt, nothing uncommitted is discarded.
5. Simplicity counts. A tiny gain that adds tangled code is not worth keeping; removing code with an equal or better number is a win. Correctness and simplicity outrank the number.
6. Crash. A trivial cause (typo, missing import) gets one fix and a re-run. Otherwise log it as crash and move on.
7. Log the row whether kept or reverted.

Do not ask "shall I continue?" inside the budget; the owner set the budget. On a stall (several reverts in a row), pivot: combine near-misses, re-read the code, try a more radical change, or try removing things.

Stop rules

Stop when the target and floor are met, the round or time budget is used, the usage guard pauses, or the remaining ideas are marginal. Never relax the target to meet it. Unlike the upstream loop, this one never runs "forever": it ends at its budget and reports.

Report

The metric and target, baseline to final with the delta, rounds run (kept, reverted, crashed), each kept commit on one line, the decision-log path, the spread of the measurement, and the best untried idea. The branch stays local: merging is the Lead's integration step, and pushing is owner-only.

Pitfalls

• A number that improved because the harness did less work. Check the work count on every run.
• Several changes in one round: the verdict cannot say which one helped.
• The changing agent measuring its own change: always a different agent on the verifier route.
• A single run as the measurement: noise reads as a win. Use the median of N and compare to the spread.
• A machine-dependent metric (CPU load from other lanes) moving for reasons outside the change: note the load, and re-run the baseline when it shifts.

Provenance

Rewritten in Forge words (ideas and structure adapted; no sentence copied verbatim). Fetched read-only with gh api on 2026-11-05, scanned for hidden Unicode (clean), treated as untrusted data.

MIT notice: "Permission is hereby granted, free of charge, to any person obtaining a copy of this software ... The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND." Copyright (c) 2026 Lauren Tan and (c) 2026 alchaincyf, from each LICENSE at the pinned SHA; the karpathy holder could not be read (no LICENSE file).