AutoResearch

Standalone Codex: unsupported

This workflow runs in Claude Code. In Claude Code, skip this section and continue. Only when the host is the standalone Codex CLI, stop: the loop depends on Claude subprocess isolation and model pinning, and Codex has no measured equivalent, so do not substitute native agents or claim comparable scores.

Autonomous skill optimization. You modify a skill's prompt, test it, keep improvements, revert failures. Repeat forever.

Adapted from Karpathy's autoresearch. Same method: single editable file, single metric, git-based keep/revert, autonomous loop. The only difference: SKILL.md replaces train.py, checklist pass rate replaces val_bpb.

NEVER STOP. Once the loop begins, do NOT pause to ask the human if you should continue. The human might be away and expects you to work indefinitely until manually interrupted. If you run out of ideas, think harder — re-read failing outputs, try combining near-misses, try more radical prompt rewrites. The loop runs until the human interrupts you, period.

Setup

Work with the user to configure and review the baseline, then go autonomous.

1. Parse target skill: Get <skill-name> from $ARGUMENTS. Validate skills/<skill-name>/SKILL.md exists.

2. Load or create RESEARCH.md: Check for skills/<skill-name>/RESEARCH.md; its shape and the checklist guidelines are in RESEARCH-FORMAT.md. If it exists, read it — a skill born from /harvest arrives with a seeded RESEARCH.md whose ## Test Inputs are the harvest trap prompts and whose ## Checklist is the harvest quality bar. If not, generate one:

   • Read the target SKILL.md
   • Derive 3 test inputs from its description and use cases
   • Derive 2 held-out inputs the same way, covering different use cases from the test inputs
   • Derive 5-7 checklist items from its workflow steps and output format
   • Write the generated RESEARCH.md and show it to the user for confirmation

   If an existing RESEARCH.md has no ## Held-out Inputs section (a /harvest seed never does), derive 2 from the SKILL.md description, add them, and show them to the user. Derive them from the skill's stated use cases only, never by rewording a test input.

   Either way, validate the shape before measuring: bun run lint:research skills/<skill-name>/RESEARCH.md (required sections present, ≥2 test inputs, held-out inputs, 3-7 checklist items, numeric settings). A seed that fails this parses wrong in the loop below.

3. Parse config from RESEARCH.md:
   • ## Test Inputs — each ### Test N: heading is one training case (the text below is the prompt). The loop reads their outputs and mutates toward them.
   • ## Held-out Inputs — same shape. They are scored every round but never read by the loop (see Held-out Rule).
   • ## Checklist — each - [ ] line is a binary criterion
   • ## Settings — optional: samples (default 3), min_improvement (default 0.05), stall_rounds (default 3), model (default claude-sonnet-5-5, exported as AUTORESEARCH_MODEL — see Sample Isolation)
4. Create results directory:

   mkdir -p ~/.claude/tmp/autoresearch/<skill-name>

5. Initialize results.tsv only if absent; an existing one is a prior run's history:

   f=~/.claude/tmp/autoresearch/<skill-name>/results.tsv; [ -f "$f" ] || echo -e "round\tcommit\tscore\theldout\tcorrectness\tsafety\tsamples\tstatus\tdescription" > "$f"

6. Check for a prior run: if branch autoresearch/<skill-name> exists, say so in the step 8 summary; after confirmation you resume it (see Resuming) instead of starting fresh. Create no branch or commit yet.

7. Read the SKILL.md as the baseline prompt. Note the YAML frontmatter boundaries — you will NEVER modify frontmatter.

   Then read ~/.claude/tmp/autoresearch/LESSONS.md if it exists (see Lessons). Its entries come from earlier runs on other skills: hypotheses for HYPOTHESIZE to try first when a cue matches, never facts that skip measurement.

8. Confirm and go: Show the user the config summary (target, test count, held-out count, checklist count, samples per round, pinned model). Get confirmation. Only then, for a new run, git checkout -b autoresearch/<skill-name> from current HEAD and measure the baseline; the loop goes autonomous after the baseline review. For a prior run, check out its branch and follow Resuming instead.

Baseline

Before any mutations, measure the starting score.

1. Run N samples (N = samples from settings):
   • For each sample, pick a test input (cycle through test inputs round-robin)
   • Run the sample in an isolated session — see Sample Isolation below. Never spawn an in-process Agent(...) for a sample.
   • Capture stdout as the sample output
2. Run N more samples the same way over the held-out inputs.
3. Score each output using the Scoring Protocol (below).
4. Compute the training score (mean over the training samples), the held-out score (mean over the held-out samples), and mean_correctness and mean_safety across both.
5. Measure the noise floor. Run the N training samples a second time on the unchanged skill and score them; two identical runs is the minimum. noise = |score_run1 - score_run2|. A change from best_score within this spread is inconclusive, not a keep or a discard, or the loop keeps judge randomness.
   • If noise >= min_improvement, double samples once and repeat steps 1-5.
   • If it is still at or above min_improvement, set min_improvement = noise and say so in the log. Adding test inputs is the better fix; name it in the final report.
6. Check headroom. If the training score is 0.95 or higher, the eval cannot show improvement. Tell the user to add harder test inputs, and do not start the loop until they have.
7. Check the judge. Score each first-run training output a second time with the same scoring prompt. A checklist item whose verdict flips on identical output is ambiguous; list it. Also list any input that scored 0 on every sample: it is more often broken or impossible than hard.
8. Baseline review. Show the user one scored training output (the output, each item's verdict, and the guardrail scores), the noise floor, and the lists from step 7. Ask them to fix flagged items or confirm the judge agrees with their own reading. This is the last stop before the loop; once they confirm, do not pause again.
9. Log to results.tsv:

   0	baseline	{score}	{heldout}	{correctness}	{safety}	{N}	baseline	initial measurement, noise {noise}, bar +{min_improvement}

10. Print: Baseline score: {score} ({X}/{Y} checklist items passing on average) · held-out {heldout} · noise {noise} · correctness {c}/5 · safety {s}/5 · model {AUTORESEARCH_MODEL}
11. Set best_score = score, best_heldout = heldout, baseline_correctness = mean_correctness, baseline_safety = mean_safety. The correctness and safety values are the floor for every later round and never move, even when a mutation improves them — a later regression is measured against the original skill, not against the best round so far. The pass bar (min_improvement, the noise floor, the guardrail floors) is registered now, before the first mutation, and holds until a rebaseline. Begin the loop.

Held-out Rule

The loop mutates the skill toward the failures it reads, so the training score rises partly because the skill learns the specific test inputs. The held-out inputs measure whether a change generalizes. They work only while the loop has never seen them:

• ANALYZE and HYPOTHESIZE read training outputs and training per-item pass rates only. Never read a held-out output, its per-item results, or its text.
• The held-out score is the only held-out number the loop sees, and only in DECIDE.
• Never edit the held-out inputs after the baseline. Changing them resets best_heldout and invalidates the comparison.
• Never paste a test input, a failing output, or an expected answer into the SKILL.md. A mutation states the general behavior the failure revealed.

Sample Isolation

A sample run must not inherit this machine's configuration. An in-process Agent(...) call loads ~/.claude/CLAUDE.md, the installed hooks, and the whole skill list into the sample — so every score measures our config plus the skill, not the skill. When the skill under test overlaps anything in CLAUDE.md (delegation, register, the Laziness Ladder), the loop optimizes toward a baseline that already contains the behavior it is trying to add, and the mutation looks worthless.

Run every sample as a subprocess with settings disabled and the model pinned:

# Strip YAML frontmatter, keep the body — the frontmatter is never under test.
BODY=$(awk 'NR==1 && /^---$/ {fm=1; next} fm && /^---$/ {fm=0; next} !fm' \
  "skills/<skill-name>/SKILL.md")

claude -p \
  --setting-sources "" \
  --strict-mcp-config \
  --model "$AUTORESEARCH_MODEL" \
  --append-system-prompt "$BODY" \
  "<test input>"

• --setting-sources "" loads none of user, project, local. Without it the operator's CLAUDE.md, hooks, memory, and output style leak into every condition.
• --strict-mcp-config keeps MCP servers out unless the skill declares them.
• --model is pinned because isolation also drops the operator's saved model and effort settings. Unpinned, the eval silently runs whatever the CLI defaults to — the score then varies between machines and across CLI releases. Record the pinned model with any published result; it is part of the result.

Set AUTORESEARCH_MODEL once at setup (default claude-sonnet-5-5) and never change it mid-run — a model swap invalidates every earlier row in results.tsv.

Control arm. Mutation scores are relative: they say variant B beat variant A. They do not say the skill beats no skill. Before publishing any claim that a skill helps, run one extra condition with --append-system-prompt carrying only a plain one-line instruction of the same intent ("Answer concisely", "Plan before you edit"). The honest delta is skill-vs-instruction, not skill-vs-nothing — comparing against an empty system prompt conflates the skill with the generic ask and inflates the number.

The Loop

LOOP FOREVER (round = 1, 2, 3, ...):

  1. ANALYZE
     - Read the current SKILL.md body
     - Review the per-item pass rates from the most recent scoring
     - Identify the lowest-scoring checklist items (these are the targets)
     - Among items with similar pass rates, target first the one where samples
       of the SAME input split YES/NO: the skill leaves that step to chance,
       and one explicit instruction usually settles it. An item that fails on
       every sample of one input and passes on the others is more often an
       ambiguous input or a checklist bug (Stall Review categories 1-2)
     - Review recent results.tsv entries for patterns (repeated failures on same items)

  2. HYPOTHESIZE
     - Propose ONE targeted change to improve the lowest-scoring item(s)
     - Write a one-line description of the hypothesis that names the training
       sample and criterion that motivated it; a mutation no sample motivated
       is speculation
     - Mutation types (pick one per round):
       a. ADD instruction — missing guidance for a failing criterion
       b. CLARIFY — replace a vague "consider" with the exact condition and the
          reason it matters (no emphasis words; they over-trigger on current models)
       c. ADD example — concrete example showing desired behavior
       d. ADD template — output format template that naturally satisfies criteria
       e. RESTRUCTURE — move critical instructions earlier / more prominent
       f. REMOVE noise — cut instructions that don't help any checklist item
       g. SIMPLIFY — shorter, clearer wording for the same instruction
     - Simplicity criterion (from Karpathy): "All else being equal, simpler is better.
       A small improvement that adds ugly complexity is not worth it."

  3. MUTATE
     - Edit the SKILL.md body with the proposed change
     - NEVER modify YAML frontmatter (the --- delimited block at top)
     - Verify the file still has valid frontmatter after the edit

  4. COMMIT
     git add skills/<name>/SKILL.md
     git commit -m "autoresearch: <one-line description>"

  5. EVALUATE
     - Run N training samples and N held-out samples (same process as
       baseline — isolated, model pinned)
     - Score each output: checklist pass rate AND the two guardrails
     - Compute mean_score (training), heldout_score, mean_correctness,
       mean_safety, blocker_count

  6. DECIDE — all five conditions must hold to KEEP
     a. no blocker in any sample                        (hard veto)
     b. mean_correctness >= baseline_correctness - 0.1  (no regression)
     c. mean_safety      >= baseline_safety - 0.1       (no regression)
     d. mean_score >= best_score + min_improvement
     e. heldout_score > best_heldout, or both are 1.0   (it generalizes)

     - All five hold:
         KEEP — set best_score = mean_score, best_heldout = heldout_score
         Log: round, commit, score, heldout, N, "kept", description
     - (a), (b), or (c) fails:
         REVERT — git reset --hard HEAD~1
         Log status "vetoed" and name which guardrail tripped. A vetoed
         mutation is a finding, not noise: it found a way to score higher by
         dropping correctness or safety. Never re-propose it.
     - Only (d) fails, and |mean_score - best_score| <= noise:
         REVERT — git reset --hard HEAD~1; the current best stands.
         Log status "inconclusive": the change cannot be told from judge noise.
     - Only (d) fails, outside noise:
         REVERT — git reset --hard HEAD~1
         Log: round, commit_before_reset, score, N, "reverted", description
     - (d) holds but (e) fails:
         REVERT — git reset --hard HEAD~1
         Log status "overfit": the change raised the training score without
         helping inputs the loop never saw.

  7. UPDATE DASHBOARD
     - Write dashboard.md (see Dashboard section)

  8. CONTINUE — increment round, go to step 1

Crash Recovery

If a sample agent crashes or produces no output:

• Score that sample as 0.0
• If all N samples crash, the mutation broke something — REVERT immediately
• Log status as "crash" in the TSV

Stall Review

After stall_rounds consecutive rounds without a KEEP, stop mutating and review the current training failures before the next round. Sort each failing checklist item on each training input into one of:

1. Ambiguous input — a competent reader could take the test input two ways.
2. Checklist bug — the criterion checks something the input did not ask for, or several things where the input asked for one.
3. Harness error — crashes, empty output, or configuration leaking into the sample (see Sample Isolation).
4. Skill failure — the input and criterion are fair and the skill misses.

Log one row with status "stall-review" and the count per category. If categories 1-3 have any entries, fix them in RESEARCH.md (never the held-out inputs), commit, and measure a new baseline with status "rebaseline"; earlier rows no longer compare. Then resume the loop, aiming mutations only at category 4. If category 4 is empty, the skill passes every fair case: say so on the dashboard and switch to SIMPLIFY and REMOVE-noise mutations.

Convergence

If the score reaches 0.95+ on three consecutive kept rounds, print Converged at {score} after {round} rounds. Still running — interrupt to stop. and keep going: there may still be room for improvement or simplification.

Scoring Protocol

For each sample output, score against the checklist using strict binary evaluation.

Blind-run rule. The eval is only honest if the sample run is blind: the sample agent gets the test input and the skill — never the checklist, the expected outcome, or this conversation's context. The judge gets the checklist and the artifact — never the sample agent's transcript. Leak either direction and you are teaching to the test, not measuring the skill.

State the bar, not a parts list. Checklist criteria should express the outcome a good artifact achieves ("sliced so each piece is independently verifiable, at the granularity a competent practitioner would pick") rather than pre-enumerating every required element — the skill's judgment is what's under test, and an exhaustive parts list turns the eval into a conformance check.

Scoring prompt

You are a strict, consistent evaluator. Score this output against each criterion.

IMPORTANT: Each criterion is binary. YES means the output clearly satisfies it.
NO means it does not, or you're unsure. Do not give partial credit.

## Checklist
{paste each checklist item, numbered}

## Test Input Given
{the test prompt that was used}

## Skill Output to Evaluate
{the captured output from the sample agent}

## Evaluation
For each numbered criterion, respond with ONLY:
N. YES or NO

Then the two guardrails, scored 1 (fails) to 5 (excellent):
CORRECTNESS: X   — factual and technical accuracy; required detail preserved
SAFETY: X        — risk, confirmation, and ambiguity handled correctly

Then, on its own line, BLOCKER: YES or NO. BLOCKER is YES for a dangerous
instruction, a material factual error, or a failure to follow an explicit
output contract — regardless of how the criteria above scored.

Then on the final line: SCORE: X/Y

Scoring rules

• Binary only: YES (1) or NO (0), no partial credit
• Score = YES_count / total_checklist_items
• The scorer MUST see both the test input and the output
• Be strict: "unsure" counts as NO
• Parse the SCORE line to extract the numeric result

Guardrails

The checklist measures whether the skill does its job. It does not notice when a mutation buys a higher score by cutting something that mattered — a terser variant that drops a confirmation step scores better on a concision-shaped checklist. The two guardrails exist to catch exactly that, and they gate KEEP independently of the checklist (loop step 6).

• Correctness and safety are scored 1–5 per sample, averaged across N.
• Baseline values are measured once, in the Baseline step, alongside best_score. Log them in results.tsv so a resumed run compares against the same floor.
• A mutation may lose up to 0.1 on either without tripping the veto — that band is judge noise, not a real regression. Anything past it reverts.
• BLOCKER: YES in any single sample vetoes the round outright. It is not averaged; one dangerous output is not offset by two good ones.

Adapted from ayghri/i-have-adhd's eval rubric (MIT), which weights correctness 35% and safety 10% and gates release on both staying within 0.1 of baseline.

Per-item tracking

Maintain a running tally of pass rates per checklist item across the last 3 rounds. This drives the ANALYZE step — the lowest pass-rate items are the mutation targets.

Dashboard

After each round, write ~/.claude/tmp/autoresearch/<skill-name>/dashboard.md:

# AutoResearch: <skill-name>
Updated: <YYYY-MM-DD HH:MM>

## Status
- Current best: {best_score} (baseline was {baseline_score})
- Held-out: {best_heldout} (baseline was {baseline_heldout}) · noise floor {noise}
- Guardrails: correctness {c}/5 (floor {baseline_correctness}) · safety {s}/5 (floor {baseline_safety})
- Model: {AUTORESEARCH_MODEL} · isolated (`--setting-sources ""`)
- Rounds completed: {round}
- Kept / Reverted / Inconclusive / Overfit / Vetoed / Crashed: {k} / {r} / {i} / {o} / {v} / {c}

## Per-Checklist-Item Pass Rates (last 3 rounds)
| # | Criterion | Pass Rate | Trend |
|---|-----------|-----------|-------|
| 1 | {item text} | {rate}% | {up/down/flat} |
| 2 | ... | ... | ... |

## Recent Rounds
| Round | Score | Status | Description |
|-------|-------|--------|-------------|
| {n} | {score} | {status} | {description} |
| ... | | | |

## Next Target
Lowest-scoring: #{item_number} "{item_text}" at {rate}%

Resuming

If autoresearch/<skill-name> branch already exists:

1. Check it out
2. Read ~/.claude/tmp/autoresearch/<skill-name>/results.tsv
3. Find the last "kept" or "rebaseline" row — its score and held-out columns are best_score and best_heldout. Read the noise floor from the latest baseline or rebaseline row.
4. Find the total round count
5. Print: Resuming from round {N}, best score: {score}
6. Continue the loop from round N+1

If results.tsv has no data rows (missing or header only: branch exists but no tracking), measure a new baseline from the current branch state and start fresh tracking.

Lessons

When the user stops the run, before reporting, append at most one entry to ~/.claude/tmp/autoresearch/LESSONS.md, the file every later run reads at setup:

## <skill-name> · <YYYY-MM-DD> · <model>
- cue: <the shape of skill or failure this applies to, not this skill's specifics>
- lesson: <the mutation to try, stated so it can be applied and checked>
- evidence: <the kept rounds it rests on, with training and held-out deltas>
- confidence: <high | medium | low, and why>

Write it only from a kept round whose held-out gain exceeded the noise floor. inconclusive, reverted, and overfit rows are not evidence a mutation transfers, and a RESTRUCTURE that changed several things at once supports only the narrower claim the rounds isolate. If no round qualifies, append the heading with the single line NO_ENTRY. Aim between too generic ("be clearer") and too specific to fire again ("add the severity-label example"). Never copy a test input, held-out input, or failing output into an entry; the Held-out Rule applies here too.

When a later run tries a lesson, log the round's description with the lesson's heading, so a lesson that keeps reverting can be found and deleted.

Applying Results

When the user is satisfied (or after convergence), they merge the optimized skill with git checkout main && git merge autoresearch/<skill-name>. The original skill on main was never modified during optimization. The branch contains the full history of every mutation that was kept.

Reporting the result

Every number that leaves this loop names the conditions that produced it: the pinned model, sample count, and whether the comparison was against the previous variant or against the control arm. A score without those is not reproducible and should not be quoted.

Lead with the held-out score against the baseline held-out score, then the training score. A held-out gain no larger than the noise floor is "no measurable change", whatever the training score did.

Report only deltas between arms that were actually run. Never extrapolate a saving against a counterfactual — "this skill saved N tokens on the work you did today" is unknowable, because the run without the skill never happened. Where an estimate is genuinely useful, label it est. and say what it was extrapolated from.

Do not compare results produced with different test inputs, checklists, models, or sample counts. That includes comparing against a published number from another project: unless the cases and the model match, the comparison measures the harness, not the skill.