self-improve — improve the improver

The maintainer loop (/hmd:maintain-check) fixes issues. This skill makes the maintainer get better at fixing issues over time. It is karpathy/autoresearch applied to Heimdall's own routing and planning — see docs/analysis/autoresearch-distilled.md for the distilled mechanics this is built on.

autoresearch's one non-negotiable rule, imported wholesale: an improvement persists only with a measured delta over a baseline — never because it seemed better. The mechanical parts (aggregation, hypothesis generation, override apply/rollback, experiment logging) live in bin/heimdall-self-improve; your job is the judgement of which hypothesis is worth an experiment.

Where this DEPARTS from autoresearch, on purpose. autoresearch runs one 5-minute training run per experiment and compares val_bpb — a low-variance continuous scalar, where n=1 is a usable measurement. It has no sample-size rule at all. Heimdall's scalar is a proportion (first-try acceptance pass-rate), which is far noisier: at 3 samples, a routing variant that is genuinely no better still clears a 0.10 delta about 73% of the time. So the sample-size floor is Heimdall's own addition, not something inherited — and it is load-bearing. Do not describe it as autoresearch's discipline; autoresearch simply never needed one.

When this runs

• On demand: the user runs /hmd:self-improve or asks Heimdall to improve/tune itself. One experiment per invocation.
• Between maintainer fix batches. /hmd:maintain-check documents this as a step you run yourself. There is no automatic every-Nth-cycle trigger — self_improve is not a key heimdall-state init creates, nothing in bin/ hooks/ sentinels/ modules/ reads it, and nothing counts cycles toward it. See Not implemented: the automatic every-Nth-cycle trigger in commands/maintain-check.md.
• Overnight (/dream): the /dream command wraps this loop for an off-hours run and leaves a morning report (.planning/dream/YYYY-MM-DD.md) — see commands/dream.md. It reuses this exact keep-gate (surface-first; only measured, reversible wins persist).

Do NOT run it mid-fix. It is a deliberate step back between batches of work, exactly like autoresearch's overnight step-back between training runs.

The loop (autoresearch's five invariants, ported)

collect  ->  hypothesize  ->  experiment (bounded)  ->  evaluate (measured)  ->  keep | discard
   ^                                                                                    |
   +------------------------ the experiment LOG is the accumulated knowledge -----------+

1. Collect evidence

The comparable scalar is first-attempt AC pass-rate per (task_type, model) (plus avg retries and wall-secs) — Heimdall's val_bpb. Read it from .planning/metrics.jsonl (task-outcome records) and the queue dead/done stats:

heimdall-self-improve collect --repo .          # the aggregated scalar per (task_type, model)

Evidence does not appear on its own — something has to emit it. The producer is bin/heimdall-metric, called by the orchestrator after every completed task (see agents/heimdall.md → Pattern Learning). If nobody calls it, collect returns zero records and /dream honestly reports "nothing to suggest" — which is the correct behavior, not a bug to route around. Check the corpus before blaming the loop:

heimdall-metric where --repo .                                   # the corpus path
grep -c '"metric":"task"' "$(heimdall-metric where --repo .)"    # how much evidence exists

Task-outcome record shape (one JSON line; non-task records are ignored):

{"ts":"...","metric":"task","schema":1,"task_type":"lint","model":"sonnet","effort":"default",
 "outcome":"fail","final":"pass","retries":1,"escalated_to":"opus","wall_secs":42,
 "source":"orchestrator","session":"..."}

outcome is first-try acceptance at that tier — the scalar routing is graded on. final is the eventual verdict, kept separate so "we got there in the end" cannot launder a bad routing decision. One record = one routing observation, so an escalating task emits one row per tier it touched.

Queue dead/done stats (optional) live at .planning/queue-stats.json: {"dead":[{"reason":"lint-timeout","task_type":"lint","count":4}], "done":[...]}.

2. Hypothesize

heimdall-self-improve hypotheses --repo . --min-samples 20

Emits testable candidates, each with its measured baseline evidence:

• escalate (hyp-esc-<type>-<model>) — a task_type failing on its current tier → bump a tier. ("If haiku tasks fail > 30% → bump to sonnet." — agents/heimdall.md.)
• cheapen (hyp-cheap-<type>-<model>) — a task_type passing flawlessly with zero retries → try the cheaper tier and keep only if the pass-rate holds. (cost savings, same outcome.)
• precheck (hyp-precheck-<reason>) — a recurring dead-task reason cluster → suggests a pre-check or a new .planning/skills/*.md pattern (surfaced for a human, not auto-A/B'd).

The --min-samples gate is the invariant: no hypothesis without enough evidence. Pass 20 or more. The CLI's own default is 3, which is a historical default and far too low for a proportion — /dream hard-clamps to a floor of 20 (MIN_SAMPLES_FLOOR in bin/heimdall-dream) and will not let a caller argue it down. When you drive the CLI by hand, you are the floor: pass it.

Pick the ONE highest-value hypothesis. Prefer an escalate on a high-traffic failing type; a cheapen only when the baseline is genuinely flawless.

3. Experiment (bounded)

Apply the variant as a routing-override. It is bounded — the next matching tasks get the variant, tracked — exactly like autoresearch's fixed 5-min budget makes runs comparable:

heimdall-self-improve experiment start --hypothesis hyp-esc-lint-sonnet \
    --min-samples 20 --min-delta 0.15 --repo .

This writes .planning/routing-overrides.json, snapshots the baseline and the prior override (for exact reversibility), and appends an OPEN record to .planning/experiments.jsonl. Let the next maintainer cycles run the variant.

Nothing consumes routing-overrides.json yet. heimdall-self-improve is its only reader and writer; no planner, agent or spawn path looks at it, so a validated override does not change which model actually gets spawned. Until that wiring exists, treat an override as a recorded, measured finding — and apply it by hand if you want it to take effect.

4. Evaluate (measured — the falsifier)

The evaluator is the recorded task outcomes from the real gate — the routing change being tested cannot grade its own homework (autoresearch invariant 4). After enough matching tasks have run:

heimdall-self-improve experiment evaluate --id <exp-id> --repo .

• KEEP iff variant_samples >= min_samples AND delta_pass_rate >= min_delta. The override is marked validated and stays.
• DISCARD otherwise (too few samples, or no measured improvement) — the override is rolled back to its prior value and the experiment is logged discarded. Never persist on vibes.

5. The log is the memory

.planning/experiments.jsonl is the append-only record you resume from — every hypothesis, its baseline, its measured delta, its verdict. heimdall-self-improve status --repo . folds it to the current picture (active overrides, validated wins, open experiments).

What a validated improvement becomes

• Routing: a validated entry in .planning/routing-overrides.json — a recorded measured win, not yet a live routing change (nothing reads that file; see step 3).
• Pattern: a precheck cluster → a new .planning/skills/*.md reusable pattern (write it, cite the dead-reason cluster + count as evidence).
• Capability gap the repo should own: if the fix belongs in the codebase (a flaky test, a missing pre-commit hook), file a GitHub ISSUE via heimdall-feedback describing the measured cluster — do not silently paper over it with a routing tweak.

Discipline (do not skip)

• One experiment per invocation. Compounding many at once destroys comparability.
• Never mark an improvement kept without a fresh evaluate verdict in this run — quote its result block as evidence.
• A cheapen experiment that even ties the baseline on too few samples is DISCARDED, not kept — the burden of proof is on the change.
• Archive failed hypotheses by leaving them in the log; do not re-run an identical experiment that was already discarded without new evidence.

Reference

• Producer: bin/heimdall-metric (emits the metric:"task" records everything below reads).
• CLI: bin/heimdall-self-improve (stdlib python3; -h for full usage).
• Distilled source mechanics: docs/analysis/autoresearch-distilled.md.
• The SONA-inspired feedback loop this formalizes: agents/heimdall.md → Pattern Learning.
• Acceptance: test/heimdall-self-improve.test.sh (hermetic; proves the falsifier), test/heimdall-metric.test.sh (proves the producer, the floor, and both falsifiability directions: recommends against an obviously-worse tier, declines on a null corpus).