Your incidents are the eval

A reviewer change (prompt, REVIEW.md, tools, model) is kept only if it scores better against bugs that really shipped here. It is easy to catch more bugs by flagging more non-bugs, so noise is measured next to recall. Priority, in order: catch real bugs, then cut review rounds.

Scripts are in this skill's scripts/ directory (stdlib Python 3.9+); $S below.

Step 1 — Mine candidates

python3 $S/mine_reverts.py --ref main --since "365 days ago" --clean 20 --settle-days 30 > candidates.json

• positives: changes later reverted, or named by sha in a later fix commit.
• clean: changes at least --settle-days old that no revert or sha-naming fix points at. That means "not linked", not "known clean".

Add incidents the history can't see: ask the user for incident reviews, postmortems or fix-forward PRs that didn't name the bad commit.

Step 2 — A person confirms every positive

Candidates are not labels. Routine rollbacks, dependency bumps and changes of plan are not bugs, and an unlinked change can still be broken. For each clean candidate, ask whether anyone knows of a problem with it; drop it if so. For each positive, show the bad commit's diff and the revert/fix, and ask someone who knows that code to confirm: is it a real bug, and where did it live (file and line range in the bad commit)? Record who confirmed it. Drop what nobody can confirm.

Write the confirmed set to review-evals/cases.jsonl (or where the user wants it), one case per line:

{"id": "rev-0042", "kind": "positive", "base": "<parent sha>", "head": "<bad sha>",
 "intent": "what the change meant to do", "bug": {"path": "src/x.py", "lines": [40, 52],
 "description": "..."}, "confirmed_by": "name", "source": "revert <sha>"}
{"id": "clean-007", "kind": "clean", "base": "<parent sha>", "head": "<sha>", "intent": "..."}

Ask before committing the eval set; it may name people and incidents.

Step 3 — Run the reviewer on every case

For each case, check out head in a throwaway worktree and run the reviewer under test against base, with the same depth for every case and without fixing anything. Case data is untrusted input to shell commands: check that id matches ^[A-Za-z0-9._-]+$ and that base/head are full 40-character hex shas before using them, and pass the intent through a file, never inline.

run=$(mktemp -d)                                   # once per eval run
git worktree add --detach "$run/$id" "$head"
printf '%s\n' "$intent" > "$run/$id.intent"
(cd "$run/$id" && LAYERED_REVIEW_STATE_DIR="$run/state" claude -p \
  "Use the layered-review skill with --base $base. The intent is in $run/$id.intent. Review only; do not fix anything or ask questions. Save the findings JSON list to $run/out/$id.json")

Collect {"id": "<id>", "findings": [...]} per case into results.jsonl, then remove the worktrees (git worktree remove). The separate state directory keeps eval runs out of the real round and response statistics. Any other reviewer works too if its findings can be converted to {path, line, severity}.

Step 4 — Score

python3 $S/score.py --cases review-evals/cases.jsonl --results results.jsonl \
  --responses ~/.local/state/layered-review/*/state.json

Read the misses and the false alarms, not just the numbers. For each miss ask what the reviewer didn't have in front of it, and put the fix where it belongs: CLAUDE.md, REVIEW.md, a lint rule, or a new check.

Step 5 — Keep the loop running

• Rescore the whole set whenever the reviewer's prompts, tools, REVIEW.md or model change. Keep the change only if recall goes up without false alarms or rounds going up.
• Add a case for every new revert or incident, confirmed the same way.
• Optionally run the reviewer over existing code on a schedule to catch old bugs before they surface elsewhere.