evals

This skill covers the two jobs the eval harness does not do for you: turning an observed failure into a case, and reading a finished run. Everything with a correct answer — is this a regression? — belongs to interlock evals triage, not to you. Your job is evidence, grader choice, and explanation.

1. Authoring a case — only from an observed failure

A case regression-tests a reproduced failure: a model read correct instructions and did the wrong thing anyway. Before you write anything, require the evidence:

• an archived proposal (openspec/changes/archive/*/proposal.md ## Why),
• a CHANGELOG.md entry,
• a run artifact or a transcript,
• a captured skeleton — interlock evals capture --run <runId> [--task <id>] --out <dir> turns a recorded ship-run trajectory into a case draft whose provenance names the trajectory and the events it was derived from, or
• a file:line span in the prompt or skill under test that shows the surface the model talked past.

A captured skeleton is a draft, not a finished case. It is admissible evidence because it is a run artifact and a reader can follow its citation back to the trajectory. But every value capture derived rather than read carries a CONFIRM marker — the grader pattern when no value could be quoted, and the prompt when the trajectory recorded none. Resolve every CONFIRM marker against the cited trajectory, or ask the user, before authoring the case. Never author a case that still carries one, and never move a skeleton into evals/ unresolved: an unconfirmed pattern is a regression test for a failure nobody has shown happened.

If no observed failure is offered, say so and stop. Do not author a speculative case. A suite of hypotheticals teaches the team to ignore it, which is the one thing an advisory gate cannot survive. A case that only duplicates an existing node:test assertion is refused for the same reason — the unit suite already covers it deterministically.

Record the citation in the case (provenance: in case.yaml, plus a comment explaining what went wrong). The structural gate test/evals.test.mjs fails a case with no provenance.

The case files

Every case pins schema_version and lives in its own directory under evals/ — never under skills/, agents/, commands/ or workflows/, which would shadow a component. Follow evals/handoff-status-enum/ as the format exemplar; it was written to establish these conventions. Where a case needs a fixture workspace, use case.yaml with context.scaffold_script and run with --scaffold; otherwise the same case.yaml shape without it.

2. Choosing a grader — cheapest that expresses the assertion

The harness has six grader types. Four are free and deterministic — regex, tool_used, tool_order, file_exists — and two call a judge — llm, baseline.

Pick the cheapest grader that expresses the assertion. If the assertion is set-membership, tool-invocation, file-existence, or ordering shaped, it is deterministic and a judged grader is the wrong tool — more expensive, and prone to run-to-run variance triage will (correctly) refuse to call a regression. The strongest seed case, handoff-status-enum, needs no judge at all: the status is in a fixed set, so a regex decides it.

When you do reach for a judged grader, state in the grader body which part of the assertion could not be expressed deterministically. If you cannot name that part, the assertion was deterministic and you chose the wrong grader.

A case you intend to run per-pull-request tag smoke, and a smoke case uses only deterministic graders — the gate enforces this. A grader that asserts a skill fired (tool_used: Skill) is marked arm: with-only: it is a plugin-fired indicator, reported against the baseline arm, not part of the score.

3. Reading a run — triage owns the verdict

Given a finished results file, get the classification from interlock evals triage --results <file>, do not derive it yourself. Its exit code is the verdict: 0 pass, 1 regression, 2 no signal, 3 configuration. This is deliberate — the one gate whose rule a model could re-argue on each run would be exactly the wrong gate to ship in a suite built to catch models re-arguing rules.

Your contribution is explanation, not classification:

• When triage reports a regression, explain which recent change to the surface under test plausibly caused it — the prompt sentence, the skill edit, the contract change. Do not reclassify it as variance because it looks flaky; triage already separated a single judged dip (variance) from a judged majority (regression).
• When triage reports variance, report it as variance and say what would confirm it (further runs failing the same judged grader).
• When triage reports no signal — a partial run, an interrupted run, a rejected credential — report that evals could not run. Do not present any case as passing or failing.

Never restate, override, or soften a verdict triage gave you.

Read a transcript before you explain a judged case

For every case whose score depends on a judged grader, read at least one transcript for that case before you write your explanation of it. A score is the outcome; the trace is the only place the cause is visible. The failure this closes is a model that satisfied a judge by narrowing the task rather than by doing it — the score moves, every other step in the loop sees a number, and nobody sees why.

Reading is for explanation and never for reclassification. If the transcript suggests a different classification than triage produced, report what you saw in the trace and leave the verdict alone — the rule above still holds without exception.

A run whose graders are all deterministic needs no transcript: the tool trace the grader already read is the evidence.

Your report carries a transcripts read: line, always present:

transcripts read: <case>/<id>, <case>/<id>
transcripts read: none
transcripts read: unavailable, because <what stopped you reading one>

Use the none form when you read no transcript — never omit the line, because a reader cannot otherwise tell an unexamined result from an examined one. Use the unavailable, because … form when the run produced no readable transcript, and in that case do not present your explanation as transcript-grounded. Degradation is spoken.

4. A failing case is a report, never a prompt edit

When a case fails, your output is a report: the case, the grader that failed, and the sentence in the surface under test that plausibly caused it. Do not edit the skill, shared contract, agent definition, or prompt under test to make a case pass. Changing the surface under test is a separate, explicit decision the user makes — silently editing a prompt to turn a case green is exactly the "edit the test until it passes" failure the suite exists to prevent, one layer up.

5. When the harness is gated

The host command claude plugin eval prints that it is currently in early access and does nothing until CLAUDE_CODE_WALNUT_SPIRE=1 is set in the environment — never in committed .claude/settings.json, which would produce a suite that looks configured and does not run.

When the harness refuses because early access is not enabled, report the prerequisite and where to set it (the environment, or the CI job's env: block). Do not report an empty run as a passing suite. Authoring a case and triaging an existing results file both work without the harness enabled — only running a suite needs it.