Add an eval case

An eval protects one behavior of an LLM-driven feature. This skill decides where that protection lives and writes the strongest possible version of it. If the repo has a scoped copy of this skill with concrete fixture names, that copy wins; this is the general procedure.

Step 0: Check the harness

The per-case procedure below needs four capabilities: deterministic world seeding, a one-turn runner, a tool-call recorder, and a judge helper. If the repo has all four, continue; if any is missing, build it first by following HARNESS.md.

Step 1: Place the behavior

Walk down; stop at the first yes.

1. Pure code can produce the failure -> ordinary unit test. Stop; this skill does not apply.
2. Needs routes/auth/DB but not a model's decision -> integration test with the LLM stubbed. Stop.
3. The failure is a wrong action or wrong prose -> continue below; this is an eval case.

Step 2: Seed known ground truth

Seed the world so every fact the test relies on is known in advance. An eval without known ground truth cannot hard-assert anything - it can only judge, which is the weakest possible test.

Step 3: Run one real agent turn

Run the user message through a real agent turn against the seeded world. One paid model call per scenario; stack all checks against it.

Step 4: Assert everything assertable, cheapest first

Order matters - a red must point at one kind of defect, and later checks are meaningless if earlier ones fail:

1. Trajectory: which tools were called, with what args - and which were not. assert not called(...) is as load-bearing as assert called(...).
2. State: query the store for rows the tools should have created.
3. Strings: facts the seeded world makes checkable (the seeded total was 30, so assert "30" in reply).

Step 5: Judge only the residue

Only what survives every hard assertion goes to a judge, one binary criterion per case:

• Meaning ("must not imply the edit was applied") -> fail-closed: ambiguity is failure.
• Voice ("reads like the strict coach persona") -> fail-open: only clear violations fail.

Never delegate an assertable fact to the judge - if half a criterion could be an assert, split it out. Phrase the criterion so a yes/no answer exists, and name the test after it: test_exhausted_proposes_edit_and_never_claims_applied is its own criterion.

Step 6: Verify both ways

Run your case and confirm it passes. Then break the behavior (or the criterion) and confirm it fails for the stated reason - a judge case that cannot go red proves nothing.

Red-case triage (when an existing eval fails)

• Red once after a change: the change regressed the behavior. Fix the prompt/code, not the test.
• Red chronically across reasonable prompts: wrong owner - move the behavior into code and replace the eval with a cheap assertion.
• Flaky on an unchanged prompt: criterion not binary enough - fix the rubric to reduce variance, never to reduce standards.