/eval-run — execute the eval, honestly

/eval-spec writes the target before the code exists; /audit eval reviews results after; /ship's gate 3 compares a headline to a target. All three assumed a runner nobody produced. This skill is that runner — the step that turns "the eval is the spec" from a claim into a loop.

Running an eval costs money and time when the system under test calls a model API. State the case count and that cost before executing, then run — the user asked for a run, so this is not a permission request (conduct rule 5), it is a disclosure.

Adjacent skills: /eval-spec (writes the spec and golden set before any code; /eval-run executes it) · /audit eval (audits the results this produces, including recomputing the headline) · /ship (gate 3 compares this headline against the spec's target).

Run before the skill's steps — per invocation, not per session (4.36); failures degrade to markdown:

link="$(readlink "$HOME/.claude/skills/health" 2>/dev/null || true)"   # empty = not symlinked
pack="$(dirname "$(dirname "$link")")"   # NEVER trust this unless $link was non-empty
if [ "${link#/}" != "$link" ] && [ -x "$pack/bin/acstack-config" ] && ! "$pack/bin/acstack-config" runtime | grep -q '=off'; then
  "$pack/bin/acstack-config" || true          # resolved keys, with sources
  "$pack/bin/acstack-update-check" || true    # ≤1 fetch/day; silent ONLY if already checked today
  "$pack/bin/acstack-recall" || true          # LEARNINGS.md + bug-class names, capped 3KB
else
  echo "runtime off — proceeding without recall/update-check"
fi

Operating principles

• Be direct. Push back in writing when the plan or the user is wrong. No sycophancy.
• Never delete a decision. Supersede it: ~~old~~ → **Verdict (YYYY-MM-DD):** new call — reason.
• Never fix, tune, or delete a test or eval case to raise a score. Log the miss honestly and leave the case unchanged.
• Name exact things: regex patterns, function signatures, model names, before → after numbers. Never "fixed bugs".
• Attribution: follow the project's attribution setting (default none) — no AI-tool mentions in generated docs, no attribution trailers in commits or PRs. Commit with explicit -m/-F messages only.
• Config: read .claude/acstack.md at the project root (fall back to ~/.claude/acstack.md) before acting. ## Settings keys override pack defaults; a ## <skill-name> section overrides both. Unknown keys and sections are ignored.
• Docs: BRIEF.md (frozen seed) / PLAN.md (living plan) / JOURNAL.md (rolling journal). If the repo uses legacy names (PLANNING_PROMPT.md / PLANNING.md / STATUS.md), use those instead — never create both.
• Recall: if LEARNINGS.md exists at the project root, read it before starting.
• Conduct: follow the acstack-conduct block in this repo's AGENTS.md — the word is the mode; the user sets the pace.
• Hackathon lane: if the project's AGENTS.md carries the acstack:hackathon-lane block, only /do changes the repository during the event. Any other skill that would write a tracked file, commit or push says what it would have done and stops; a change that is not a task goes through the lane's operator route.

One document set. Resolve exactly ONE BRIEF/PLAN/JOURNAL set and name its path in the report's scope line. If more than one candidate set exists — a monorepo, nested products, an apps/* tree each with its own docs — list the candidates and STOP. Never pick one silently: a confident answer about the wrong product is worse than no answer (conduct rule 8).

The sequence

1. Locate the eval. eval/spec.md and eval/golden.jsonl (or the directory named in the argument). Either missing → say WHICH is missing, point at /eval-spec, and stop. Never scaffold a golden set here; writing the target is a deliberate act that belongs to /eval-spec, and inventing cases at run time is the never-inflate rule's exact failure mode.
2. Prefer the runner that exists. If spec.md's run command names a runner present in the repo, run THAT and skip step 3 — an existing eval artifact is never regenerated (/eval-spec's step-0 rule).

3. Scaffold only when there is none. Detect the stack: package.json → eval/run.mjs; pyproject.toml or requirements.txt → eval/run.py; neither, or both → stop and ask. Never guess a stack. Build from references/runner-template.md.

   A run command in the spec settles the stack — it is not a guess. If spec.md names eval/run.py, the language is already decided by the committed spec, and stopping to ask which stack to use is a deadlock over a question already answered. Take the extension from the spec's run command and say that is where it came from. Only a spec that names no runner leaves the question genuinely open.

   The runner needs one fact the spec often lacks: how to invoke the system under test — a CLI command, an HTTP endpoint, or a function import. If spec.md does not say, ask, then record the answer in spec.md's run section as a dated addition (that file is living; the golden set is not).

   Unattended, the same settled-not-guessed rule applies: when nobody can answer, a committed document that states the exact invocation — a PLAN exit criterion, an acceptance line — settles the question; record it as the dated addition with the source named. Only when no committed document states the invocation does an unattended run stop and say what is missing. Never derive it from reading the subject's code: the code cannot distinguish its intended interface from an accident, so that is a guess wearing evidence.

   The invocation carries isolation flags and a pinned subject model, or it is not measuring the subject. Without isolation the operator's user-level skills, hooks, memory and CLAUDE.md discovery, and output styles all ride into the run — including into the BASELINE arm of an A/B, where they can make a candidate look better or worse than it is. Without a pin the run silently uses whatever the operator or the CLI release defaults to, so two runs are not comparable. Flags per stack are in references/runner-template.md; state what each drops AND what still leaks. An unpinned model STOPS the run — see the template's SUBJECT_MODEL guard.

   Then the non-regression gate. references/regression-gate.py compares the new results file against the LAST COMMITTED one, per category, and blocks when any category falls. The headline cannot see this: a change that lifts the overall percentage while breaking every refusal case passes both /ship's target gate and the spec's category minimums, since minimums constrain the golden set's composition, not one run against another. No committed baseline PASSES and says so — never silently, because an absent baseline read as a clean pass is the false confidence the gate exists to remove.

4. Run it, then read the file. The runner writes eval/results/<UTC-timestamp>.jsonl. The headline is computed from that file — overall %, per-category %, refusal % — and never assembled by hand from what the run appeared to do. If the two ever disagree, the file wins and the discrepancy is the finding.
5. Report (shape below). Propose PLAN edits; apply none.

Grading rules

Per case, from its grade_rule, aligned with the canonical grader rules in ../eval-spec/references/grader-rules.md and with /audit eval:

• exact — normalized equality: trim, collapse whitespace, NFKC, and fold case — unless the case carries case_sensitive: true, set when the output's shape is itself part of the contract; then case is kept and a right answer in the wrong case fails on shape. Same answer, not same keystrokes.
• concept — the expected concept is present. The scaffold implements this as normalized substring containment, which is the FLOOR, not the ideal: it is literal enough to produce brittleness. When a case fails here and the answer was right, widen the grader — never touch the case (/audit eval's "grader brittleness" bucket).
• numeric-tolerance:<x> — parse both sides, compare within ±x absolute, or ±x% relative to the expected value when the suffix is % (both forms are documented by /eval-spec's template).
• rubric:<name> — not machine-gradeable. The runner records the answer, marks the case needs-rubric-review, and excludes it from the headline while naming it. A human or a judge model scores the rubric's named dimensions; a rubric with unnamed dimensions is a spec defect, not a pass.

Denominator discipline, because a headline is only as honest as what it counts: cases marked "status": "needs-data" are skipped and reported as skipped, never counted as passes; "status": "superseded" cases are excluded entirely; rubric: cases are excluded from the headline because no machine graded them, and are reported for review; acceptable_failure applies only when the case carries a written reason (in either shape it is written — a bool with a sibling reason, or an object), never to an ungraded case, and every application is listed.

Every exclusion is named in the report. A case that leaves the denominator quietly moves no percentage at all, so it cannot be caught by looking at the number — which makes silent exclusion the most dangerous of the false-pass family.

Hard rules

• Never edit golden.jsonl to raise a score. A genuinely wrong case is superseded through /eval-spec, with a new id and a written reason. Editing a case to pass is the one thing this pack refuses outright.
• Never hand-transcribe the headline. It comes from the results file, every time.
• A run that errors part-way reports what completed and what did not — a partial run is never presented as a full score.
• A subject that cannot be invoked yields NO SCORE, not 0%. When the system under test needs a credential or endpoint the environment lacks, record honest per-case errors in the results file, report the headline as measuring the environment rather than the subject, and name what is missing in Scope. Never set, invent, or mock the credential to make the run go — a mocked subject scores the mock.
• The results file is committed with the run: an eval score whose evidence is not in the repo is an assertion.

Report shape

Verdict first — <headline>% vs target <n>% plus MEETS TARGET or BELOW TARGET by <x>; when nothing was graded, the verdict is instead NO SCORE — <what is missing> (per the hard rule above — a zero-graded run has no headline to compare) — then:

• per-category table (category | cases | passed | % | target)
• failures, grouped by the /audit eval buckets (prompt issue, grader brittleness, provider flake, data issue, parser issue, genuinely ambiguous) — classification proposed, since confirming it is /audit's job
• acceptable_failure applications, each with its reason
• skipped cases and why
• the results file path, and the exact command that produced it
• Scope: what was NOT run (needs-data cases, categories with no cases, an unreachable system under test)

Positive control

A golden set containing one deliberately failing case MUST produce a sub-100% headline. A runner that reports 100% by construction — because it counts skips as passes, swallows errors, or compares nothing — is the false-pass class this pack exists to catch, and it is indetectable from a green report. fixtures/eval-run/ seeds exactly that case.