Agent Eval

Turns "does this actually work" into a repeatable, evidence-based answer instead of a gut feeling.

How this works

1. Figure out what "good" means first. Before writing any eval, get a concrete definition of success from the user, or infer it from context and confirm it back to them. What does a correct/good output look like? What does a clearly bad one look like? Is there a reference answer, or is this judgment-based?
2. Pick the eval type — don't default to one without considering the fit:
   • Reference-based: there's a known correct answer (exact or fuzzy/semantic match). Cheapest and most reliable, but only works when "correct" is well-defined.
   • Rubric-based (LLM-as-judge): quality is graded against explicit criteria (e.g. "factually grounded," "follows the required format," "appropriately concise"). Use ${CLAUDE_PLUGIN_ROOT}/references/llm-judge-prompt.md as the starting template — don't write a judge prompt from scratch each time.
   • Pairwise comparison: judging which of two outputs is better, not scoring each in isolation. More reliable than absolute scoring for subjective quality, but watch for position bias — always run both orderings and average. Use ${CLAUDE_PLUGIN_ROOT}/scripts/run_pairwise.py for this — it runs both orderings itself and reconciles them (see ${CLAUDE_PLUGIN_ROOT}/references/pairwise-comparison.md), rather than leaving "remember to run it twice" as a step to repeat by hand each time.
   • Programmatic/structural: format compliance, schema validation, code that actually executes, tool calls matching an expected sequence. Cheapest and most deterministic when applicable — prefer this over LLM-as-judge whenever the criterion is mechanically checkable.
   • Trajectory evaluation (agents specifically): don't just grade the final output — check whether the agent picked the right tool, passed the right arguments, and used a reasonable sequence of steps. A correct final answer reached via a broken process is still a signal of an unreliable agent. Use ${CLAUDE_PLUGIN_ROOT}/references/trajectory-eval.md for the case shape, the judge-rubric addendum, and the flattening convention — don't invent one from scratch each time.
   • Adversarial: the correct outcome is refusal, hedging, or graceful degradation — the question is "did the agent handle this appropriately?" not "was the answer correct." Never blend adversarial scores with correctness scores; run them as a separate eval set, in their own results file. category should be the specific failure mode under test (prompt-injection, jailbreak, over-refusal, ...), not a single flat adversarial bucket — that's what lets score_eval.py's per-category breakdown show pass rate per attack type, not just one undifferentiated adversarial score. See skills/agent-redteam/examples/adversarial_results.jsonl for the real convention, and skills/agent-redteam/ for a dedicated case-generation skill.
   • Multi-turn: not a distinct scoring type so much as a case-shape variant of the ones above — use when the conversation leading up to a response is itself part of what makes it right or wrong (a frame established earlier, an incremental escalation), not just the final message in isolation. Use ${CLAUDE_PLUGIN_ROOT}/references/multi-turn-eval.md for the case shape (a turns field, rendered into the judge prompt as {transcript} by ${CLAUDE_PLUGIN_ROOT}/scripts/run_judge.py) — don't collapse a real conversation into one flattened string.
3. Build a small, reusable eval set, not a one-off. Even 10-20 representative cases beats eyeballing a handful of outputs. Include a mix of: clear-pass cases, known-hard edge cases, and at least a few cases the current system is expected to fail — a sanity check that the eval can actually detect failure, not just confirm success.

4. Score it. For rubric/LLM-as-judge evals, use the judge prompt template and request structured JSON output (a score plus a one-line rationale per criterion) — never a vibe-based pass/fail. For programmatic evals, write the check directly.

   For rubric/LLM-as-judge evals specifically, use ${CLAUDE_PLUGIN_ROOT}/scripts/run_judge.py rather than calling the judge and flattening its response by hand each time — it fills the template, calls the judge, parses the structured JSON, and writes already-flattened rows in the exact schema score_eval.py reads:

   export ANTHROPIC_API_KEY=...
   python ${CLAUDE_PLUGIN_ROOT}/scripts/run_judge.py cases.jsonl --template ${CLAUDE_PLUGIN_ROOT}/references/llm-judge-prompt.md --out results.jsonl --category accuracy
   
   # Or judge with Gemini instead of Claude — same output, same schema:
   export GEMINI_API_KEY=...
   python ${CLAUDE_PLUGIN_ROOT}/scripts/run_judge.py cases.jsonl --template ${CLAUDE_PLUGIN_ROOT}/references/llm-judge-prompt.md --out results.jsonl --provider gemini

   --provider (default: anthropic; gemini also supported) picks which judge API gets called — ${CLAUDE_PLUGIN_ROOT}/scripts/run_pairwise.py takes the same flag. --model defaults to the chosen provider's own default model if not given. See call_judge()'s own docstring in run_judge.py for one honest caveat: Gemini's token-usage field (which cost_usd depends on) isn't independently live-verified in this repo the way Anthropic's is, so treat cost_usd from a Gemini-judged run as lower-confidence than from an Anthropic one.

   Cases are graded concurrently (--concurrency, default 5) rather than one judge call at a time — bounded, not unlimited, since this is a real (possibly rate-limited) external API. Raise it for a large case set against a generous rate limit; lower it if you start seeing failed cases from 429s. ${CLAUDE_PLUGIN_ROOT}/scripts/run_pairwise.py applies the same concurrency to its own two calls per case, unconditionally (position-bias reconciliation needs both orderings regardless of case-set size, so there's no separate flag for it there).

   cases.jsonl is one row per case (id, plus whatever fields the template's {placeholder} tokens need — typically input/output, or input/trajectory/final_output for a trajectory case). It's generic over criterion names, so it handles both the plain rubric shape and ${CLAUDE_PLUGIN_ROOT}/references/trajectory-eval.md's shape without any mode flag. A per-case failure (judge call error, unparseable response, or a malformed case that fails template-filling) is reported to stderr and that case is skipped — never given a fabricated score. cost_usd is only included when both --input-price-per-mtok/--output-price-per-mtok are given, computed from the API's own real token counts.

   Before any judge call, both scripts run a preflight check: every case is verified against the template's own {placeholder} tokens, and against each other for duplicate ids. A systemic mistake — case file built against the wrong template, a field typo'd across the whole set — is reported as one upfront summary and aborts, instead of surfacing as a string of individually-quiet per-case skips after budget has already been spent on whichever cases happened to still pass. Pass --skip-invalid to grade just the valid subset instead of aborting.

   The judge prompt returns one nested object per case (per-criterion scores plus an overall_score) — that's a different shape from what ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py reads. If scoring by some other means than run_judge.py (a different judge model/provider, a notebook), flatten each case before saving, to this schema (one JSON object per line):

   {"id": "case_001", "score": 0.83, "category": "accuracy", "rationale": "...", "cost_usd": 0.003, "latency_ms": 1240}

   • id — a stable identifier for the eval case, assigned when you build the eval set (not produced by the judge).
   • score — the judge's overall_score for LLM-as-judge evals, or 1.0/0.0 for a programmatic pass/fail check.
   • category — the criterion group or failure mode you're tracking (e.g. accuracy, format, tool_use), assigned by you, not read from the judge's per-criterion keys — this is what score_eval.py breaks results down by.
   • rationale — a one-line reason for the score. For a multi-criterion judge response, use the rationale from the lowest-scoring criterion, since that's the one explaining the failure.
   • cost_usd (optional) — API cost for generating the output under test.
   • latency_ms (optional) — wall-clock time to generate the output, in milliseconds.

   Save the flattened lines to a JSON/JSONL file, not just a summary — the failure examples are what make this actionable.

5. Calibrate the judge periodically. Every 25-50 judge calls (or whenever you revise the judge prompt), hand-score 5-10 cases yourself and compare to the judge's scores. Use ${CLAUDE_PLUGIN_ROOT}/scripts/calibrate_judge.py rather than eyeballing the delta — it computes the mean delta, flags whether it exceeds the threshold (default 0.2), and can log the result for you:

   python ${CLAUDE_PLUGIN_ROOT}/scripts/calibrate_judge.py judge_results.jsonl human_scores.jsonl --update-log ${CLAUDE_PLUGIN_ROOT}/references/llm-judge-prompt.md

   human_scores.jsonl only needs the 5-10 IDs you actually hand-scored (same JSONL schema as any score_eval.py results file) — it compares whichever IDs appear in both files. See examples/README.md's calibration example for a worked case where this catches a judge fooled by confident, verbose wrong answers. --update-log appends a row to ${CLAUDE_PLUGIN_ROOT}/references/llm-judge-prompt.md's calibration table automatically, replacing the "not yet calibrated" placeholder on the first real run — don't edit that table by hand. Add --action "what you changed" to replace the auto-generated "Action taken" text with your own note; | and newlines in it are escaped, so free text is safe.

   To hand-score blind rather than in a spreadsheet, use ${CLAUDE_PLUGIN_ROOT}/scripts/human_score.py. prepare freezes a shuffled queue (the judge's near-threshold cases first; judge scores, categories and real ids are never shown, and --show-fields refuses id, category, score and blind_id so they cannot be shown by mistake), score is a resumable interactive loop, rescore appends a revision without overwriting your original, and export writes the human_scores.jsonl that calibrate_judge.py reads. prepare needs a judge results file for ordering; if the judge has not run yet, pass a placeholder file with every score 0.5 so no judge information exists before you score (the order is then a plain shuffle). Report export --which first (the first entry for each item) as the calibration figure and report revisions separately, since they are judgment changes. Both rescore and the b key in score count as revisions: --which first ignores them even when b fixed a slip, and export prints a note whenever any score changed after its first entry, so say which of first or latest you reported. A score of exactly 0.0 or 1.0 is read back for confirmation, because a mistyped 0.0 once skewed a real run; --no-confirm-extremes turns that off for scripted input. The tool cannot hide what a reply itself reveals (for example its length, when length is the criterion), so spot-check a few items yourself. If the biggest gaps turn out to be you and the judge reading the rubric differently, fix the rubric and re-test on new items instead of rewriting the judge prompt, and say so in --action.

6. Aggregate and report using ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py. Don't manually tally pass rates — run the script against the results file:

   python ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py results.jsonl
   python ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py results.jsonl --baseline previous_results.jsonl
   python ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py results.jsonl --threshold 0.8
   python ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py results.jsonl --json-out summary.json

   It computes pass rate, mean score (overall and per-category), surfaces the lowest-scoring cases for review, and reports mean cost and latency per category when those fields are present. With --baseline, it flags regressions — cases that passed before and fail now. --threshold sets the score-≥-this-counts-as-a-pass cutoff (default 0.7) used everywhere in the report and every gate below — set it once per eval set rather than eyeballing which scores "feel like" a pass. --json-out writes the same summary summarize() computes (including by_category and any regressions found) as JSON, for a dashboard or a second script to consume instead of scraping the printed report.

   Category grouping is case/whitespace-insensitive — accuracy, Accuracy, and ACCURACY all land in the same row rather than silently fragmenting one category's stats across several rows, a real risk since category is typed per-run, not drawn from a fixed enum anywhere in this pipeline. A category name that's merely similar to another (a likely typo, not an exact match after normalizing) is never auto-merged — that would risk silently combining two genuinely different categories — but score_eval.py prints a warning naming the suspect pair so you can fix the eval set's spelling.

   As a CI gate, add --fail-under and/or --fail-on-regression so the script exits non-zero (failing the build) when quality drops:

   python ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py results.jsonl --fail-under 0.85
   python ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py results.jsonl --baseline eval_set_v2.jsonl --fail-on-regression

   This is what makes an eval a gate rather than a report — the same run that scores your change also blocks it if it regressed. See examples/ for a worked before/after where a change looks like a win on cost, latency, and format but the gate catches three silent accuracy regressions.

   --fail-on-regression alone only catches a case crossing the pass/fail line — a case that drops 1.00 → 0.71 against the default 0.7 threshold stays "passing" and is invisible to it, even though that's a real, large quality decay. Add --regression-min-drop (e.g. 0.2) to also flag any case whose score fell by more than that amount, threshold-crossing or not — both kinds still gate under the same --fail-on-regression flag, just labeled separately in the report ("passed before, failing now" vs. "still passing, but dropped sharply") so you can tell which happened:

   python ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py results.jsonl --baseline previous_results.jsonl --fail-on-regression --regression-min-drop 0.2

   Off by default — existing --fail-on-regression configs keep their exact prior behavior unless this is explicitly set.

   The reverse gap has a gate too: a change that holds accuracy perfectly steady while cost or latency quietly balloons was previously invisible to every flag above. --fail-on-cost-regression/--fail-on-latency-regression (needs --baseline; tolerance via --cost-regression-tolerance/--latency-regression-tolerance, default 20%) catch that directly; --fail-if-mean-cost-above/--fail-if-mean-latency-above set an absolute ceiling with no baseline needed:

   python ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py results.jsonl --baseline previous_results.jsonl --fail-on-cost-regression --fail-on-latency-regression
   python ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py results.jsonl --fail-if-mean-cost-above 0.01 --fail-if-mean-latency-above 2000

   See examples/README.md's cost/latency regression example for a worked case where accuracy is unchanged (same 90% pass rate) but cost/latency both roughly triple, and the gate catches it.

7. Be honest about sample size — and don't just say so, measure it. With under ~20 cases, a 2-3 case swing can look like a large percentage shift. --ci reports a 95% bootstrap confidence interval on pass rate and mean score, and — with --baseline — a paired significance test on both the pass-rate delta and the raw mean-score delta between the two runs:

   python ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py results.jsonl --ci
   python ${CLAUDE_PLUGIN_ROOT}/scripts/score_eval.py results.jsonl --baseline previous_results.jsonl --ci

   --fail-on-significant-regression gates on those tests instead of --fail-on-regression's bare threshold-crossing count — it only fails the build when a 95% CI excludes zero, not whenever any single case wobbles past --threshold (real LLM-judge noise, not necessarily a real regression). Checking both pass-rate and mean-score matters because binarizing to pass/fail throws away magnitude — a case that drops from 0.9 to 0.4 counts exactly the same as one that drops from 0.71 to 0.69 once both are "fail," so a real, consistent quality drop that never crosses --threshold is invisible to the pass-rate signal but not the raw-score one. Still say the plain-language version too ("3/10 passed (30%) — too small a sample to call this a real regression yet") — the CI number backs up the sentence, it doesn't replace it.

   The CI's significance verdict and its width aren't the same claim, and only one of them is reliable at small n. A narrow bootstrap CI at, say, n=3 isn't reassuring — it's often just an artifact of resampling a tiny, low-variance sample, not evidence the estimate is well pinned down. Confirmed empirically: mean CI width barely narrows from n=3 (~0.55 of the full range) to n=8-10 (~0.5), only dropping meaningfully past n=30-50. --ci now prints an explicit ⚠ next to any CI (run-wide or per-category) whose n is below 20 — reusing the same 20-case threshold as the sample-size warning above rather than inventing a second cutoff. The p-value next to it can still be trusted (that's what min_n in the next paragraph is for); the interval's width specifically can't. This warning shipped print-only at first — anything reading --json-out instead of the terminal (a dashboard, a script comparing runs) had no way to reconstruct it short of hardcoding the same threshold itself. Every CI and paired-diff dict now also carries a "ci_width_reliable" boolean in the JSON, derived from the identical comparison, so a machine consumer gets the same signal without duplicating the constant.

   Both of those are still run-wide aggregates, which have their own blind spot: a real regression concentrated in one category can get diluted away against unaffected cases from every other category. --ci (with --baseline) also runs both paired tests once per category — skipping any category under 3 matched cases, too few for a bootstrap to say anything meaningful, and skipping a sole category spanning every matched case (no category field anywhere, or every row sharing one label) since that's testing the identical data the run-wide diffs already cover, not a second opinion — and --fail-on-significant-regression gates on those too.

   Running that many tests together has its own cost: at a plain alpha=0.05 each, the chance that at least one comes back "significant" purely by chance climbs well past 5% as more tests are added — confirmed empirically at ~27.5% on genuinely stable data across 4 categories, not a theoretical worry. Every verdict — run-wide and per-category — is corrected for however many tests actually ran together, bringing that back to ~8%, close to the nominal target, via the Benjamini-Hochberg (FDR) procedure. An earlier version of this used the simpler Bonferroni correction instead; it was replaced after a second, independently confirmed problem: fixing one real regression's data and only varying how many unrelated, entirely stable categories existed alongside it, Bonferroni flipped that same regression from detected to undetected once 3+ unrelated categories were added — a real eval set's detection power silently depending on how many categories it happens to have. Benjamini-Hochberg kept detecting it through 7 unrelated categories in the same test, with the false-positive rate just as controlled. See bootstrap_stats.py and apply_multiple_comparisons_correction()'s docstring in score_eval.py for the confirmed numbers; bonferroni_alpha() is kept as a stricter, still-available alternative.

   A third fix on the same feature, same discipline: regression candidates (diff < 0) and everything else (improvements, and the exact-zero-diff case) are corrected as two separate Benjamini-Hochberg families, not one pooled ranking. Confirmed necessary, not a style choice — pooling them let an unrelated, genuinely significant improvement elsewhere in the same run lower the effective bar for a borderline regression to pass (BH's threshold grows with rank, and low-p-value improvements occupy the earliest ranks ahead of it), even though only regression candidates can ever fire --fail-on-significant-regression. Direct, confound-free check: a fixed regression's significance now stays identical regardless of how many unrelated improvement (or stable) diffs are added alongside it. Improvements still get their own BH-corrected significance for reporting — "SIGNIFICANT IMPROVEMENT" stays a real, corrected claim — just never at the regression family's expense.

   See examples/README.md's statistical-confidence section for a real worked case, including an honest consequence: the same 8-case category regression that used to trip the gate before correction existed (p=0.043, uncorrected) no longer does once 10 simultaneous tests are properly accounted for (corrected alpha=0.005) — the correction working as intended, not a capability regression. A second, stronger worked example there shows the gate still firing correctly once the per-category evidence is actually strong enough to survive correction. Three gates now cover three different questions — --fail-on-regression (any named case flipped), --fail-on-significant-regression's run-wide half (is the topline number moving for a real reason), and its per-category half (is a properly corrected regression hiding inside one slice) — meant to be used together, since none of them subsumes the others.

8. When re-evaluating after a change (new prompt, new model, new tool definition), always run the same eval set as before and diff against the saved baseline. That's what catches regressions — a fresh set of cases each time doesn't.

   When the case set itself changes (new cases added, old ones removed), save it as a new versioned file (eval_set_v2.jsonl, eval_set_v3.jsonl) rather than overwriting. Before reporting a regression, confirm both runs used the identical case set — same filename, same line count, same case IDs. A score drop may just be a case-set change, not a model regression.

Output discipline

• Never let an LLM-as-judge grade its own output unflagged — if the system under test and the judge share a model or prompt, say so as a caveat. Self-grading is a known source of inflated scores.
• Don't report an aggregate score without 2-3 concrete failure examples alongside it. A percentage with no examples isn't actionable.
• If the user hasn't defined success criteria and won't, don't silently invent a rubric and present results as objective — flag that the rubric is your best guess at their intent.
• Keep judge prompts in ${CLAUDE_PLUGIN_ROOT}/references/llm-judge-prompt.md versioned alongside the eval set, not rewritten ad hoc each run. Consistency between runs is what makes before/after comparisons valid.
• Report adversarial pass rates separately from correctness pass rates — never blend the two into a single aggregate.

Files