Agent Evals: Bootstrap, Grader Design, and Maintenance

This skill turns "the agent feels worse" into a measurable, fixable signal. It gives you an operating procedure for building evals for AI agents: how to bootstrap a first suite from real failures, how to choose graders (code-based, model-based, human) for each agent architecture, how to write tasks that grade fairly, which consistency metric to report, and how to keep the suite trustworthy over time. Distilled from Anthropic's engineering guide "Demystifying evals for AI agents."

When to use

Use this skill when:

• You are asked to create an eval suite for an agent (coding, conversational/support, research, or computer-use) and none exists.
• You are adding eval cases, graders, or a CI eval gate to an existing agent project.
• An eval score looks suspicious (0% pass rate, sudden jump/drop, plateau) and you must diagnose whether the agent or the eval is at fault.
• You must choose or explain metrics (pass@k vs pass^k) or decide when a capability eval should become a regression eval.
• A model upgrade is planned and the team needs to verify it quickly.

Do NOT use this skill for: single-turn prompt/response benchmarking with no tools or state (standard eval practice suffices), production A/B test design, or training-data curation. If the team already has a healthy suite and just wants a new task added, jump straight to Rules 6-9.

Rules

Framing and vocabulary

1. Use the shared vocabulary precisely: a task has defined inputs and success criteria; a trial is one attempt at a task; a grader scores one aspect of performance (a task may have several); a transcript records all outputs, tool calls, and reasoning; the outcome is the final environment state — verify it, don't trust what the agent says it did; an eval suite groups related tasks; the eval harness runs it all end-to-end.
2. Maintain two suite types with opposite pass-rate targets. Capability evals probe what the agent can do well — they should start at a LOW pass rate, targeting tasks the agent struggles with. Regression evals confirm the agent still handles everything it used to — they should hold a nearly 100% pass rate; any drop signals broken functionality. When a capability eval reaches high pass rates, graduate it into the regression suite.
3. Start early; don't wait for a perfect suite. "20-50 simple tasks drawn from real failures is a great start." Early on, each change has a large effect size, so small sample sizes suffice; mature agents need larger, harder suites to detect smaller effects. If you delay, product requirements stop translating cleanly into test cases and you end up working backwards from live-system behavior to reconstruct what success even means.

Sourcing tasks

4. Convert existing manual testing first: the behaviors someone verifies before each release, and the common tasks end users try.
5. If the product is live, mine the bug tracker and support queue. Converting user-reported failures into eval tasks keeps the suite reflecting actual usage; prioritize by user impact.

Task design

6. Write unambiguous tasks: "A good task is one where two domain experts would independently reach the same pass/fail verdict." Everything the grader checks must be inferable from the task description — agents must not fail due to ambiguous specs, and vague rubrics produce inconsistent judgments.
7. Create a reference solution for every task. It proves the task is solvable and the grader accepts a correct answer. If you (or a domain expert) could not pass the task yourself, refine the task.
8. Treat a 0% pass rate as a bug in the eval, not the agent: with frontier models, zero passes over many trials (0% pass@100) usually indicates a broken task rather than an incapable agent — recheck the task spec and graders.
9. Build balanced problem sets: test both where a behavior SHOULD occur and where it SHOULDN'T. If you only test that the agent searches when it should, you will train/tune toward an agent that searches for almost everything. (Anthropic's web-search evals needed many refinement rounds balancing should-search queries like weather against answer-from-knowledge queries like "who founded Apple?".)

Harness and environment

10. Isolate every trial: start from a clean environment. Leftover files, cached data, or resource exhaustion cause correlated failures — and shared state can inflate scores (internal evals caught Claude gaining an unfair advantage by reading git history left by previous trials).
11. Pick an eval framework fast and spend your energy on the tasks, not the infrastructure. Options: Harbor (containerized execution, standardized task format), Braintrust (offline evals + production observability), LangSmith (tracing, LangChain ecosystem), Langfuse (self-hosted open source), Arize Phoenix/AX. No framework compensates for weak tasks — task quality is the bottleneck.

Grader selection — by type, then by architecture

12. Follow the preference hierarchy: deterministic (code-based) graders where possible, LLM graders where necessary or for extra flexibility, human graders judiciously. Code-based graders (string/regex/fuzzy matching, pass/fail tests, static analysis, outcome verification, tool-call verification, transcript analysis) are fast, cheap, objective, reproducible, easy to debug — but brittle to valid variations. Model-based graders (rubric scoring, natural-language assertions, pairwise comparison, reference-based grading, multi-judge consensus) capture nuance and handle open-ended tasks — but are non-deterministic, cost more, and require calibration against humans. Human graders (SME review, spot-check sampling, A/B tests, inter-annotator agreement) are the gold standard used to calibrate everything else — but expensive and slow.
13. Grade the outcome the agent delivered, not the route it took to get there. Checking for a specific sequence of tool calls is too rigid: agents routinely discover legitimate solution paths the eval author never considered, and path-checking punishes that creativity.
14. Build in partial credit. A support agent that gets partway — diagnoses the issue, verifies the customer, then stumbles on the refund step — is far ahead of one that fails at the first turn; binary all-or-nothing grading erases that gradient. Combine graders with weighted, binary (all-pass), or hybrid scoring.
15. Calibrate LLM graders deliberately: validate against human expert judgments before trusting them; give the judge a way out (instruct it to return "Unknown" when it lacks information) to prevent hallucinated verdicts; write clear structured rubrics and grade each dimension with an isolated LLM-as-judge rather than one judge grading all dimensions.
16. Make graders bypass-resistant: passing must genuinely require solving the problem, not exploiting a loophole in the checker.
17. Match the grading strategy to the agent architecture:

• Coding agents: well-specified tasks in stable test environments; deterministic graders (unit tests) verify outcomes; code-quality rubrics on the transcript; static analysis (linters, type checkers, security scanners). Reference points: SWE-bench Verified, Terminal-Bench.
• Conversational agents: state-based outcome verification (ticket resolved, refund processed); LLM rubrics for interaction quality; user simulation with a second LLM playing a persona (as in τ-Bench / τ2-Bench); multidimensional metrics combining completion, efficiency (turn count), and tone.
• Research agents: combine groundedness checks (claims supported by sources), coverage checks (key facts included), and source-quality verification; exact matching for objective sub-questions; LLM checks for coherence; calibrate frequently against expert humans because ground truth shifts and experts disagree on synthesis quality. Reference point: BrowseComp (tasks easy to verify, hard to solve).
• Computer-use agents: run in real or sandboxed application environments; check URLs and page state; verify in the backend that the action actually happened (order placed, not just confirmation page rendered); balance DOM-based interaction (fast, token-heavy) vs screenshot-based (slower, token-efficient). Reference points: WebArena, OSWorld.

Metrics under non-determinism

18. Run multiple trials per task and report a consistency-aware metric. pass@k = probability of at least one success in k attempts (rises with k) — use for tools where one success matters. pass^k = probability that ALL k trials succeed (falls with k) — use for agents where consistency is essential. Anchors: 50% pass@1 = the agent clears half the tasks on its first attempt; a 75% per-trial success rate over 3 trials gives pass^3 = (0.75)³ ≈ 42%.

Reading transcripts and maintenance

19. Read the transcripts. You cannot know whether graders work until you read transcripts and grades from many trials. Every failure should be explainable — a reader can point at what the agent did wrong and why it counts as wrong. Trust no score until someone has dug into the eval's details and read actual transcripts; if grading is unfair, tasks are ambiguous, valid solutions are penalized, or the harness constrains the model, revise the eval, not the agent. (Case studies: Opus 4.5 on CORE-Bench scored 42% until grading bugs — e.g. rigid matching of "96.12" vs "96.124991…" — were fixed, then 95%. A METR benchmark graded "exceed the threshold" while the task said "reach the threshold," penalizing instruction-following models. On τ2-Bench, Opus 4.5 "failed" a flight-booking task by finding a policy loophole that was actually better for the user.)
20. Watch for saturation. When an agent passes all solvable tasks, the eval stops discriminating and big capability gains show up as tiny score changes (SWE-bench Verified went from ~30% to >80% in about a year; Qodo's one-shot coding evals missed gains on longer agentic tasks until they built a new agentic framework). Add harder tasks or a new suite before you hit the ceiling.
21. Assign ownership. An eval suite is a living artifact: a dedicated evals team owns the core infrastructure, while domain experts and product teams — the people closest to requirements and users — contribute most tasks and run the evals themselves. Treat owning and iterating on the suite as routine engineering work, on par with keeping unit tests green. Lower the contribution bar (e.g., contribute an eval task as a PR via Claude Code).
22. Practice eval-driven development: write the eval that defines a planned capability before the agent can pass it, then iterate the agent until it does.
23. Layer complementary methods — no single layer catches everything (Swiss-cheese model): automated evals pre-launch/CI, production monitoring post-launch, A/B testing for mature products, ongoing user feedback, ongoing manual transcript review, systematic human studies at calibration points.

Checklist

From-zero eval bootstrap — execute in order:

• Collect 20-50 realistic tasks from real failures: pre-release manual checks, common user tasks, bug tracker, support queue. Prioritize by user impact.
• For each task: write input + success criteria so two domain experts would reach the same pass/fail verdict independently.
• Write a reference solution per task; confirm it passes the grader. If you can't solve it, fix the task.
• Balance the set: include should-do AND should-NOT-do cases for every behavior under test.
• Stand up the harness with per-trial isolation (clean environment, no shared files/cache/history between trials). Pick a framework quickly (Harbor, Braintrust, LangSmith, Langfuse, Arize) and move on.
• Choose graders per task: deterministic first, LLM where needed, humans sparingly. Grade outcomes and final environment state, not tool-call sequences. Add partial credit where a task has meaningful intermediate milestones.
• For each LLM judge: structured rubric, one isolated judge per dimension, an explicit "Unknown" escape hatch, and calibration against human expert labels.
• Check graders for bypasses: can a wrong-but-clever transcript pass? Fix before trusting scores.
• Run multiple trials per task; pick pass@k (one success matters) or pass^k (consistency matters) to match the product.
• Read transcripts from many trials; verify every failure seems fair. Fix eval bugs (0% pass rate = broken task until proven otherwise).
• Split suites: capability (low pass rate, hard tasks) vs regression (~100% pass rate, gate on drops). Graduate saturating capability evals into regression.
• Assign an owner for infrastructure; route task contributions to domain experts/product. Schedule transcript reviews and saturation checks as routine maintenance.

Anti-patterns

• Waiting for the perfect suite. Delay forces you to work backwards from live-system behavior to define success; start with 20-50 tasks now.
• Over-specified path grading. Asserting an exact tool-call sequence produces brittle tests; agents find valid alternative approaches. Grade the product/outcome.
• Class-imbalanced evals. Testing only "should do X" cases optimizes the agent into doing X for everything.
• Shared state between trials. Leftover files, caches, or git history inflate scores (agents have exploited prior trials' git history) and correlate failures.
• Trusting the agent's claim of success. Verify the outcome in the environment/backend, not the confirmation message.
• Taking scores at face value. Score jumps, plateaus, and 0% floors are eval bugs until transcripts prove otherwise (CORE-Bench 42%→95% after grader fixes).
• One monolithic LLM judge for all dimensions. Use isolated per-dimension judges with structured rubrics; uncalibrated judges without an "Unknown" option hallucinate verdicts.
• All-or-nothing scoring on multi-step tasks. Hides real differences between agents that get partway and agents that fail instantly.
• Ignoring saturation. A near-ceiling eval makes large capability gains look like noise; add harder tasks.
• Ownerless suites. Unowned evals rot; treat eval maintenance like unit-test maintenance.
• Framework-first energy allocation. A framework adds nothing beyond the quality of the tasks you feed it; invest in tasks.

Source

• Demystifying evals for AI agents — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents — 2026-01-09

Distilled from the official document(s) above on 2026-08-12. If this skill and the source disagree, trust the source.