Write and manage ADK eval cases

An eval case says: given this input, the agent should do these things, in this order, with these values. Writing them well is mostly about scope — one behaviour per case — and about covering the branches the instruction actually has, rather than the one path someone demonstrated. Keeping a suite good over time is the same discipline applied to renames, splits and deletions.

${CLAUDE_SKILL_DIR} is this skill's directory.

Flow

1. Discover. What the agent can do, what is already covered, where the gaps are:

   python3 ${CLAUDE_SKILL_DIR}/../adk/scripts/discover.py <agent-dir> --focus layout,agent,evalsets,configs,coverage
   python3 ${CLAUDE_SKILL_DIR}/../eval-review/scripts/coverage_matrix.py <agent-dir>
   python3 ${CLAUDE_SKILL_DIR}/scripts/manage_cases.py list <agent-dir> --by-category

   The coverage matrix names the specific uncovered branches, which is what the new cases should be. The list shows what a suite already asserts.

2. Report. Tools and their argument contracts, existing cases by category, the branch count against the case count, the uncovered rules by name, and anything the suite carries that it should not (duplicates, vague ids, a stale app_name).
3. Options. Usually: close the branch gaps, capture a golden trace from a real run, add the missing adversarial or memory layer, migrate a legacy file, add a regression case for a bug just fixed, or tidy the suite (rename, split, dedupe). All free.
4. Confirm, then generate with --dry-run, show the write plan, write, and validate with /adk:eval-review's scripts. An explicit prior grant from the user ("proceed without asking") satisfies this step; the plan is still shown.

The categories

Roughly 70% narrow single-turn cases, 20% multi-turn, 10% adversarial. Weight adversarial higher for an agent that spends money or writes to a system of record. A baseline case with no tools bound proves the agent degrades honestly instead of inventing capabilities.

Decision rules

• One behavioural concern per case. If naming what it proves needs an "and", it is two cases; manage_cases.py split turns one into several.
• N branches need N+1 cases. One per conditional or prohibition in the instruction, plus the happy path. The coverage matrix counts them for you.
• At least one case must be a refusal. A suite of happy paths cannot detect an agent that approves everything.
• A forbidden call needs no special syntax. Write the trajectory that should happen — which stops before the forbidden call — and let exact matching do the work. Add a restraint rubric via /adk:eval-criteria to say it in words too.
• Assert on arguments, not just names. The right tools in the right order with the wrong claim id is a broken agent that a name-only check passes.
• Name the behaviour in the id. The id is what a person reads first when a run goes red; edge_frozen_cost_centre_no_submit tells them what broke before they open the file, test_3 does not.
• Leave final_response empty when only the trajectory matters. With response_match_score at 0, an empty expected answer is deliberate.
• Never edit an existing case to make a run pass. Every writer here refuses to overwrite an eval_id. Change a case deliberately, with the reason recorded, through manage_cases.py, and re-capture rather than hand-edit.
• Every turn of a multi-turn case must get a reply. Through 2.13.0 a turn the agent answers with no event aborts the whole adk eval run and scores no case. If the agent may legitimately stay silent there, give it a fallback reply or end the case before that turn.
• A skill pair needs a negative sibling case. A request meant for skill A must load A and not B, worded the way B's description is (AGT083 flags the riskiest pairs).
• Capture only runs you have read. A golden trace of the agent doing the wrong thing locks in the wrong thing.
• Legacy files come in two shapes. The wrapped shape still runs under adk eval; the flat-turn shape crashes it and only AgentEvaluator reads it. Migrate both for a stable id and a readable diff.

Capturing a golden trace

adk run <agent-dir> --jsonl "the request to record" > /tmp/run.jsonl
python3 ${CLAUDE_SKILL_DIR}/scripts/session_to_evalset.py --trace /tmp/run.jsonl \
    --eval-id happy_travel_claim --evalset tests/eval/evalsets/happy.evalset.json

Without --jsonl, adk run prints text only and every tool call is invisible. The dev UI does the same conversion: adk web, run the request, then Eval → add the current session. That tab also edits and deletes cases and runs them with threshold sliders; see managing-evalsets.md.

References

Templates: assets/templates/evalsets/ — one per category (including activation.evalset.json, a positive and a negative skill-load case, and resume.evalset.json, a hold-at-gate and a proceed-from-step case seeded with state), a baseline no-tools case, and a scenario case.

Scripts

Completion checklist

• Every new case has one concern and a descriptive eval_id.
• Each uncovered branch named by the coverage matrix now has a case, or was consciously left.
• At least one case in the suite is a refusal.
• Arguments are asserted, not just tool names.
• session_input.app_name matches what the agent resolves to.
• No existing case was modified without the user deciding it.
• validate_evalset.py and crosscheck_tools.py exit 0.