Run ADK evaluations

The thing to know before anything else: through ADK 2.12.0, adk eval exits 0 whatever happened. It prints

Eval Run Summary
expense_claims:
  Tests passed: 3
  Tests failed: 1

and returns success. A CI step calling it directly on a 2.12 or older pin passes forever. It also counts NOT_EVALUATED in the failed column, so its own summary cannot distinguish a regression from an unreachable judge. Everything here runs through a wrapper that reads the result files and reports the three statuses separately. (From 2.13.0 adk eval exits 1 when a case fails or none ran; the wrapper stays the recommended command on every version, because only it gates on a pass rate, regressions or budgets and keeps never-scored apart from failed.)

${CLAUDE_SKILL_DIR} is this skill's directory.

Flow

1. Discover. Layout, runner, backend, criteria and their cost:

   python3 ${CLAUDE_SKILL_DIR}/../adk/scripts/check_env.py <agent-dir> --config <config>
   python3 ${CLAUDE_SKILL_DIR}/scripts/run_evals.py <agent-dir> --dry-run

   --dry-run prints the exact commands, before any credential check and without needing ADK installed.

2. Report. The runner it resolved, the config it will use (and any others it found), the metrics and their cost tier, and how many cases will run. When several evalsets or configs exist, ask which before running.
3. Options and cost. Before spending anything, say what it costs. A judge metric costs one model call per invocation per sample: 20 cases with 2 turns, two rubric metrics at 3 samples is about 240 judge calls, times --runs. Offer the cheap subset first.
4. Confirm, run, hand off. Then send the results to /adk:eval-analyze.

Which command for which layout

Recipes layout — the config is never auto-discovered, so always pass it:

adk eval <project>/app <project>/tests/eval/evalsets/happy.evalset.json \
    --config_file_path <project>/tests/eval/eval_config.json

Flat layout — test_config.json beside a single evalset is found automatically; with several evalsets, pass it explicitly:

adk eval <agent> <agent>/happy.test.json

Selected cases only: append them to the evalset path (happy.test.json:edge_frozen_cost_centre,edge_over_limit).

The honest gates

Never adk eval alone in CI. A project with no pytest gate gets one from gen_test_eval.py.

Decision rules

• Repeat runs with --runs N, not num_runs. AgentEvaluator(num_runs=N) averages scores across runs, which hides the variance you are measuring. --runs N keeps each run's results separate so /adk:eval-analyze can compute pass@k.
• Concurrency is fixed at 4 cases through 2.13.0. adk eval has no --parallelism and drops a parallelism config key (CFG077); main after 2.13.0 adds both, plus AgentEvaluator.evaluate(parallelism=). To go easier on a quota today, split the evalset; run_evals.py --dry-run says the same.
• An empty suite fails on 2.10+, and used to pass. AgentEvaluator.evaluate and evaluate_eval_set raise ValueError when there are no eval cases, no *.test.json under the directory, or num_runs < 1. On 2.9.2 all three passed silently, so a gate that had lost its cases was green.
• Efficiency values are not verdicts. On 2.10+ every run also reports tool_call_count_v1, inference_call_count_v1, token_usage_v1 and invocation_duration_v1 as informational. They cannot be configured and they gate nothing; /adk:eval-analyze keeps them out of the pass/fail count.
• NOT_EVALUATED is not a regression. Default --not-evaluated fail in CI, because a suite that silently stops scoring is worse than one that fails. Use warn locally, where exit 3 means "inconclusive" rather than "broken".
• Check the backend before spending. Metrics scored by the Vertex evaluation service need GOOGLE_API_KEY, or a project and location with ADC; without either you get a page of never-scored cases.
• A flat-turn legacy file cannot go through adk eval. It crashes inside ADK; the wrapper refuses it and points at migrate_legacy.py or the pytest path.
• A turn with no reply sinks the whole run through 2.13.0. If the agent answers a user turn with no event, adk eval and AgentEvaluator raise ValueError: Inferences should match conversations in eval case and score no case; run_evals.py names it and exits 2. Give the agent a fallback reply or move the turn; main records it as a turn with no final response, which only a response metric fails (cli-reference.md).
• Deferred capacity is not an adk eval option. No eval entry point takes a RunConfig, so a bulk run on ServiceTier.DEFERRED (2.10+, Preview, allowlisted) is your own runner loop (batch-and-gating.md).
• Run the cheap tier first. There is no point paying a judge to score an agent that called the wrong tool.
• Resolve the runner honestly. $ADK_PYTHON, a project venv, an adk on PATH, then uv run --with "google-adk[eval]==2.13.*". If none works, say so and give the install command.
• Recording conformance needs a running dev server on 127.0.0.1:8000 started with the recordings plugin; the script checks the port and prints the command.

References

Templates: assets/templates/{eval_registry.json,test_eval.py.tmpl,test_behaviors.py.tmpl,conformance_spec.yaml}.

Scripts

Completion checklist

• The runner and ADK version were reported, not assumed.
• The cost tier was stated before anything was spent.
• The recipes layout, or any multi-evalset run, got an explicit --config_file_path.
• Failures and never-scored cases were reported separately.
• Result files were located and handed to /adk:eval-analyze.
• Nothing in CI relies on bare adk eval (an error before 2.13, a warning from 2.13: CI001).