Choose and configure ADK evaluation criteria

Every metric answers one specific question. Picking them is choosing which questions the suite asks, and what each run costs. The two decisions that go wrong most often: leaving judge_model unset (it defaults to a deprecated model sampled five times), and gating pull requests on metrics that pay a model.

${CLAUDE_SKILL_DIR} is set by the skill runtime to this skill's own directory. Use it verbatim in every command below; a sibling skill's scripts are at ${CLAUDE_SKILL_DIR}/../<skill>/scripts/.

Flow

1. Discover. What exists, what the backend can serve, what the cases look like:

   python3 ${CLAUDE_SKILL_DIR}/../adk/scripts/discover.py <agent-dir> --focus layout,env,agent,evalsets,configs
   python3 ${CLAUDE_SKILL_DIR}/scripts/validate_config.py <agent-dir>     # when a config exists

2. Report. Current criteria and their cost tier, the backend, whether cases carry reference responses, and what is missing.

3. Options. Usually a choice among:

   Option	Cost	When
   deterministic gate	free	there is no config, or CI needs something to run per change
   add semantic response matching	judge	wording is free but meaning is fixed
   add groundedness	judge	the agent repeats numbers and ids from tools
   add restraint guarantees	judge	the agent can do something it must sometimes refuse
   add safety	vertex	the agent talks to end users

4. Confirm, then generate with --dry-run first, show the config, write it, and validate what you wrote.

Decision rules

• Start deterministic. tool_trajectory_avg_score at 1.0 costs nothing, runs in seconds, and is the only metric that catches a skipped validation. Add paid metrics when there is a question they answer that this one cannot.
• Pick the match type deliberately. EXACT for a pipeline that must not vary. IN_ORDER when a schema retry or a self-correction is legitimate and should not fail the case. ANY_ORDER when several calls are genuinely unordered, such as parallel lookups.
• Leave ignore_args off in the suite that gates releases. With it on, the right tool called with the wrong claim id passes.
• response_match_score compares wording, not meaning. Set it to 0.0 for a trajectory-first suite, use 0.8 only where the reference text is genuinely stable, and reach for final_response_match_v2 when the meaning is what matters.
• Always set judge_model and num_samples. Unset means gemini-2.5-flash — which this plugin's policy treats as retiring — at 5 samples per invocation. Use an alias id, and 3 samples for a majority vote at moderate cost.
• Rubrics are yes/no claims. No 1-5 scales: graded prompts make scores cluster in the middle, where they decide nothing. One property per rubric, named in terms a reader of the transcript could check: a compound sentence grades inconsistently (RUB006).
• A rubric about actions needs a judge that sees actions. "Did it call / run / check X" under the final-response metric is graded through the final answer (its judge reads the tool calls only as evidence for it); put it under rubric_based_tool_use_quality_v1 or beside a trajectory metric (CFG075, info). Through ADK 2.13.0 the multi-turn trajectory judge crashes on an agent with a built-in tool (CFG076, warn; needs --agent-dir).
• A parallelism key is a promise only main keeps. Every release through 2.13.0 drops a top-level parallelism silently (cases run 4 at a time); main after 2.13.0 reads it. CFG077 warns on a runner ≤ 2.13, is info on a newer one, and is an error on every version for a value main would reject (below 1, fractional, not a number). It is not the judge's parallelism_limit.
• Through 2.13.0 a rubric the judge skips is averaged away. A missing or "almost" verdict leaves the rubric out of the mean, so two yeses and a skip pass 0.7; main scores no such sample. Keep num_samples at 3 or more and check the rubric ids in the results.
• Through 2.13.0 keep one rubric list on every turn. The judge parses every reply against the last turn's list, so an earlier turn's own rubrics are dropped as "not found" and the case can pass with one failing (EVS121 in /adk:eval-review); main after 2.13.0 scores each turn on its own. Put shared rubrics at case level, and give every invocation rubric a type.
• Restraint needs its own metric. Trajectory matching and response quality both pass an agent that does too much. rubric_based_tool_use_quality_v1 at threshold 1.0 with "does NOT call X when Y" is the only check that fails it.
• Groundedness is not correctness. hallucinations_v1 asks whether each sentence is supported by what the tools returned. Set evaluate_intermediate_nl_responses to true or only the last message is read.
• Know which metrics go through the Vertex evaluation service. safety_v1, response_evaluation_score and the multi_turn_* metrics need either a GOOGLE_API_KEY or a project and location with ADC, whatever GOOGLE_GENAI_USE_VERTEXAI says. They are the vertex tier because it is a separate, separately billed service that ignores judge_model_options — not because a key is insufficient.
• The four efficiency metrics cannot be configured. On ADK 2.10+ tool_call_count_v1, inference_call_count_v1, token_usage_v1 and invocation_duration_v1 are appended to every eval automatically and reported as informational values. Naming one in criteria is an error at run time, so never write one into a config — there is no flag for it and nothing to add.
• Check the range before the threshold. response_evaluation_score runs 1 to 5; a threshold of 0.8 on it can never fail.
• Where the file goes matters. test_config.json beside a single evalset is found automatically; anything else must be passed with --config_file_path.

References

Templates: assets/templates/configs/ (one per criterion family) and assets/templates/rubrics/ (restraint, response quality, trajectory quality).

Scripts

Completion checklist

• Every judge metric names its model and sample count explicitly.
• No Vertex-only metric is configured without a project and location, or it is deliberately confined to a nightly config.
• Thresholds are inside each metric's range, and none is accidentally 0.
• Rubrics pass the lint: binary, observable, one property each.
• The config is where the command that runs it will actually find it.
• validate_config.py exits 0 on the file that was written.