Review ADK evalsets and configs

An evalset is a specification. This skill checks that it says what its author meant, that ADK will read it that way, and that it covers what the agent promises. It reports; it does not edit. Fixing is /adk:eval-create or /adk:eval-criteria, after the user has seen the findings.

${CLAUDE_SKILL_DIR} is set by the skill runtime to this skill's own directory. Use it verbatim in every command below; a sibling skill's scripts are at ${CLAUDE_SKILL_DIR}/../<skill>/scripts/.

Flow

1. Discover and validate. Three scripts, cheap and deterministic:

   python3 ${CLAUDE_SKILL_DIR}/scripts/validate_evalset.py <agent-dir-or-file>
   python3 ${CLAUDE_SKILL_DIR}/scripts/crosscheck_tools.py <agent-dir-or-file> --agent-dir <agent>
   python3 ${CLAUDE_SKILL_DIR}/scripts/coverage_matrix.py <agent-dir>

   For the config, call the sibling skill's validator:

   python3 ${CLAUDE_SKILL_DIR}/../eval-criteria/scripts/validate_config.py <config> --agent-dir <agent>

2. Report. Show Found, then errors, warnings and notes in that order, then the coverage gaps. Lead with the count: "3 errors, 6 warnings, 7 cases short of branch coverage."
3. Options. Typically: fix the schema errors, close the coverage gaps, adjust the criteria, or migrate a legacy file. All are free; the work happens in another skill.
4. Confirm, then hand off with the Skill tool. Do not start editing here.

What the checks are looking for

Three classes, and the middle one is why this skill exists.

ADK will reject it. eval_set_id outside ^[a-zA-Z0-9_]+$, a case with both conversation and conversation_scenario, a missing user_content.

ADK accepts it and ignores it. Eval cases allow unknown keys, and criterion objects do too, so a misspelled field raises nothing and silently changes nothing. Also here: a per-case or per-invocation rubric with no type (no criterion will read it), and an expected call to a tool that does not exist (it can never match, so the case fails forever without telling you anything).

It runs, but it proves less than you think. A threshold of 0, ignore_args turned on, a rubric asking for a 1-5 rating, a suite with no case designed to fail, a branch in the instruction with no case at all.

Decision rules

• An expected call to a nonexistent tool is an error, not a warning. The case can never pass, and it will be read as an agent regression.
• A reference-based metric with threshold 0 needs no reference response. Disabling response_match_score is the normal way to run a trajectory-first suite; an empty final_response is then a note, not a fault.
• session_input.app_name mismatches are warnings. The run still works, since the eval runner takes its app_name from the session the case creates. What breaks is scoping: app: state, memory and artifacts live under that name.
• Coverage is evidence, not a verdict. Rule-to-case matching is heuristic. Present uncovered branches as questions for the user, not as failures.
• Per-turn rubrics are a released-version trap. Through ADK 2.13.0 every judge reply is matched against the last turn's rubric list, so a turn whose list differs loses its own verdicts and the case can pass with a failing rubric. EVS121 warns (info on a newer runner); suggest one list for every turn or case-level rubrics. A rubric_id repeated between case and turn (EVS122) stops the whole run after inference, so it is an error.
• Never edit a golden trace to make a check pass. If a case looks wrong, say why and let the user decide; a changed specification is a decision, not a fix.

References

Scripts

Completion checklist

• All three validators were run, and their counts reported (plus validate_eval_yaml.py when the target is an EVAL.yaml).
• Every error was explained with its fix, not just its id.
• Coverage gaps were named as specific uncovered branches.
• Nothing was edited; the user chose what happens next.