ADK agents and evaluations — start here

Everything this plugin does rests on one move: turning behaviour that currently exists only as an instruction — and as whatever the author remembers testing — into cases a machine replays on demand. Nothing else here is worth much until that exists, and once it does, most questions about the agent have an answer you can measure. So the first step is never advice: it is finding out what this project already has.

${CLAUDE_SKILL_DIR} is set by the skill runtime to this skill's own directory. Use it verbatim in every command below; a sibling skill's scripts are at ${CLAUDE_SKILL_DIR}/../<skill>/scripts/.

Flow

1. Discover. Run it before saying anything about the project:

   python3 ${CLAUDE_SKILL_DIR}/scripts/discover.py <path> --format md

   Add --no-probe to skip the interpreter check when you only need file facts. --focus layout,evalsets narrows the work. For the environment alone: python3 ${CLAUDE_SKILL_DIR}/scripts/check_env.py <agent-dir>.

2. Report. Show the user the Found / Missing / Warnings blocks the script produced. Do not paraphrase the numbers; they are the evidence for every option you are about to offer.
3. Options. Offer the skills from the report's Options section, each with its cost tier (free deterministic, judge model calls, vertex Vertex AI evaluation service). Exactly one is marked recommended.
4. Confirm. Ask with AskUserQuestion, then invoke the chosen skill with the Skill tool. Nothing is written and no model call is paid for before that answer. Full protocol: interaction-protocol.md.

Route by intent

A request that names none of these is usually "is my agent any good?", which starts with /adk:eval-review when cases exist and /adk:eval-create when they do not.

Several agents in one repository

When discovery reports a collection, or the request is about sub-agents or a monorepo, inventory them before choosing one:

python3 ${CLAUDE_SKILL_DIR}/scripts/inventory.py <repo-root>

One row per loadable agent: layout, app name, tools, cases by category, config cost tier, last result, maturity level. Read down the maturity column and the next piece of work names itself. A sub-agent can only be evaluated on its own through the pytest path (/adk:eval-run --agent-name); adk eval has no such option. run_registry.py --init turns the inventory into a batch manifest. An adk-recipes Agent-Plugins skill layer (plugin.json, skills/*/EVAL.yaml) is not an agent: discovery reports ENV014 and a plugin_layer fact instead of a layout, and the inventory lists the layer after the agents with its cases, rubric lines and assertion types. Send it to /adk:eval-review (validate_eval_yaml.py); nothing runs it. An agents-cli project with tests/eval/datasets/ (the voice recipe has two suites and two eval_config_system*.yaml) is listed as agents_cli_datasets with its configs; none of it is an ADK evalset.

What to tell the user when they ask what this can do

Six things, in the order they usually matter:

1. Build an eval-ready agent, or add eval scaffolding to one that exists.
2. Review the agent statically against ADK 2.x rules and a model policy.
3. Write eval cases: happy path, edge, adversarial, memory, regression.
4. Choose criteria from the 13 configurable ADK metrics, with thresholds and rubrics.
5. Run them, locally or in CI, and read the results honestly.
6. Measure reliability across repeated runs, and triage what fails.

Ground rules

• Never guess the layout, version or backend. Run the discovery script.
• An existing golden trace is a specification. Never edit one to make a failing test pass without saying so; add a sibling case instead.
• Say what a run costs before running it. Judge and Vertex metrics spend money per case per sample.
• Cite the source when you explain an ADK behaviour, so the user can check.
• A refused app name is a blocker, not style. ENV015 as an error means adk run, adk web and adk eval cannot load the app directory (a hyphen, dot, space or non-ASCII letter; 2.13.0's adk create writes such names): offer the rename, with every app_name reference, before any eval work. As an info (a keyword, a leading digit) it loads today; mention it and move on.
• Main-only findings keep the released verdict. CFG077 (a parallelism key) and AGT082 (memory tools inside an AgentTool) warn because 2.13.0, the latest release, ignores the key and isolates the memory; they become info only on a runner newer than 2.13.0. Say which version the warning is about.

References

Scripts

Completion checklist

• Discovery was run, and its output shown to the user rather than summarized from memory.
• The ADK major version is known, or the user was told it could not be verified.
• Options were offered with cost tiers, and the user chose.
• The chosen skill was invoked, not re-implemented here.