ADK agent evaluation

Produce evidence about an agent's decisions, executed effects and final account of those effects. Adapt the target project; keep its business domain, architecture, dependency pins and test conventions. This skill works without the book or its companion repository.

Inspect and define the job

1. Read repository instructions, manifests/lockfiles, test configuration and existing agent factories, tools, callbacks, session services and evaluation assets. Use the project's existing interpreter to inspect installed package metadata before importing application code. Imports and test collection can have side effects; inspect their entrypoints first. Check the pinned Python, google-adk, google-genai and any managed-evaluation SDK versions against compatibility.md. Preserve pins; a version mismatch requires interface verification or a clearly reported blocked live path, not an automatic dependency change.
2. Locate trusted identity, authorised mutations, idempotency keys, business-store lifetime, conversation-state lifetime and the model-injection seam. Read credential variable names and presence, never print secrets or dump .env. Identify existing offline/live markers and which commands could reach remote services. Finish inspection with a concise map of these boundaries and missing prerequisites.
3. Translate the user's requirement into observable success and forbidden actions: correct tool/arguments, allowed order, resulting effect, truthful response and bounded work. Distinguish deterministic assertions from model-dependent decisions. For an audit or recommendation, report each gap with its evidence and the concrete change that would close it, and leave implementation for a separate request. For implementation, present a short plan before broad changes and make the smallest coherent change.
4. Before model tuning or a quality cohort, pass an honest reference for each important outcome class through the actual response schema, validators and relevant Runner/served boundary. Include nondecisive issues, required controls and unresolved or contradictory evidence. A pass obtained by inventing associations or dropping obligations is not a solvable contract. Diagnose a failed reference first; evaluation design gives the checkpoint.

Select the mode

Combine modes only when the requested outcome needs them. Read the linked reference when entering that mode, rather than loading every file.

Planning a production suite can stop at a reviewable design with cases, budgets and acceptance criteria. Recommendations about holdouts, remote-write reconciliation or production telemetry are not claims those features exist or passed testing.

Load additional implementation detail only for the relevant branch:

• Wrong decisions or answers, or measuring a prompt, exemplar, output-schema or model change: quality iteration runs the labelled-set, error-analysis, one-change loop and measures decision quality before further controls are added. The wording and exemplar design themselves are adk-agent-instructions' work; this loop measures them. Run and categorise errors before editing code; the first change targets the largest category among defects that change a downstream decision, ranked by cost when the brief prices each error type. For example, two outputs that miss an applicable policy and one condition that contradicts its own reason put retrieval first and a consistency check second. Whether a score change is beyond noise, and how many cases a holdout or CI set needs, is its tell a change from noise section with paired_difference.py; whether one version's rate is shown to meet a fixed target, such as an acceptance threshold, is its score one version against a fixed target section, with the same helper.
• Labels that do not exist yet, expert time to plan, or criteria that keep changing: expert review runs the session with the session plan and labelling guide, writes the gold set and checks it with label_agreement.py (Cohen's kappa, disagreements to adjudicate, criteria drift, judge TPR and TNR). Read it before quality iteration when the development set has no reviewed labels.
• Setup, project/provider changes or access failures: environment and troubleshooting gives configuration tracing, API-state reconciliation and scoped cleanup checks.
• Grading with a third-party scorer or store (MLflow, deepeval, Opik): provider compatibility for what their ADK scorers record and how to keep results honest; adk-agent-observability's export paths for keeping traces and telemetry in-house.
• An incident to turn into a regression: deterministic tests for the fail_to_pass and pass_to_pass lists and their grading rule, then step 2 of evaluation design for the base-commit and reference-fix runs and the do-nothing control.
• Runtime wiring, fault injection, observers, deadlines or browser timing: execution evidence gives concrete recipes and the checks that make their results trustworthy. For a flaky or failing tool, its fault matrix pairs with the resilience scoring at the end of evaluation design.
• Dataset curation, judge calibration, work budgets, ambiguous writes or release gates: evaluation design gives production implementation procedures and acceptance criteria, explicitly separate from historical companion coverage.
• A rubric for a multi-stage pipeline, or one an optimiser or iteration loop will climb: shape the rubric before writing it. It sets code checks first, scores each stage separately on the input that stage received, judges the work on the outcome measure rather than a rubric score, and, when the brief prices each error type, gates on the priced total against today's process. A judge whose verdict stops the run is re-checked on its costly side (expert review). The behaviour a change targets gets its own fixed, categorical metric with a not-applicable class, beside the adaptive managed metrics (evaluation design), and whoever proposed the change never grades it (quality iteration, step 4). A managed trajectory verdict that flags a declared tool as not permitted makes that score incomplete (live evaluations).
• Document-grounded claims or citation judges: source-support calibration provides synthetic contrasts and an offline scorer that separates valid citations from supported conclusions.
• Dynamic output schemas or provider rejection: provider compatibility checks the actual serialized contract and plans a representative canary. Recurring structural failures: experiment re-entry records the reproducer, honest reference and new hypothesis before another broad paid cohort.
• Staged paid failures, saved-stage recovery or repeated full-suite runs: progressive validation schedules focused checks, qualified continuation and fresh end-to-end evidence before the frozen release suite.
• Agents CLI project (agents-cli-manifest.yaml, tests/eval/): its dataset, trace and result files are agentplatform EvaluationDataset and EvaluationResult JSON, not ADK EvalSet or CSV; live evaluations maps the assets and result auditing the completeness checks. Consult the installed google-agents-cli-eval skill for the commands; if it is not installed, adk-engineer explains how to get it. This skill decides the cases and the evidence; Google's skill operates the commands.
• Mandatory labels, document representations or prompt/schema/validator disagreement: contract qualification establishes source-reviewed solvability and the normal-route gate before paid inference.

Apply the pattern

• Put business permission and validation in tools/backends. Use the real runner with a scripted model to force invalid decisions. When a workflow needs fresh verification, bind it to the intended invocation or validity window; an earlier conversational lookup is not automatically permission for a later write.
• For transactional confirmations, consider rendering from trusted tool receipts. Keep facts dynamic, receipt authority separate from conversation text, and successful effects intact after later failures. Compose existing callbacks. Reserve fixed response templates for transactional confirmations; give open-ended synthesis a schema sized to the judgment it owns.
• Preserve the distinction between a new session and a fresh backend. Reset or namespace business fixtures at the experimental unit being claimed. Assert attempted calls and stored effects separately.
• Choose trajectory matching for meaningful invariants. ADK 2.8.0's relaxed order modes still compare arguments and allow extra calls; a prohibited extra mutation needs its own assertion. ROUGE checks wording overlap, not truthful completion or authorisation.
• Capture the whole invocation and close its streams/runners. Bound synchronous grading in a worker controlled by an external deadline. Timeouts can leave partial evidence and cannot recall accepted provider requests. Model, simulator, judge, retry, tool and time limits are distinct.

Authorisation and data boundaries

Inspection is read-only, and code generation authorises no deployment, paid call or live test. Before any consequential operation (an API, billing or quota change, a billable resource, a secret, IAM, a data migration, a deployment, a deletion, or a paid or destructive test), read consequential operations, present the exact scope and limits, obtain explicit approval and end the turn to wait for it. Cleanup is a separate confirmed operation on resources this workflow created.

Permission to implement tests is not permission to make paid calls or deploy them; name the isolated test target and work limits in every request. Never delete shared project resources to obtain a clean test.

Validate and report

1. Run the project's meaningful offline checks with external access prevented where hermeticity is claimed. Validate eval/config schemas with its pinned ADK models. Check guards and effects, not just expected strings. Keep failing regressions when diagnosing a real defect.
2. For approved live work, follow the planned preflight/canary and cohort protocol within its bounds; include a paid preflight only when covered by that plan. Preserve a fixed cohort's cases, attempts and stop rules. Retain case/trial/metric counts and versioned criteria. Count a skipped paid test, a missing score, a zero-case conformance run and a bare successful process exit as not run. The result-auditing reference documents the bundled read-only CSV checker.
3. Check a second invocation for duplicate writes, duplicate test registration and needless configuration churn. Run only newly justified checks after a passing validation; retain failed observations instead of rerunning to green.
4. Report files changed, commands and exit/results, actual versions, what used real providers, what used doubles, what remains unverified, and any precise manual action or approval needed. Distinguish implementing a test from executing it successfully. The target project's results come from its own runs.

Label each result with its tier (offline, local, live or release) and outcome (pass, fail, blocked or not run) as evidence labels defines them; earlier, vendor and documentation-only results are historical, vendor-measured or guidance, never this session's evidence.

The skill's own checks and limitations are recorded in validation.md. Independent community project; not affiliated with or endorsed by Google.