Gate changes on ADK evals

One fact shapes every workflow here: adk eval exits 0 whatever happened on every release through 2.12.0. A CI step that calls it directly on such a pin is green whether the cases passed or not, and that is the most common way an eval suite ends up proving nothing. From 2.13.0 it exits 1 when a case fails or none ran, yet it still cannot gate on a pass rate, on regressions against the default branch or on a budget, so the workflows here call run_evals.py on every version. lint_workflows.py reports bare adk eval as CI001: an error when the pin is below 2.13 or unknown, a warning from 2.13 (--adk-version overrides the pin it detects in pyproject.toml, requirements*.txt or the workflow's --with).

The second fact is cost. A judge metric costs model calls per invocation per sample, so a suite that is thorough enough to trust for a release is too expensive to run on every push. Hence two tiers.

${CLAUDE_SKILL_DIR} is set by the skill runtime to this skill's own directory. Use it verbatim in every command below; a sibling skill's scripts are at ${CLAUDE_SKILL_DIR}/../<skill>/scripts/.

Flow

1. Discover. What exists, and where the project stands:

   python3 ${CLAUDE_SKILL_DIR}/scripts/assess_maturity.py <repo-root>
   python3 ${CLAUDE_SKILL_DIR}/scripts/lint_workflows.py <repo-root>     # if workflows exist

2. Report. The maturity level with its evidence, any existing workflow that does not actually gate, and what the configs contain.
3. Ask where this plugin's scripts will live in the CI checkout. The generated workflow gates by calling run_evals.py by path, so this is not a detail: a wrong path fails the job at the wrong step, and looks like a broken gate rather than a missing file. Three usual answers — vendored under .github/adk-skills/ (the default), a git submodule, or installed by a step in the job — and --scripts-path P writes whichever they pick. See workflows.md, "Where the scripts live in CI".
4. Options. Usually: add the PR tier only (a good first step), add both tiers, add conformance replay, or fix an existing workflow that passes vacuously.
5. Confirm, generate with --dry-run first, write, then lint what you wrote.

The two tiers

Both gate on the wrapper's exit code; the nightly tier adds a reliability report. Timings, what belongs in each, the release tier and the NOT_EVALUATED rule are in tiered-pipeline.md.

The split is about cost and speed, not importance. A slow, paid gate on every push gets disabled, and a disabled gate is worse than a cheap one.

What the generated workflows do

• install a pinned google-adk[eval];
• authenticate with Workload Identity (or pass an API-key secret);
• run run_evals.py, not adk eval, so the exit code is real and can carry a pass-rate, regression and budget gate;
• fail on a case that was never scored, because a suite that silently stops scoring is worse than one that fails;
• upload .adk/eval_history/ with if: always(), so a failure can be examined;
• on the nightly tier, write a pass@k table and any failure explanation into the job summary, and compare with the baseline committed under eval_baseline/ so a drop in a scored metric's mean fails the run and raises the alert (compare_runs.py --max-mean-drop 0.05).

Repository setup

With Workload Identity (recommended):

With an API key: the GOOGLE_API_KEY secret. It serves every metric, the Vertex-backed ones included — the evaluation facade reads the key before any project. What you give up is Workload Identity: a long-lived secret instead of a short-lived minted token.

The maturity ladder

Each level needs the one below it, and each is checkable rather than a matter of opinion. Most projects sit between L0 and L1 while believing they are higher, because someone ran the agent and it looked right.

Decision rules

• Never gate on bare adk eval. The linter treats it as an error, and it is the check most worth running on a workflow you inherited.
• Keep judge metrics out of the per-change tier. If they are there, the tier will eventually be turned off.
• Fail on NOT_EVALUATED in CI. A never-scored case means the gate has stopped working, which is more urgent than a failing case.
• Always upload the eval history. A red pipeline with no artifact makes everyone re-run locally to learn anything.
• Reliability belongs in the nightly tier. Repeated runs are too slow to gate a push and too valuable to skip entirely.
• Pin the ADK version. An eval suite that changes behaviour because a dependency floated is a suite nobody trusts.
• Cheap gates first. Lints and validators (lint_agent.py, validate_evalset.py, validate_config.py, lint_workflows.py) call no model; run them before anything judged, and trigger the expensive suites on purpose (workflow_dispatch, a PR comment or label, paths: on the agent's files).
• Evidence must be fresh. An eval-affecting PR links a report from a run on its latest commit; one from an earlier commit does not count.
• Alert on a drop, not on a low score. The nightly run compares with the baseline and fails on the drop; a schedule without that exit code is a log, not a gate.
• Merge on two rules, not one. A global pass threshold (run_evals.py --min-pass-rate 0.95) and zero regressions against the default branch (compare_runs.py --min-pass-rate 0.95 --max-regressions 0), plus token, latency and tool-call budgets where cost matters. Freeze the golden suite ("frozen": true) so tuning cannot delete its way to green.
• Gate, then score; never blend. A failed safety or security case fails the merge whatever the token count, and budgets are separate exits, not weights; a blended score rewards a candidate that cheats on efficiency. Report the per-case pass rate beside the per-metric ones, and keep a small held-out set nobody tunes against.
• Evals run in CI, not on a laptop. Cached credentials and exported project ids make a run pass that fails elsewhere. Re-run the gating suite on every model upgrade before switching.

References

Templates: assets/templates/workflows/.

Scripts

Completion checklist

• The PR tier contains only deterministic metrics.
• Nothing gates on bare adk eval.
• Authentication is wired, with id-token: write where Workload Identity is used.
• No credential is hardcoded in a workflow.
• Eval history is uploaded on failure as well as success.
• The nightly tier has a run count and a reliability report.
• --scripts-path matches where the plugin actually lives in the CI checkout.
• The linter exits 0 on what was generated.
• The reply itself says why the gate is the wrapper (bare adk eval exits 0 through 2.12.0 and only 1 from 2.13.0, with no pass-rate or regression gate) and lists the repository variables or secrets the workflow needs.