Debug a misbehaving ADK agent

The expensive mistake is a correct fix applied at the wrong layer: tightening a prompt when the tool's error message was the problem, or changing a model when the instruction was ambiguous. So this works out which layer before suggesting anything.

${CLAUDE_SKILL_DIR} is set by the skill runtime to this skill's own directory. Use it verbatim in every command below; a sibling skill's scripts are at ${CLAUDE_SKILL_DIR}/../<skill>/scripts/.

Flow

Two of the steps below spend money — a fresh adk run costs a model call per turn, and five suite runs cost five suites. Neither happens before the user has seen the evidence and chosen.

1. Rule out the evaluation first. Before debugging the agent, check the test is right:

   • Is the golden trace still correct, or did the behaviour change deliberately?
   • Are the thresholds sensible — is anything scored on wording that should not be?
   • Did the run actually score, or is this a NOT_EVALUATED in disguise?

   A wrong eval means you are debugging a false positive.

2. Read the evidence that already exists. Free, and usually enough to name the shape:

   python3 ${CLAUDE_SKILL_DIR}/scripts/analyze_trace.py --result <agent-dir>/.adk/eval_history --eval-id <id>
   python3 ${CLAUDE_SKILL_DIR}/scripts/analyze_trace.py --trace <existing.jsonl> --agent-dir <agent-dir>

3. Report. The shape, the layer it points at, the first divergence, and what the existing evidence cannot settle.

4. Options, with their cost, then confirm. Typically:

   Option	Cost	Answers
   capture a fresh trace	one model call per turn	what the agent does now, with tool calls visible
   five runs, then pass@k	five full suite runs	deterministic or stochastic
   fix at the named layer	free to propose	—

   Ask with AskUserQuestion and wait. Nothing below runs before the answer.

5. Run what was chosen.

   adk run <agent-dir> --jsonl "the request that fails" > /tmp/run.jsonl
   python3 ${CLAUDE_SKILL_DIR}/scripts/analyze_trace.py --trace /tmp/run.jsonl --agent-dir <agent-dir>

   Without --jsonl, adk run prints text only and every tool call is invisible.

   python3 ${CLAUDE_SKILL_DIR}/../eval-run/scripts/run_evals.py <agent-dir> --runs 5
   python3 ${CLAUDE_SKILL_DIR}/../eval-analyze/scripts/pass_at_k.py --dir <agent-dir>/.adk/eval_history

   Same failure every time is a different diagnosis from three times in five, and the fixes have nothing in common.

6. Name the fix, confirm, close the loop. One change at the named layer, a regression case, re-run several times, then the full suite.

The three layers

Plus the evaluation itself, which is step 1 for a reason.

Failure shapes

Three worth memorising, because they cover most first reports:

The other five — oscillation, invented tool, cascade, wrong order, wrong tool — plus what each one rules out, are in failure-shapes.md. analyze_trace.py names them all, so read the script's output rather than matching by eye.

Deciding between layers

Deterministic — same failure every run:

• right tool, wrong arguments → tool layer: tighten the schema, or improve the argument's description in the docstring.
• right arguments, wrong sequence → prompt layer: add the ordering rule.
• a tool that should have been called never is → prompt layer: the descriptions do not distinguish it.
• everything looks right and the outcome is wrong → cascade: work backwards to the first divergence.

Stochastic — passes sometimes:

• temperature=0.0 fixes it → sampling variance. Lower it, or add a structural constraint so there is less to vary.
• an explicit "do not skip steps" fixes it → the instruction was underspecified. Make the requirement explicit rather than louder.
• neither → a capability limit. In order: add pipeline constraints, split the work across agents, then consider a stronger model for that step.

Decision rules

• Fix the first divergence, not the last symptom. Everything after it is consequence.
• One change at a time, then re-measure. Two simultaneous fixes on a non-deterministic system teach you nothing.
• A tool failure does not raise. ADK returns it as a result with an error key and the run continues, so "the tool silently did nothing" means reading the tool responses.
• Five runs before and after. One green run after a change proves nothing that one green run before it did not. Both sets cost model calls, so say so and get the go-ahead before either.
• Every fix earns a regression case. The repro that found it is the case that keeps it fixed.

References

Scripts

Completion checklist

• The evaluation was ruled out before the agent was blamed.
• Existing evidence was read, and reported, before anything billable ran.
• Every billable option was offered with its cost, and the user chose.
• The failure was reproduced, and shown to be deterministic or stochastic.
• A shape and a layer were named, with the evidence for them.
• One change was made at that layer.
• A regression case was added.
• The fix was confirmed across several runs, not one.
• For a loop, the reply itself leads with the layer, says that the tool returned its error instead of raising (so the run kept going), and proposes both an error message that says what to change and a retry cap.