Simulated users and simulated environments

Two different things, both called simulation:

• User simulation replaces the fixed user turns with an LLM playing a plan. It finds what scripted cases cannot: phrasings you did not think of, follow-ups you did not anticipate, and the human habit of not repeating yourself.
• Environment simulation replaces the tools with mocks that can return errors and add latency. It is how you test failure paths without breaking anything real. Through 2.13.0 it is an experimental ADK feature (it warns); main after 2.13.0 marks it stable.

The constraint that shapes all of user simulation: no fixed conversation means no reference, so this plugin keeps reference-based metrics off simulated cases (ADK itself would leave them never-scored).

${CLAUDE_SKILL_DIR} is this skill's directory.

Flow

1. Discover. What exists, and whether the current config could score it:

   python3 ${CLAUDE_SKILL_DIR}/../adk/scripts/discover.py <agent-dir> --focus layout,env,agent,evalsets,configs
   python3 ${CLAUDE_SKILL_DIR}/scripts/validate_sim_config.py <agent-dir>

2. Report. Scenario cases versus static cases, whether a simulator is configured, which metrics can score simulated runs, and which cannot.

3. Options. Four, and they have different costs:

   Option	Cost	What it gets you
   add scenario cases with a plan	judge	unscripted multi-turn coverage
   add a separate simulation config	judge	metrics that can actually score them
   generate scenarios from a description	vertex	breadth, quickly, and some noise
   add environment simulation	free	failure paths without touching production

4. Confirm, generate, validate. Then run the simulated suite with its own config, not the deterministic one. Generated scenarios are a draft to read before committing, like a captured trace.

What a scenario case looks like

{
  "eval_id": "memory_simulated_repeat_claims",
  "conversation_scenario": {
    "starting_prompt": "I need to claim some expenses from last week's trip.",
    "conversation_plan": "Say you work in engineering. Ask about a 900 dollar flight, then a 40 dollar meal, without repeating the cost centre. Confirm both outcomes, then stop.",
    "user_persona": "NOVICE"
  },
  "session_input": {"app_name": "app", "user_id": "evaluator", "state": {}}
}

The plan is the specification. Write it as behaviour, not topic: what the user states early, what they ask for later without repeating, when they push back, when they stop.

Metric compatibility

Keep two configs. The deterministic one scores the scripted cases; a simulation one scores the scenarios. Mixing them means half the metrics silently apply to half the cases.

Decision rules

• Simulation is a supplement, not a replacement. Scripted cases are deterministic, cheap and localizing; simulated ones are broad and noisy.
• Cap the conversation. max_allowed_invocations bounds it; every turn costs a model call for the simulator and one for the agent.
• Name the simulator's model. Unset, it defaults to gemini-2.5-flash; this plugin's policy prefers an alias id.
• Write the plan as behaviour. "Ask about expenses" produces nothing reproducible; "state your cost centre once, then ask about two claims without repeating it" tests something specific.
• A custom simulator prompt must keep its placeholders — stop_signal, conversation_plan, conversation_history — or the simulator cannot see the plan or know when to stop.
• Persona violation rubrics judge the simulated user, not the agent. They are how you check the harness behaved; per_turn_user_simulator_quality_v1 scores them.
• Environment-simulation fields are exact. mock_strategy_type is an integer enum, injection_probability defaults to 1.0, an InjectedError needs an HTTP code and a message, and every injection carries exactly one of an error or a response. Keep probabilities at 0 except in the case that is about the failure.
• A voice agent gets a voice suite. type: llm_audio plus live_model_config runs the cases through TTS and the Live API; nothing in the audio block is required and ADK accepts misspelled keys, so generate it with gen_sim_config.py --audio and let validate_sim_config.py check it (SIM032–SIM034). Drop live_model_config to debug the same cases in text. Judge the conversation with an intent rubric, vary voices and personas, and keep live suites out of the PR tier unless the budget allows. Only the live path exercises streaming and non-blocking tools; in text they do not stream (AGT081). Grade acknowledgement-before-result, not update counts.
• Semantic argument matching is main-only. InjectionConfig.args_matcher (a semantic matcher that asks a model whether two values mean the same) arrives after 2.13.0; on 2.13.0 the key is ignored, so keep match_args exact until a release has it.
• Scenario generation spends on the Vertex evaluation service and needs GOOGLE_API_KEY or a project and location; the wrapper checks before paying.

References

Templates: assets/templates/ — a simulation eval config, a custom persona, a generation config, and an environment-simulation module.

Scripts

Completion checklist

• Every scenario case has a plan that describes behaviour, not a topic.
• The simulated suite has its own config, with metrics that can score it.
• The simulator names its model and caps its invocations.
• Personas are pre-built ids or fully specified objects with list-valued instructions and violation rubrics.
• Injected environment failures are off unless a case needs one.
• validate_sim_config.py exits 0.
• The reply itself names conversation_scenario (starting prompt plus plan, not fixed turns), the persona used, which reference metric cannot score a simulated run (tool_trajectory_avg_score) and what replaces it, and that the simulated user costs model calls too.