foundry-extension-scenario-authoring

Authors and validates goal-based scenarios for the azure.ai.agents cli-interactive-tester suite under cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/. This skill writes and lints scenarios; it never runs them (running is foundry-extension-scenario-suite-run / foundry-extension-scenario-pr-regression, driven by the foundry-extension-scenario-worker agent, and incurs Azure cost for Tier 1b / Tier 2).

Tools invoked: the cli-interactive-tester MCP list_scenarios tool (for tag/lint validation only — never start_session, which would incur Azure cost), read/edit of scenario YAML files, and ask_user.

The authoring contract (read this first)

A scenario's goals: are its contract: a run PASSES only when the product's real behavior matches what the goals literally describe. So the whole point of authoring is to write goals that are a verifiable spec of correct behavior — not a loose description the driver has to interpret charitably. The judging rules are single-sourced in the scenarios README: How scenarios are judged (authoring contract). The consequences for how you write goals:

• Write goals as literal, checkable assertions. "Confirm the process exits cleanly and prints a version string" is checkable; "make sure version works" is not. A driver will FAIL a scenario whose goals aren't met — it will not rationalize a near-miss into a pass.
• Never encode a workaround in a goal. If a goal names a flag/command that doesn't exist or expects output that never appears, the scenario FAILS by design. Keep goals current with the product; don't paper over a bug in the goal text.
• Key interactive pickers off stable text labels, not positions. The driver prefers choice_text over choice_index, so phrase picker goals around the label (e.g. select "Use the code in the current directory") — indices shift between releases.
• Enumerate expected product prompts. Goals define every product prompt allowed during the scenario. Name each prompt with stable user-visible text and state the response. Use "if asked" only for a genuinely optional prompt. Never write "follow the prompts" or another catch-all; an unlisted product prompt must fail the scenario.
• Give every executed invoke explicit input. Write the literal command with a quoted positional message. Use --input-file only when file input is what the scenario tests, and define the file contents first. Never leave command construction to the worker with wording such as "run invoke and send a simple message"; bare invoke does not prompt for a message.
• Guard pre-filled prompts. When a prompt comes pre-populated (e.g. the agent name), the goal must tell the driver to clear the field first before typing, or the typed value appends to the default. See the RESOURCE NAMING / AGENT NAME goals in tier1/1.04-init-from-code.yaml for the canonical phrasing.
• Gate cost explicitly. Any goal that reaches azd provision / real resource creation belongs in Tier 1b or Tier 2 and must be tagged and tiered accordingly (see below).

The deeper why behind the picker / clear-field / no-retry rules is the executor spec driving-mechanics.md. You don't need to restate it — just author goals that hold under it.

Scenario anatomy

One file targets one command or flow. The taxonomy behind each field is single-sourced in the README — this is the annotated skeleton; follow the links for the rules:

# One-line comment: tier, what it targets, and any cost/prerequisite note.
name: "short-kebab-name"                 # tester run label; keep it short and unique-ish
command: "azd ai agent <subcommand> …"   # the installed extension entry point
cwd: "~/working/azd-agents-<slug>-{instance}"   # see cwd + idempotency below; /tmp for read-only
tags: ["tier:N", "cmd:<name>", "parallel-safe"] # REQUIRED — see the Tags taxonomy
requires: "tier1/1.0X-….yaml"            # optional; only when this scenario needs another to PASS first
produces: "~/working/azd-agents-…-{instance}/generated-agent" # optional producer output
env:                                      # optional; init scenarios disable agent auto-detect
  AZD_DISABLE_AGENT_DETECT: "1"
allocate_ports: [agent]                   # optional; only for scenarios that bind a port (e.g. 2.12)
pre:                                      # optional host-side setup (reset dir, seed fixture, auth guard)
  - run: "rm -rf {cwd-or-path}"
    cwd: "~/working"
    name: "reset working dir"
post:                                     # optional host-side cleanup
  - run: "…"
    name: "cleanup"
goals:                                    # product behavior is the literal, checkable contract
  - "Wait for … and confirm …."
  - "Take a screenshot of the final output." # best-effort evidence; capture failure is an observation
  - "Report a finding if …."

Field references (do not restate these — link to them):

• Tiers & placement — which directory (tier0/…tier2/) and cost profile: README § Tiers.
• Tags — the tier:N / cmd:* / trait namespaces and rules: README § Tags.
• requires: — cross-scenario prerequisites (Tier 1b → Tier 1): README § The requires: field.
• produces: — the verified Tier 1 scaffold handed to Tier 1b: README § Producer/consumer scaffold handoff.
• Profile/session placeholders ({prefix}, {subscription}, {region}, {model}, {model_version}, {model_sku}, {tenant}, {run_id}, {shared_agent_name}, {foundry_project_name}, {fixtures_dir}, {instance}): README § Profile / overrides.
• Pre/post hooks — semantics (host-side, sequential, fail-fast), fields, and the reset / fixture-seed / auth-guard patterns: README § Pre/post hooks.
• Fixtures — the {fixtures_dir} seed pattern: README § Fixtures.
• Idempotency — how a stateful scenario resets itself so re-runs start clean: README § Re-running scenarios.
• Conventions & parallel-readiness — resource naming, {instance} suffixing, port allocation: README § Conventions and § Parallel-readiness.

Authoring procedure

1. Confirm the target and scope. Identify the single command / flow under test and whether it is a happy path or a negative-path (error / non-zero-exit) assertion. One command per file — if you're tempted to test two, that's two scenarios (or a documented lifecycle scenario, like 2.07-sessions-lifecycle). If the request is broad or ambiguous, ask via ask_user which command and tier are intended.
2. Pick the tier and file placement. Choose the lowest tier that can actually exercise the behavior (offline Tier 0 > auth-only Tier 1 > deploy-verify Tier 1b > cloud Tier 2), because cost and flakiness rise with tier. Place the file in the matching tierN/ directory and name it <tier-number>.<seq>-<slug>.yaml, continuing the existing numbering in that directory.
3. Write command and cwd.
   • command: invokes the installed extension (azd ai agent …) or azd … for provision/deploy/down steps.
   • Read-only, stateless scenarios (version, --help, sample list) use cwd: "/tmp" and declare no hooks.
   • Stateful scenarios use a dedicated working dir suffixed with -{instance} (e.g. ~/working/azd-agents-<slug>-{instance}) so parallel instances stay isolated.
4. Make it idempotent with hooks. For a stateful scenario add a pre hook that rm -rfs its own cwd (and, if it needs source code, a second hook that seeds a fixture from {fixtures_dir} — see tier1/1.04-init-from-code.yaml). Prefer pre-wipe only for local-only scenarios so the scaffold stays on disk for inspection. Tier 1b scenarios that provision paid resources require an always-run post hook with a suitable timeout for azd down --force --purge. Tier 2 setup additionally downs a project at its current-run path before removing that directory; teardown failure must abort setup and preserve state for recovery. Do not use continue_on_error when cleanup failure would orphan resources.
5. Add a fixture only if required. If the flow needs pre-existing source (the "use the code in the current directory" path), add the minimal tree under fixtures/<name>/ and seed it via the hook. Keep fixtures minimal, but make them runnable when a later scenario deploys and invokes the generated scaffold; detection-only placeholders are not sufficient for deploy-verification prerequisites.
6. Declare producer/consumer handoff when required. A Tier 1 scenario whose scaffold is deployed by Tier 1b sets produces: to the exact generated project directory. Its Tier 1b consumer points requires: at that Tier 1 scenario (a relative path from the scenarios root) and uses {prerequisite_scaffold_dir} for cwd and every scaffold hook path. Never duplicate the producer's path in the consumer. Don't use requires: to sequence independent scenarios — it is a hard prerequisite, not an ordering hint.
7. Write the tags: list. At minimum: one tier:N, at least one cmd:*, and exactly one of parallel-safe / serial-only (Tier 0/1/1b → parallel-safe; Tier 2 → serial-only). Add negative-path, picker, or verify-deploy when they apply. Multi-command flows carry multiple cmd:* tags.
8. Write the goals: as the contract. Follow the authoring-contract rules above: literal, checkable product behavior, stable-label picks, clear-field-before-typing, best-effort screenshot evidence, and a final "report a finding if …" goal. Screenshot errors and timeouts are observations and do not block later product goals; all product expectations remain strict. For multi-select prompts, choose one explicit intent:
   • A default assertion lists what must already be selected and unselected, then says to leave that selection unchanged and continue. A mismatch must fail without repair.
   • A final-state instruction lists what should end selected and unselected without asserting the initial state, allowing the worker to change only the differences. An already-correct selection advances without changing any choice. Keep action names, indices, payloads, and keystrokes out of scenario goals; the worker owns those mechanics. For resource-creating flows, include the RESOURCE NAMING and AGENT NAME goals (prefix {prefix}-, suffix -{instance}) so parallel runs don't collide. Foundry project prompts must use the orchestrator-derived {foundry_project_name} because the service limits those names to 32 characters.

Validation loop (no execution)

Validate statically — never start_session, never drive the scenario, never provision.

1. Tag lint via list_scenarios. Run list_scenarios(root="<scenarios-dir>", tags=[<the new scenario's tags>]) and confirm the new file appears under each of its tags. If list_scenarios reports tags: [] for the file, the tags: field is missing or malformed — fix it. (list_scenarios uses a real YAML parser, so this also catches YAML syntax errors in the file.)
2. YAML shape. Re-read the file and confirm required fields (name, command, cwd, tags, goals) are present and well-formed, that goals is a non-empty list of strings, and that any Tier 1 producer declares the exact scaffold directory in produces:.
3. requires: resolves. If requires: is set, confirm the referenced path exists relative to the scenarios root and points at the intended prerequisite.
4. Fixture / hook paths resolve. If a pre hook seeds a fixture, confirm the referenced fixtures/<name>/ tree exists and the hook uses {fixtures_dir} (not a hardcoded path).
5. Placeholders only reference known variables. Every {name}-shaped token in command / cwd / hooks / goals is processed as a placeholder and must be a documented profile/session placeholder (including {model_version}, {model_sku}, and {foundry_project_name}) or, for Tier 1b, {prerequisite_scaffold_dir}. This includes embedded shell syntax: format an awk action as { print }, not {print}, so it cannot be mistaken for an unknown placeholder. Reject every unknown brace-delimited token during static validation.
6. Invoke input is explicit. For every goal or command that executes azd ai agent invoke, confirm the literal command includes a positional message or an intentional --input-file whose contents are defined by the scenario. Command-list and help assertions that only mention invoke are not executable calls.
7. Coverage cross-check. If this scenario closes a coverage gap flagged by a PR regression run, confirm its cmd:* tag matches the changed command so future impact-mapping picks it up.

Report the validation result to the user. If they then want to run the new scenario to confirm it drives cleanly, that's a separate, cost-gated step — hand off to the foundry-extension-scenario-suite-run skill / foundry-extension-scenario-orchestrator agent (Tier 1b / Tier 2 incur Azure cost and need explicit consent). This skill stops at a validated, unexecuted scenario.

Exit criteria

• A single-command scenario YAML was authored (or corrected) in the correct tierN/ directory with a compliant tags: list, appropriate cwd + idempotency hooks, any needed fixture / requires:, and goals: written as a literal, checkable contract.
• The static validation loop passed: list_scenarios lists the file under all its tags (not tags: []), the YAML parses, and every requires: / fixture / placeholder reference resolves.
• No scenario was executed and no Azure resources were created; any run-to-confirm was handed off to the run skills with a cost note.