AZD Tips & Patterns

Conventions and best practices for azd workflows, hooks, and ACA job deployments gathered from real repos.

ACA Job Deployment

azd does not natively deploy Container Apps Jobs — only Container Apps. Jobs need a separate deployment step.

Recommended: Python + postdeploy hook (cross-platform)

Used in newer repos. The pattern:

1. azure.yaml — wire a postdeploy hook that runs after azd deploy:

   hooks:
     postdeploy:
       shell: pwsh
       run: 'cd infra/scripts && uv sync --frozen && uv run deploy_job.py'
       interactive: false
       continueOnError: false

2. infra/scripts/deploy_job.py — update the job image via Azure SDK:

   import asyncio, os, sys, logging
   from utils import load_azd_env
   from azure.identity.aio import AzureCliCredential
   from azure.mgmt.appcontainers.aio import ContainerAppsAPIClient
   
   async def main():
       load_azd_env()
       image_name = os.getenv("SERVICE_AGENTS_IMAGE_NAME")
       job_name = os.getenv("JOB_NAME")
       resource_group = os.getenv("RESOURCE_GROUP")
       subscription_id = os.getenv("SUBSCRIPTION_ID")
       tenant_id = os.getenv("AZURE_TENANT_ID")
   
       async with AzureCliCredential(tenant_id=tenant_id) as credential:
           async with ContainerAppsAPIClient(credential, subscription_id) as client:
               job = await client.jobs.get(resource_group, job_name)
               job.template.containers[0].image = image_name
               # Remove stale sidecars — they cause "Failed" status if they exit non-zero
               job.template.containers = [job.template.containers[0]]
               poller = await client.jobs.begin_create_or_update(resource_group, job_name, job)
               await poller.result()
   
   if __name__ == "__main__":
       try:
           asyncio.run(main())
       except Exception as e:
           logging.critical("Error deploying job: %s", e)
           sys.exit(1)

3. infra/scripts/utils.py — load azd environment variables:

   import json, os, subprocess
   from pathlib import Path
   from dotenv import load_dotenv
   
   def _ensure_azd_config_dir():
       """Walk up from script dir to find repo-root .azure directory.
       Needed when azd hooks run from a temp directory."""
       if os.environ.get("AZD_CONFIG_DIR"):
           return
       search = Path(__file__).resolve().parent
       for _ in range(10):
           candidate = search / ".azure"
           if candidate.is_dir() and any(
               (candidate / d / ".env").exists()
               for d in os.listdir(candidate)
               if (candidate / d).is_dir()
           ):
               os.environ["AZD_CONFIG_DIR"] = str(candidate)
               return
           if search.parent == search:
               break
           search = search.parent
   
   def load_azd_env():
       """Load the default azd environment's .env file."""
       _ensure_azd_config_dir()
       result = subprocess.run("azd env list -o json", shell=True, capture_output=True, text=True)
       if result.returncode != 0:
           raise Exception("Error loading azd env")
       env_json = json.loads(result.stdout)
       for entry in env_json:
           if entry["IsDefault"]:
               load_dotenv(entry["DotEnvPath"], override=True)
               return
       raise Exception("No default azd env file found")

4. infra/scripts/pyproject.toml — managed by uv:

   [project]
   name = "infra-scripts"
   requires-python = ">=3.11"
   dependencies = [
       "azure-identity",
       "azure-mgmt-appcontainers",
       "python-dotenv",
   ]

Benefits over the old PowerShell approach

Legacy: PowerShell script (publish_aca.ps1)

Older repos (e.g. acme-demo-form) use a postprovision hook with a PowerShell script:

# azure.yaml
hooks:
  postprovision:
    shell: pwsh
    run: |
      $RG = azd env get-value AZURE_RESOURCE_GROUP
      $APP_NAME = azd env get-value AZURE_CAJOB_NAME
      .\infra\scripts\publish_aca.ps1 -RG $RG -APP_SOURCE_FOLDER .\src\ca-job -APP_NAME $APP_NAME -TYPE job
    interactive: true
    continueOnError: false

The script does az acr build + az containerapp job update. This works but is not cross-platform and depends on az CLI subscription state.

⚠️ Gotcha: postprovision runs only on azd provision/azd up, NOT on azd deploy. If the job needs updating on every deploy, use postdeploy instead.

Fetch-Latest-Image Pattern (Bicep + ACR)

When Bicep provisions a Container App (or Job), it needs an image reference. On first deploy the real image hasn't been built yet, so Bicep uses a placeholder. On subsequent deploys, azd deploy builds the real image into ACR and patches the running container. This is the expected lifecycle — the placeholder in Bicep is intentional, not a bug.

Why the placeholder exists

// infra/main.bicep — first-deploy bootstrap
resource mcpApp 'Microsoft.App/containerApps@2024-10-02-preview' = {
  ...
  tags: {
    'azd-service-name': 'mcp'  // REQUIRED — azd deploy uses this tag to find the resource
  }
  properties: {
    template: {
      containers: [
        {
          name: 'mcp'
          // Placeholder: azd deploy will swap this to the real ACR image
          image: 'mcr.microsoft.com/azuredocs/containerapps-helloworld:latest'
        }
      ]
    }
  }
}

⚠️ The azd-service-name tag is MANDATORY. azd deploy locates the Container App to patch by searching for a resource tagged with azd-service-name: <service> matching the service name in azure.yaml. Without this tag, azd deploy fails with "resource not found: unable to find a resource tagged with 'azd-service-name: mcp'". This is not documented prominently in the azd docs — add it to every ACA Bicep.

On azd up:

1. azd provision runs Bicep → Container App created with placeholder image
2. azd deploy builds src/mcp/ → pushes to ACR → patches the Container App with the real image
3. Subsequent azd deploy calls repeat step 2 only

Pattern A — azure.yaml service binding (recommended)

Let azd manage the image lifecycle. Declare each container as a service:

# azure.yaml
services:
  mcp:
    host: containerapp
    project: src/mcp
    docker:
      path: src/mcp/Dockerfile
      context: src/mcp

azd deploy will:

• Build the Dockerfile into ACR (tag = azd-deploy-{timestamp})
• Set env var SERVICE_MCP_IMAGE_NAME = full ACR image ref
• Patch the Container App to use the new image

azure.yaml field traps

⚠️ language: html and language: static are NOT valid azure.yaml values. They look plausible for static or front-end services but silently break azd build detection. For custom-Dockerfile services, omit language entirely and declare only project + docker.

When the Dockerfile is not in the service root, set docker.context to the actual build-context directory:

services:
  web:
    host: containerapp
    project: .
    docker:
      path: ./src/Dockerfile
      context: ./src

Without docker.context, azd deploy builds from the service root and the Dockerfile's COPY lines fail because the expected files are outside the build context.

Pattern B — postdeploy hook with SERVICE_*_IMAGE_NAME

For Container Apps Jobs (not managed by azd deploy), use a postdeploy hook that reads the azd-generated image name:

hooks:
  postdeploy:
    shell: pwsh
    run: 'cd infra/scripts && uv sync --frozen && uv run deploy_job.py'

# infra/scripts/deploy_job.py
image_name = os.getenv("SERVICE_MCP_IMAGE_NAME")  # set by azd deploy
# ... patch the Container App Job with this image

Pattern C — Bicep containerImage parameter with default

For repos where the image is pre-built (e.g., shared ACR across teams):

param containerImage string = 'mcr.microsoft.com/azuredocs/containerapps-helloworld:latest'

resource app 'Microsoft.App/containerApps@2024-10-02-preview' = {
  ...
  properties: {
    template: {
      containers: [{ name: 'app', image: containerImage }]
    }
  }
}

Override at deploy time: azd provision --parameter containerImage=myacr.azurecr.io/app:v2

The SERVICE_*_IMAGE_NAME convention

azd auto-generates env vars for each service declared in azure.yaml:

These are available in postdeploy hooks and in .azure/{env}/.env.

⚠️ Don't treat the Bicep placeholder as a bug. When reviewing Bicep that uses containerapps-helloworld:latest, check whether azure.yaml declares the corresponding service. If it does, azd deploy handles the image swap automatically. The placeholder is a bootstrap artifact, not a deployment gap.

ACA Job: silent-failure debug playbook

Symptom we keep hitting. ACA Job execution flips to Failed within 60s of the deploy completing, but zero console + system logs reach Log Analytics Workspace even though sibling ACA apps (MCP server, agent) in the same env route logs cleanly. az containerapp job logs show hangs indefinitely. This has bitten deadline-watcher cron jobs after in-process refactors and is often the biggest end-of-session blocker.

The diagnostic ladder below goes cheap → expensive. Stop at the first rung that surfaces actionable signal.

Rung 1 — Verify the basics aren't lying

$RG  = 'rg-<your-process>-poc'
$JOB = 'aca-job-deadline-watcher'

# Confirm the job exists and what image it's actually running
az containerapp job show --resource-group $RG --name $JOB `
  --query "{state:properties.runningState, image:properties.template.containers[0].image, env:properties.environmentId}" -o table

# Confirm last execution(s)
az containerapp job execution list --resource-group $RG --name $JOB `
  --query "[].{name:name, status:properties.status, start:properties.startTime, end:properties.endTime}" `
  --output table | Select-Object -First 10

Common findings:

• Image is mcr.microsoft.com/azuredocs/containerapps-helloworld — azd provision ran but azd deploy / publish_aca didn't (placeholder leak — threadlight-safe-check Step 3.5 (image-probe) catches this; check separately).
• Image hash matches latest ACR push but jobs still Failed → real runtime crash, advance to Rung 2.

Rung 2 — Probe LAW for ANY signal at all (including system events)

$WS = az monitor log-analytics workspace list --resource-group $RG `
        --query "[0].customerId" -o tsv
$JOB = 'aca-job-deadline-watcher'

# Console logs — what stdout/stderr emitted (this is the "no logs" thing if empty)
az monitor log-analytics query --workspace $WS --analytics-query @"
ContainerAppConsoleLogs_CL
| where ContainerJobName_s == '$JOB'
| order by TimeGenerated desc
| project TimeGenerated, RevisionName_s, Log_s
| take 50
"@ -o table

# System logs — ACA control plane events (start/stop/pull/probe failures)
az monitor log-analytics query --workspace $WS --analytics-query @"
ContainerAppSystemLogs_CL
| where ContainerAppName_s == '$JOB'
| order by TimeGenerated desc
| project TimeGenerated, Type_s, Reason_s, Log_s
| take 50
"@ -o table

What each empty/non-empty combo means:

Rung 3 — Force first-line logging in the entrypoint

When the ConsoleLogs pipe is empty but SystemLogs show the container started + exited, the crash is import-side. Tactic: emit a synchronous write to stderr before any other import so even an ImportError produces a visible breadcrumb.

# main.py — top of file, BEFORE any project imports
import sys
sys.stderr.write("BOOT: entrypoint reached\n"); sys.stderr.flush()
try:
    import azure.identity     # the usual suspects first
    sys.stderr.write("BOOT: azure.identity OK\n"); sys.stderr.flush()
    import azure.cosmos
    sys.stderr.write("BOOT: azure.cosmos OK\n"); sys.stderr.flush()
    # ... your imports ...
except Exception as e:
    sys.stderr.write(f"BOOT: import failed: {type(e).__name__}: {e}\n")
    sys.stderr.flush()
    raise

Push, redeploy, kick off a manual execution (az containerapp job start --resource-group $RG --name $JOB), wait ~90s, re-run the ConsoleLogs query at Rung 2. The breadcrumbs reveal which import exploded.

Rung 4 — Activity log probe (orthogonal channel — bypasses LAW entirely)

$RG = 'rg-<your-process>-poc'
$JOB = 'aca-job-deadline-watcher'
$JOB_ID = az containerapp job show --resource-group $RG --name $JOB --query id -o tsv

# Last 4 hours of activity log entries for the job resource
az monitor activity-log list `
  --resource-id $JOB_ID `
  --start-time (Get-Date).AddHours(-4).ToString('o') `
  --query "[].{time:eventTimestamp, level:level, op:operationName.value, status:status.value, msg:properties.statusMessage}" `
  -o table | Select-Object -First 20

Catches: image pull failures (ACR auth / RBAC missing on the env's UAMI), execution-create failures (quota), control-plane denials.

Rung 5 — Portal Container Apps blade (job execution detail)

The most painful diagnostic to automate, but it has signal that is NOT exposed via CLI. Path:

1. Azure Portal → Resource group → the ACA Job resource
2. Left nav → Execution history
3. Click the failed execution name
4. The right pane shows per-replica logs including init-container output and any pre-logging.basicConfig stderr — the same data that LAW would show if the routing were healthy.

Use this rung when Rungs 1–4 give you nothing actionable. Do not skip the earlier rungs — they're 1-line probes; the portal click is several minutes of navigation.

Rung 6 — Detectors API (preview — sometimes 404s)

$RG = 'rg-<your-process>-poc'
$JOB = 'aca-job-deadline-watcher'
$exec = (az containerapp job execution list --resource-group $RG --name $JOB `
          --query "[0].name" -o tsv)
az rest --method get `
  --uri "https://management.azure.com/subscriptions/$(az account show --query id -o tsv)/resourceGroups/$RG/providers/Microsoft.App/jobs/$JOB/executions/$exec/detectors?api-version=2024-03-01"

Returns 404 on many regions / SKU combos as of May 2026 — try api-version=2024-08-02-preview or 2025-01-01 if 404. When it works, surfaces structured root-cause diagnostics (e.g. "OutOfMemoryException", "ImagePullBackOff", "MaxReplicasExceeded").

Common root causes — the "if I ran out of time, what's it most likely?" table

Anti-pattern (DO NOT do)

• ❌ Don't add time.sleep(120) "just to keep the container alive" — it hides the crash. Fix the import.
• ❌ Don't switch to a long-lived containerapp app to "get logs" — ACA Jobs are the right primitive for cron; the routing IS fixable.
• ❌ Don't blindly bump replicaTimeout — if the entrypoint crashes in 5s, more time won't help.

Captured from recent cron-debug retrospectives. Future cron debugs should rerun this ladder top-to-bottom.

Escalation rule: 3-fix limit

If you have attempted 3 or more fixes and each one reveals a new problem in a different place (auth → RBAC → identity → timing → retry), STOP. This is not a configuration problem — it is an architecture problem. Each fix exposing a new failure in a different subsystem means the design assumptions are wrong.

Action: Report the 3+ failed attempts to the user with a 1-sentence diagnosis of why the fixes diverge, and ask whether to continue with the current architecture or redesign.

Verification before completion

DO NOT report deployment success until evidence is collected. azd up returning exit code 0 means Bicep succeeded — it does NOT mean the application works.

After every azd up / azd deploy, run this 5-step gate:

1. Identify the verification command (e.g., curl, az containerapp show, App Insights KQL query).
2. Run it fresh — do not rely on cached output or prior runs.
3. Read the full output — do not truncate or skip error sections.
4. Verify the output matches expected behavior (HTTP 200 with correct body, container in Running state, traces present).
5. Only then claim success — include the verification evidence in your response.

Minimum verification checklist after azd up:

Rationalisation prevention

Hooks: postprovision vs postdeploy

Tip: azd up runs both hooks in sequence (provision → deploy). azd deploy only runs postdeploy.

Bicep: ACA Job Pattern

azd deploy does not deploy ACA Jobs — it only runs postdeploy. Keep the job image update and az containerapp job start in that hook, and treat the job Bicep as the canonical module below.

MUST: Copy verbatim from references/bicep/aca-job.bicep. Do NOT redefine inline — the validator enforces single-source-of-truth. That file is the canonical ACA Job module (manual trigger + UAMI + registry identity + immutable imageDigest contract + explicit command/args/env + id / name outputs). The module is portable: runtime/deployment precondition enforcement lives in Task13's converge_image.py, not in repo-local Bicep config.

Structural usage excerpt (short form; keep the real module in the reference file):

module deadlineJob './references/bicep/aca-job.bicep' = {
  name: 'deadline-job'
  params: {
    name: '${prefix}-deadline-job'
    location: location
    environmentId: containerAppEnv.id
    imageDigest: '${acrServer}/deadline-watcher@sha256:${jobDigest}'
    containerName: 'deadline-watcher'
    command: ['python', '-m', 'app']
    args: []
    environmentVariables: []
    uamiResourceId: uami.outputs.id
    acrServer: acr.outputs.loginServer
  }
}

The module keeps the image immutable with an explicit imageDigest description contract, wires command / args / environmentVariables as arrays, defaults replicaTimeout to 300 and replicaRetryLimit to 1, and compiles copy-verbatim without repo bicepconfig.

Use postdeploy to apply the job update and trigger the manual run after the resource group deployment finishes.

Shared UAMI Pattern

All deployed resources (ACA services, ACA jobs, Azure Functions) should share one User-Assigned Managed Identity rather than using system-assigned MIs.

Why

• One identity, one set of RBAC assignments — no need to grant roles to N different principals
• Predictable — AZURE_CLIENT_ID is the same across all resources
• Debuggable — one principal to check in RBAC, one token to trace
• Portable — UAMI survives resource recreation (system MI doesn't)

Bicep

// infra/identity/uami.bicep
resource identity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: '${prefix}-id'
  location: location
}

output id string = identity.id
output clientId string = identity.properties.clientId
output principalId string = identity.properties.principalId

Wire to all ACAs

// Every ACA gets the same UAMI
identity: {
  type: 'UserAssigned'
  userAssignedIdentities: { '${uami.outputs.id}': {} }
}

// Every ACA gets AZURE_CLIENT_ID pointing to the shared UAMI
env: [
  { name: 'AZURE_CLIENT_ID', value: uami.outputs.clientId }
  // ... other env vars
]

RBAC — assign once, applies to all resources

MUST: Copy verbatim from references/bicep/rbac.bicep. Do NOT redefine inline — the validator enforces single-source-of-truth. That file is the full role-assignment module (Foundry project MI + workload UAMIs + ACR + Foundry account, all pinned by GUID).

Why each line matters:

• Role GUIDs are stable across tenants — never ship '...' as a placeholder. Bicep compiles but RoleDefinitionDoesNotExist blows up at apply-time, after every other resource has provisioned. Built-ins referenced in the module: Foundry User (renamed from Azure AI User in May 2026; GUID unchanged) = 53ca6127-db72-4b80-b1b0-d745d6d5456d, Search Index Data Reader = 1407120a-92aa-4202-b7e9-c0e197c71c8f, AcrPull = 7f951dda-4ed3-4680-a7ca-43fe172d538d.
• principalType: 'ServicePrincipal' is mandatory for MI-based deploys; see § CI/CD trap below for why 'User' breaks GHA / ADO pipelines.
• One module, one dependsOn:[rbac] from every consumer — see § Prefer Bicep dependsOn: [rbac] below for the wiring pattern that pre-empts the AcrPull propagation race.

CI/CD trap — PrincipalTypeNotSupported

When the deploy runs from GitHub Actions or Azure DevOps, Bicep role assignments that emit principalType: 'User' often fail with PrincipalTypeNotSupported. For MI-based deploys, set allowUserIdentityPrincipal: false in the shared roleAssignment Bicep module or force principalType: 'ServicePrincipal'. Use User only when binding a real human principal.

Exception: Hosted agent containers

Hosted agent containers get a platform-managed dedicated identity (instance + blueprint) at deploy time — you don't assign a UAMI to them. The shared UAMI is for everything else: bot ACA, MCP ACA, jobs, hooks, etc.

Cross-skill ownership: UAMI & AcrPull

Ownership split: UAMI and AcrPull guidance is split across two SKILLs:

• Foundry project system MI (the identity used by Foundry to pull agent container images) → owned by foundry-hosted-agents § Identity & RBAC
• ACA workload UAMIs (backend, frontend, job containers, hooks) → owned by this SKILL

When in doubt: project MI RBAC → foundry-hosted-agents; workload UAMI RBAC → azd-patterns.

uv for Hook Scripts

Use uv for Python dependency management in hook scripts:

hooks:
  postdeploy:
    shell: pwsh
    run: 'cd infra/scripts && uv sync --frozen && uv run deploy_job.py'

• uv sync --frozen — install deps from lockfile (deterministic, fast)
• uv run deploy_job.py — run script in the managed venv
• No need to manually create/activate venvs
• Cross-platform (Windows/Linux/Mac)

azd env set + JSON arrays/objects: use .bicepparam, not JSON parameters

🔴 DO NOT pass JSON arrays via azd env set. The value gets triple-escaped through shell → azd → Bicep → ARM. Use .bicepparam files or --parameters instead.

🛑 DO NOT push array- or object-typed values through azd env set. The CLI re-escapes the value on every .env read/write, producing triple-escaped JSON that fails to round-trip. DO hard-code the literal in infra/main.bicepparam instead — that is the only safe place for array/object Bicep parameters.

Even worse: the azd ai agent extension auto-runs azd env set AI_PROJECT_DEPLOYMENTS '[...]' on EVERY azd provision (silently). So even if you set AI_PROJECT_DEPLOYMENTS cleanly by hand once, the next provision will re-mangle it. Using readEnvironmentVariable(...) + json(...) for this specific param is therefore guaranteed to fail eventually — hardcode the literal.

Don't put JSON arrays or objects through azd env set. The CLI re-escapes the value on every read/write of .azure/<env>/.env, so a clean PUT like:

azd env set AI_PROJECT_DEPLOYMENTS '[{"name":"gpt-5.4-mini","model":{"name":"gpt-5.4-mini","format":"OpenAI","version":"2026-03-17"},"sku":{"name":"GlobalStandard","capacity":100}}]'

lands in .env as triple-escaped JSON ([{\\\"name\\\":\\\"gpt-5.4-mini\\\",…}]) and fails to round-trip through azd env get-value, through main.parameters.json ${VAR} substitution, and through Bicep's json() function. Provisioning fails with invalid character '\\' looking for beginning of object key string or invalid character '$' looking for beginning of value. Verified on azd 1.25.0 (May 2026 fruocco pilot).

Workaround: use a .bicepparam file (infra/main.bicepparam) and either (a) hard-code the array literal in Bicep syntax when the value is a spec-level decision, or (b) read structured env vars via readEnvironmentVariable(…) paired with json(…) for things that must be runtime-configurable.

MUST: Copy verbatim from references/bicep/main.bicepparam. Do NOT redefine inline — the validator enforces single-source-of-truth. That file is the canonical using './main.bicep' parameter file with the literal-array aiProjectDeployments (closes A1) at capacity: 30 (closes MID-9). It is intentionally skipped by az bicep build in CI because using can't resolve in isolation.

🔴 DO NOT default model capacity to 100 K TPM in a shared subscription. Shared subs often run at 900+/1000 K TPM aggregate quota; a fresh azd provision with capacity: 100 fails with InsufficientQuota mid-deploy and the only fix is to lower the capacity and re-provision (~4 min wasted). Default to capacity: 30 for pilot deployments; preflight with az cognitiveservices usage list --location <region> and raise only after confirming free quota. The 30 K TPM ceiling also forces realistic demo pacing (~80 s between scenario calls) — useful for catching retry-handling bugs early.

// Option B (NOT in the reference file): structured env for genuinely runtime-configurable values.
// Set as a single-quoted JSON string in `.azure/<env>/.env` by HAND-EDITING
// the file once (not via `azd env set`) — azd will preserve it on subsequent
// reads, but every `azd env set` of any value re-mangles it.
// param secondaryDeployments = json(readEnvironmentVariable('SECONDARY_DEPLOYMENTS', '[]'))

In azure.yaml's infra: section, point to the .bicepparam file instead of the .json parameters file (azd accepts either when present at the standard location).

Plain string/bool/number env vars are fine to use with azd env set — the bug is array/object-typed values only.

Composable Bicep Module Library

Private network composition

Use private-dns-vnet-link.bicep for an additive resolution link between an existing Private DNS zone and VNet. The module declares only the child link, disables registration and does not enable public DNS fallback. It does not recreate the shared zone or VNet. Invoke it at the zone's resource-group scope. Before deployment, verify the link name is new or explicitly owned, inspect what-if, and record the exact created ID for targeted cleanup. A successful link is not proof of private runtime connectivity: verify DNS and requests from their real source.

For an approved internal ACA composition, use the remaining narrow modules: aca-subnet.bicep adds a delegated subnet with an existing NAT gateway; aca-environment.bicep creates only an internal Consumption environment and its diagnostic setting; aca-private-dns.bicep maps its default domain to the ILB; uami.bicep creates a workload identity; acr-pull.bicep grants only AcrPull at an existing legacy-permissions registry. Do not use that role module with ABAC-enabled registries. Confirm existing grants before creating new ones. Compose identity -> pull grant -> application dependencies explicitly. No module changes shared routes, NSGs, policy exemptions or Foundry hosts. Record the automatically managed environment resource group for cleanup; do not remove its locks or unrelated infrastructure.

The Basic private-network template also explicitly creates a project capability host: foundry-project-capability-host.bicep. Use this no-BYO-connections shape only after approving the project-wide infrastructure change and checking the project's existing hosts. It is not a proven remedy for every private MCP DNS failure and must not recreate an account host or silently migrate existing backing stores.

threadlight-deploy Phase 6 (Module Composer) reads SPEC § 11c and includes exactly the right Bicep modules. This section catalogs the modules and what SPEC inputs select them.

MUST: Use the canonical reference Bicep modules vendored from the 2026-05-29 smb-credit-memo pilot (live-deployed in swedencentral, agentic-loop SKILL Validation history row 8). Do NOT redefine them inline in this SKILL — each file is single-source-of-truth and is cross-linked from the section that explains it:

• references/bicep/foundry-account.bicep — Foundry account + project + capabilityHost (account-level FIRST per MID-10) + model deployment + AppIn connection with explicit metadata.ConnectionString (per MID-8) + endpoints['AI Foundry API'] output (per MID-2).
• references/bicep/aca-app.bicep — canonical 3-service-shape ACA container app (consumed by § ACR + ACA Registry Binding and § Prefer Bicep dependsOn: [rbac]).
• references/bicep/rbac.bicep — all role assignments pinned by GUID (consumed by § RBAC — assign once).
• references/bicep/main.bicepparam — literal-array aiProjectDeployments parameter file (consumed by § azd env set + JSON arrays/objects).

Module catalog

Every module lives at infra/modules/<name>.bicep in the generated repo. Ship them as a single self-contained Bicep file per module (no nested modules) so they can be vendored independently.

Selector grammar (SPEC § 11c)

The SPEC § 11c table is read as a selector table — kebab-case rows mapping module name → yes/no + optional params. The composer iterates rows where selected: yes and includes only those modules. The vocabulary is the source of truth for both this catalog and threadlight-deploy Phase 6.

# specs/SPEC.md § 11c (excerpted, canonical kebab-case shape)

| Module             | Selected? | Purpose / params                                            |
| `cosmos-db`        | yes       | sku: serverless                                              |
| `ai-search`        | yes       | sku: basic                                                   |
| `foundry-iq-index` | yes       | knowledgeBases: [{ name: kyc-policies, sources: [...] }]    |
| `azure-vision`     | yes       | model: gpt-5.4-mini                                          |
| `aca-job`          | yes       | [{ name: sla-watcher, imageDigest: acr.example.io/sla-watcher@sha256:<digest>, command: ['python', '-m', 'app.job_worker'] }] |
| `aca-mcp`          | yes       | [{ name: customer-data-mcp, image: customer-mcp:latest }]   |
| `aca-bot`          | yes       | manifest: ...                                                |
| `key-vault`        | no        | (threadlight: keyless-by-mandate)                            |

Resulting infra/main.bicep includes (in canonical inclusion order): uami → acr → app-insights → cosmos-db → ai-search → foundry-iq-index → azure-vision → doc-intel → azure-speech → storage-blob → event-grid → service-bus → aca-mcp → aca-job → foundry-account → aca-bot.

key-vault is NOT in the always-include set. Threadlight pilots are keyless end-to-end. Include key-vault ONLY when SPEC § 11c explicitly selects key-vault: yes for an external integration that demands a literal API key.

Cosmos firewall — pilot-grade defaults (the trap that costs ~45 min on every fresh PoC)

Observed in recent pilots. Azure's Cosmos default is publicNetworkAccess: Disabled — every ACA→Cosmos write returns Forbidden until manually patched, even with managed identity + correct data-plane RBAC. cosmos-db.bicep MUST default to pilot-friendly posture:

@description('Pilot posture: enables public network + AzureServices bypass. Set false for prod-grade (private endpoint).')
param pilotPosture bool = true

@description('Explicit IP allowlist for ACA egress (pilots: leave empty initially, then re-deploy with the IP from the Cosmos Forbidden error). Required because AzureServices bypass does NOT cover ACA traffic — see foundry-mcp-aca.')
param ipAllowlist array = []

resource cosmos 'Microsoft.DocumentDB/databaseAccounts@2024-12-01-preview' = {
  // ...
  properties: {
    publicNetworkAccess: pilotPosture ? 'Enabled' : 'Disabled'
    networkAclBypass:    pilotPosture ? 'AzureServices' : 'None'
    ipRules: [for ip in ipAllowlist: { ipAddressOrRange: ip }]
    disableLocalAuth: true
    // ...
  }
}

Pilot-only — gotcha: AzureServices bypass alone is NOT sufficient for ACA → Cosmos traffic in swedencentral (and likely other regions). The full operator runbook + post-deploy IP discovery commands live in foundry-mcp-aca SKILL.md § "Cosmos firewall + ACA egress (the trap that wastes 45 min on every fresh PoC)".

Why composable modules (not one big main.bicep)

• Per-process repos vary widely — KYC needs doc-intel + speech + foundry-iq; Order Fallout needs service-bus + aca-job + aca-mcp. A single template would bury 70% of the file in if blocks per module.
• Customer can adopt modules incrementally — they may already have a Cosmos account they want to reuse; replacing one module file is cleaner than editing a monolith.
• Each module is independently versionable — when AI Search v2025-09 ships a new index schema, only ai-search.bicep changes.

How a new module gets added to the library

1. Author the module Bicep at infra/modules/<name>.bicep (single self-contained file)
2. Document its outputs (must be stable — they're the wire format between modules)
3. Add a row to the catalog table above
4. Update threadlight-deploy Phase 6's selector parser to recognize the new key
5. If the module wires a runtime service (cosmos, search, vision, etc.), add a entry to the environment field of the selected hosted contract so the agent receives endpoints

RG Tagging — MCAPS Subscriptions (SecurityControl: Ignore)

MCAPS (Microsoft Customer & Partner Solutions — the internal subscription class GBBs and the field use for customer demos / PoCs / tenant-specific work) ships with org-wide Defender for Cloud + Azure Policy assignments that aggressively flag and (in some MCAPS rings) auto-remediate resources the security team considers risky. For pilot work this manifests as:

• Cosmos / Storage / KV firewalls flipped back to Disabled minutes after azd up finishes — every demo breaks the next morning.
• Container Apps with ingress.external = true flagged as critical findings on the team's compliance dashboard.
• Microsoft.Security policy emails landing in the deal lead's inbox the day before the customer call.

The supported escape hatch is the SecurityControl: Ignore tag. Applied at the resource-group level, every Defender / Policy assignment that reads it skips the RG and everything inside it. This is explicitly sanctioned for non-production subscriptions (PoC, demo, pre-sales sandbox) — never apply it in a customer's own production landing zone.

Scope: this section applies ONLY to Microsoft-internal MCAPS subscriptions. In a customer tenant, leave the RG untagged and let the customer's own governance posture apply.

Three ways to set it (pick one — they compose)

1 — Bicep, subscription-scope RG creation (preferred for azd repos)

When azd up creates the RG via a subscription-scope deployment, set the tag on the RG resource itself:

// infra/main.bicep — targetScope = 'subscription'
@description('When true (default for MCAPS pilot subs), tags the RG so Defender for Cloud / Azure Policy skip it. Set false in customer-owned subscriptions.')
param mcapsPilotPosture bool = true

@description('Free-form tags to merge onto the RG and propagate to every resource.')
param tags object = {}

var baseTags = union(tags, mcapsPilotPosture ? { SecurityControl: 'Ignore' } : {})

resource rg 'Microsoft.Resources/resourceGroups@2024-03-01' = {
  name: rgName
  location: location
  tags: baseTags
}

Then thread tags: baseTags into every resource-group-scope module call so resources inherit the tag too (some Defender controls evaluate at resource scope, not RG scope):

module workload './modules/main.bicep' = {
  scope: rg
  name: 'workload'
  params: {
    tags: baseTags
    // ...
  }
}

Every leaf module's resource declarations should accept tags and pass it through:

// infra/modules/<any>.bicep
param tags object = {}

resource cosmos 'Microsoft.DocumentDB/databaseAccounts@2024-12-01-preview' = {
  // ...
  tags: tags
  // ...
}

2 — AZURE_TAGS (honored by azd for resource-group-scope deploys)

If infra/main.bicep is resource-group-scope (i.e., azd creates the RG itself rather than a subscription-scope template), set the tag via the azd environment so azd provision applies it:

azd env set AZURE_TAGS '{"SecurityControl":"Ignore","Workload":"<pilot-name>"}'
azd provision

azd reads AZURE_TAGS and applies it to the RG it creates. This does not propagate to child resources automatically — pair with the Bicep tags param-threading shown above for full coverage.

3 — az group update postprovision hook (fallback for pre-existing RG)

When the RG was created out-of-band (Cowork sandbox, manual az group create, BYO landing zone) and you can't reshape the deploy:

# azure.yaml
hooks:
  postprovision:
    shell: pwsh
    run: |
      az group update --name $env:AZURE_RESOURCE_GROUP `
        --set tags.SecurityControl=Ignore `
        --output none

This is the least invasive option but only tags the RG, not the resources inside. Defender controls that evaluate at resource scope (Cosmos firewall, Storage public access, KV soft-delete) may still fire.

Verifying it landed

# RG tag present?
az group show --name $env:AZURE_RESOURCE_GROUP `
  --query "tags.SecurityControl" -o tsv      # → "Ignore"

# Resource-level tag present? (spot-check the chattiest resource type)
az resource list --resource-group $env:AZURE_RESOURCE_GROUP `
  --query "[?type=='Microsoft.DocumentDB/databaseAccounts'].tags.SecurityControl" -o tsv

Defender / Policy honor the tag on the next evaluation cycle (typically ≤ 1 hour). If a finding was already open before the tag was applied, manually dismiss it once — it won't re-open.

Cross-skill applicability

Every azd-based deploy in this catalog should expose mcapsPilotPosture

• tags the same way:

• threadlight-deploy — Phase 6 module composer threads tags into every selected module from the catalog above (when present in SPEC § 11c or via AZURE_TAGS).
• foundry-vnet-deploy — templates/main.bicep accepts tags (manual apply via Step 8 interview); when the deploy lands in an MCAPS sub, the operator should answer yes to the implicit MCAPS prompt.
• foundry-mcp-aca, foundry-teams-bot, citadel-spoke-onboarding — all consume the same module library; the tag rides along automatically when tags is passed at the parent scope.

Don't hard-code the tag value. Apply SecurityControl: Ignore only when mcapsPilotPosture is true. Letting it leak into a customer tenant's deployment files is a finding in its own right.

Input contract / Output artifacts

ACA Persistent Volume (Azure Files)

When an ACA container needs persistent storage across restarts (e.g., embedded databases like KuzuDB, file caches, local state), use an Azure Files volume mount. This is the simplest persistent storage pattern for ACA — no Cosmos DB, no Blob SDK, just a filesystem path.

Bicep module: aca-volume.bicep

// infra/modules/aca-volume.bicep
@description('Storage account for ACA volumes')
param storageAccountName string
param location string = resourceGroup().location
param shareName string = 'aca-data'
param shareQuotaGb int = 5

resource storage 'Microsoft.Storage/storageAccounts@2023-05-01' = {
  name: storageAccountName
  location: location
  kind: 'StorageV2'
  sku: { name: 'Standard_LRS' }
}

resource fileServices 'Microsoft.Storage/storageAccounts/fileServices@2023-05-01' = {
  parent: storage
  name: 'default'
}

resource share 'Microsoft.Storage/storageAccounts/fileServices/shares@2023-05-01' = {
  parent: fileServices
  name: shareName
  properties: {
    shareQuota: shareQuotaGb
  }
}

output storageAccountName string = storage.name
output shareName string = share.name
output storageAccountKey string = storage.listKeys().keys[0].value

ACA environment volume binding

In the ACA environment Bicep, register the storage as a volume:

// In the ACA app resource
resource app 'Microsoft.App/containerApps@2024-03-01' = {
  // ...
  properties: {
    template: {
      volumes: [
        {
          name: 'persistent-data'
          storageType: 'AzureFile'
          storageName: acaEnvStorage.name  // registered on the ACA env
        }
      ]
      containers: [
        {
          // ...
          volumeMounts: [
            {
              volumeName: 'persistent-data'
              mountPath: '/data'           // KuzuDB writes here
            }
          ]
        }
      ]
    }
  }
}

Register storage on ACA environment

resource acaEnvStorage 'Microsoft.App/managedEnvironments/storages@2024-03-01' = {
  parent: acaEnvironment
  name: 'persistent-data'
  properties: {
    azureFile: {
      accountName: storageAccount.name
      accountKey: storageAccount.listKeys().keys[0].value
      shareName: fileShare.name
      accessMode: 'ReadWrite'
    }
  }
}

When to use

• Embedded databases (KuzuDB, SQLite) that must survive container restarts
• File caches that are expensive to rebuild
• Demo data snapshots that persist between azd deploy cycles

When NOT to use

• High-throughput data → use Cosmos DB
• Large datasets (>5 GB) → use Blob Storage via SDK
• Multi-replica writes → Azure Files is ReadWriteMany but NOT designed for concurrent high-frequency writes from multiple containers

azd up is THE deployment path

⚠️ azd up (= azd provision + azd deploy) is the default and preferred way to deploy ACA workloads. It handles Bicep provisioning, Docker image build, ACR push, and ACA image swap in one command.

Raw az deployment sub create + az acr build + az containerapp update is an escape hatch only — use it when azd is unavailable (e.g., CI without azd installed, or debugging a specific Bicep module). Never use raw az as the primary deploy path when azd is available.

ACR + ACA Registry Binding (complete example)

Preflight the registry's authorization mode before build. The library's acr-pull.bicep is a legacy registry-wide grant, not repository isolation. Repository-only requirements need verified ABAC conditions and no broader effective grants; an incompatible existing registry is an owner decision. Never change its mode or fall back to broad roles automatically. Image-pull identity, deployer and result reader are separate subjects.

MUST: Every ACA image-pull identity needs the mode-appropriate permission: AcrPull only for a legacy registry, or the appropriate repository-reader role/condition for an ABAC registry. Verify every service separately; a deployer's successful push is not evidence of runtime pull authorization.

🛑 MUST rule. In a multi-service pilot, every ACA app's UAMI independently needs AcrPull on the ACR — not just a "shared env UAMI" or the Foundry project MI (which only covers the agent container). Granting AcrPull to ONE identity and assuming it covers the others leaves frontend / backend revisions stuck on the placeholder mcr.microsoft.com/k8se/quickstart:latest image, with FetchingKeyVaultSecretFailed: 401 in ContainerAppSystemLogs_CL (yes, that's the wrong category — the ACA system component mis-categorises image-pull credential failure as a Key Vault error). Verified live in the 2026-05-28 learn-assistant pilot.

When azd deploy pushes an image to ACR, the ACA must be configured to pull from that ACR. This requires three things in Bicep: (1) the ACR resource, (2) the ACA with registries[] + secrets[] + azd-service-name tag, and (3) the AZURE_CONTAINER_REGISTRY_ENDPOINT output for azd.

MUST: Copy verbatim from references/bicep/aca-app.bicep. Do NOT redefine inline — the validator enforces single-source-of-truth. That file is the canonical 3-service-shape ACA module (UAMI + registry binding + placeholder image), live-deployed across the 2026-05 pilots (weather-agent, learn-assistant, smb-credit-memo, hybrid-mcp-agent).

Without all three pieces, azd deploy pushes to ACR successfully but the ACA fails to pull the image (401 unauthorized). The placeholder image: lets the first revision land before the real image is built; azd deploy then patches it. The tags: { 'azd-service-name': 'agent' } is what binds this Bicep resource to the services: entry in azure.yaml — missing or mistyped, azd deploy skips this app silently.

AcrPull propagation retry loops (post-azd provision, pre-azd deploy)

💡 Preferred fix first. If your main.bicep invokes a separate rbac module + ACA-app modules in the same deployment, use Bicep dependsOn: [rbac] on every ACA-app module to pre-empt the race at provision time. See § Prefer Bicep dependsOn: [rbac] below for the canonical pattern. The retry loop here is the fallback for cross-deployment scenarios (e.g. role granted between provision and deploy, or RBAC propagation lags ARM completion).

If azd provision grants AcrPull to the ACA environment or shared UAMI, do not immediately run azd deploy. RBAC propagation can lag by a few minutes, so the first deploy fails with ImagePullError even though the assignment exists. Poll the role assignment before proceeding:

# Wait for AcrPull RBAC propagation before deploying
MAX_RETRIES=5; SLEEP=60
for i in $(seq 1 $MAX_RETRIES); do
  ROLE=$(az role assignment list --assignee "$IDENTITY_PRINCIPAL_ID" --role "AcrPull" --scope "$ACR_ID" --query "[0].roleDefinitionName" -o tsv 2>/dev/null)
  [ "$ROLE" = "AcrPull" ] && echo "✅ AcrPull propagated" && break
  echo "⏳ AcrPull not yet visible (attempt $i/$MAX_RETRIES), waiting ${SLEEP}s..."
  sleep $SLEEP
done
[ "$ROLE" != "AcrPull" ] && echo "❌ AcrPull not propagated after $((MAX_RETRIES * SLEEP))s" && exit 1

# Wait for AcrPull RBAC propagation before deploying
$maxRetries = 5; $sleep = 60
for ($i = 1; $i -le $maxRetries; $i++) {
    $role = az role assignment list --assignee $identityPrincipalId --role "AcrPull" --scope $acrId --query "[0].roleDefinitionName" -o tsv 2>$null
    if ($role -eq 'AcrPull') { Write-Host "✅ AcrPull propagated"; break }
    Write-Host "⏳ AcrPull not yet visible (attempt $i/$maxRetries), waiting ${sleep}s..."
    Start-Sleep -Seconds $sleep
}
if ($role -ne 'AcrPull') { Write-Error "❌ AcrPull not propagated after $($maxRetries * $sleep)s"; exit 1 }

Prefer Bicep dependsOn: [rbac] over the retry loop when both modules live in the same deployment

MUST: Add dependsOn: [rbac] to every ACA module that references a UAMI. Without it, azd up races RBAC propagation and the first deploy fails with 401.

The retry loop above patches the symptom at deploy time. The cleaner fix is to pre-empt the race at provision time: when main.bicep invokes a separate rbac module (the canonical pattern when role assignments are batched) and ACA-app modules in parallel, Bicep does not infer a dependency between them unless one references the other's outputs.

Add an explicit dependsOn: [rbac] to every ACA-app module invocation so the ARM engine waits for role assignment PUTs to return before starting the container app provision:

// infra/main.bicep
module rbac './modules/rbac.bicep' = {
  name: 'rbac'
  scope: rg
  params: {
    foundryProjectPrincipalId: foundry.outputs.projectMiPrincipalId
    backendUamiPrincipalId:    backendUami.outputs.principalId
    frontendUamiPrincipalId:   frontendUami.outputs.principalId
    acrName:                   acr.outputs.name
    foundryAccountName:        foundry.outputs.accountName
  }
}

module backendApp './modules/aca-app.bicep' = {
  name: 'backend'
  scope: rg
  dependsOn: [ rbac ]   // ← waits for AcrPull to be assigned before first revision
  params: { /* ... */ }
}

module frontendApp './modules/aca-app.bicep' = {
  name: 'frontend'
  scope: rg
  dependsOn: [ rbac ]   // ← same
  params: { /* ... */ }
}

The retry loop is still the right fallback for deploy-time scenarios (e.g. the user grants a new role between azd provision and azd deploy, or RBAC propagation lags ARM completion as it occasionally does in swedencentral / eastus2). Use dependsOn first; keep the retry loop as belt-and-braces.

Two failure surfaces this prevents (both verified on real pilots, May 2026):

• Frontend / backend ACA app stuck at mcr.microsoft.com/k8se/quickstart:latest placeholder after azd deploy: the first revision tried to pull the real image before AcrPull existed, fell back to the placeholder, and azd reported Success.
• Cryptic FetchingKeyVaultSecretFailed: 401 in ContainerAppSystemLogs_CL after the first revision (it's actually the image-pull credential failing — the error is mis-categorised by the ACA system component).

az deployment sub validate with .bicepparam — pass the .bicep directly, NOT the compiled .json

🛑 DO NOT pre-compile your Bicep to ARM JSON and then validate with --template-file infra/main.json --parameters infra/main.bicepparam. Recent az (verified on 2.86 / azd 1.25, May 2026) rejects this combination with: "Only a .bicep template is allowed with a .bicepparam file".

DO pass infra/main.bicep directly:

az deployment sub validate \
  --location <region> \
  --template-file infra/main.bicep \
  --parameters infra/main.bicepparam

az compiles the Bicep itself as part of the validation pipeline; one fewer step and the only shape .bicepparam validates cleanly against.

This shows up as a noisy red error during automated Verify pipelines (any CI workflow that does plan-coherence checks before provisioning). The fix is one CLI argument; no Bicep / parameter rewrites needed.

azd reformats azure.yaml to 4-space indent after first deploy

azd 1.25 silently rewrites azure.yaml from your authored indent (typically 2-space) to 4-space after the first successful azd deploy <service>. The file content is semantically identical but any tool that parses azure.yaml with a hand-rolled 2-space-only YAML regex (e.g. ad-hoc shell/Python scripts that grep for ^ <name>: or ^ host:) will break on the second invocation.

Fix in any tool that parses azure.yaml: make the indent detection dynamic. Read the first non-blank indented line under services:, take its indent as the service-name indent, then anything deeper is a property.

Verified live on 2026-05-28 (azd 1.25, hosted-agent pilot): first azd deploy left azure.yaml at the authored 2-space indent; the second azd deploy (same project, same env) rewrote it to 4-space. azd reports no diff in azd env get-values.

💡 If your tool MUST stay 2-space-strict, lock azure.yaml after first deploy with chattr +i (Linux) / file system attributes (macOS/Win), but that's a workaround for poor parsers; fix the parser instead.

ACA Container Debugging — check logs FIRST

⚠️ When an ACA container fails, ALWAYS check system logs before retrying. Port mismatches, startup probe failures, image pull errors, and OOM kills are all visible in system logs. Retrying without reading logs wastes time.

Canonical owner of container-debugging diagnostics: This section (§ ACA Container Debugging) is the authoritative reference for container-failure diagnosis. Sibling SKILLs (foundry-hosted-agents, foundry-observability, etc.) should link here rather than duplicate.

# System logs — infrastructure events (port mismatch, probe fail, OOM)
az containerapp logs show -g <rg> -n <app> --type system --tail 20

# Console logs — application stdout/stderr
az containerapp logs show -g <rg> -n <app> --type console --tail 20

Common failures and what they look like in logs

ACA Scale-to-Zero Gotcha

minReplicas: 0 means the container doesn't start until the first request arrives. This is efficient for production but confusing during deploy testing — you deploy, curl the endpoint, get a timeout, and think the deploy failed when it's actually just cold-starting.

For pilots and deploy testing: use minReplicas: 1 so the container starts immediately after deploy. Switch to 0 for production when you want cost savings.

scale: {
  minReplicas: 1   // 1 for pilots/testing, 0 for production cost savings
  maxReplicas: 3
}

Azure Tenant Isolation (mandatory preflight)

⚠️ Before ANY az or azd command in this skill, verify tenant isolation per the azure-tenant-isolation skill. Set AZURE_CONFIG_DIR + AZD_CONFIG_DIR, assert the tenant + subscription with az account show, then proceed. This is the #1 cause of "deployed to the wrong subscription" incidents.

See Also

Note: External threadlight-skills links (threadlight-deploy, threadlight-design, threadlight-event-triggers) point to a separate repository. For offline or agent-context use, abbreviated summaries of those skills are often included in local threadlight-skills/SKILL.md installations.