Foundry Observability

End-to-end telemetry across every component of a Threadlight pilot: Foundry hosted agent, MCP servers on ACA, ACA jobs (cron triggers), bot service, workspace UI. Default discipline, not optional.

For the complete per-agent adoption and release-evidence workflow, see foundry-agentops. This skill remains authoritative for OpenTelemetry and App Insights wiring; AgentOps aggregates evidence and never replaces instrumentation or verification of the telemetry path.

Downstream FinOps consumer. foundry-cost-monitoring joins the gen_ai.usage.* spans this skill emits with the Azure Retail Prices API to compute per-agent / per-project / per-tenant cost projection — wire it whenever a FinOps stakeholder needs to answer "what is this agent costing us right now?"

Why this skill exists. Recent pilots deployed cleanly (azd up returned 0, all resources provisioned) but App Insights stayed completely empty — no agent traces, no MCP tool calls, no cron logs. Root cause: no one wired the connection at any layer. The intel for each layer lives scattered across threadlight-deploy, foundry-hosted-agents, foundry-mcp-aca, threadlight-event-triggers — but no single skill walks an operator through the full chain. That's what this skill does. Pair with threadlight-safe-check Step 5.6 (App Insights existence + first-trace probe) to gate it shut.

Mental model — three layers, one signal

┌─────────────────────────────────────────────────────────────────────┐
│ Layer 3: ACA workloads (MCP / bot / workspace / cron jobs)          │
│   • configure_azure_monitor() reads APPLICATIONINSIGHTS_CONNECTION_STRING │
│   • Env var set by Bicep from app-insights.outputs.connectionString  │
│   • OTel exporter ships spans + logs + metrics over HTTPS           │
└─────────────────────────────────────────────────────────────────────┘
                                  ▲
                                  │ direct push from container code
                                  │
┌─────────────────────────────────┼───────────────────────────────────┐
│ Layer 2: Foundry hosted agent (the runtime)                         │
│   • Account-level AppInsights connection (category: AppInsights)    │
│   • Platform AUTO-INJECTS APPLICATIONINSIGHTS_CONNECTION_STRING     │
│   • RBAC: Monitoring Metrics Publisher on agent identities           │
│   • Tracing emitted by the runtime — no app code change             │
└─────────────────────────────────┼───────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│ Layer 1: Bicep substrate                                            │
│   • app-insights.bicep — workspace-based (LAW-bound)                │
│   • log-analytics.bicep — single LAW for ALL workloads in the RG    │
│   • ACA env wiring: dapr.appInsightsConnectionString OR direct env  │
│   • Output `connectionString` consumed by every workload            │
└─────────────────────────────────────────────────────────────────────┘

Single connection string, three fan-outs. All telemetry lands in the same App Insights resource. No per-workload AppIn — that fragments the trace graph and makes correlation impossible.

What ships from this skill

foundry-observability/
├── SKILL.md
└── references/
    ├── bicep/
    │   ├── log-analytics.bicep          # LAW (workspace) — required by both AppIn and ACA env
    │   ├── app-insights.bicep           # AppIn workspace-based + UAMI Monitoring Metrics Publisher RBAC
    │   └── aca-env-monitoring.bicep     # ACA env wired to LAW + AppIn
    ├── python/
    │   └── otel_init.py                 # configure_azure_monitor() for ACA workloads
    ├── postprovision/
    │   └── connect_foundry_appinsights.py  # creates the account-level AppInsights connection
    └── queries/
        ├── agent-traces.kql             # hosted-agent traces, last 1h
        ├── mcp-tool-calls.kql           # MCP tool invocation breakdown
        ├── silent-cron-debug.kql        # ACA Job exec failures with no console logs
        └── first-trace-probe.kql        # smoke query — "did ANY trace land in last 5 min?"

Drop these into a PoC's infra/modules/, infra/scripts/, src/<svc>/, and docs/queries/ respectively. The patterns work as-shipped — replace parameter values, re-deploy, traces flow.

Layer 1 — Bicep substrate

Step 1.1 — Single LAW for the whole pilot

// infra/modules/log-analytics.bicep — drop-in
@description('Log Analytics workspace. ONE per pilot. AppIn binds to it; ACA env binds to it; cron jobs ship console+system logs to it.')
param location string = resourceGroup().location
param name string

resource law 'Microsoft.OperationalInsights/workspaces@2023-09-01' = {
  name: name
  location: location
  properties: {
    sku: { name: 'PerGB2018' }
    retentionInDays: 30
    features: { enableLogAccessUsingOnlyResourcePermissions: true }
  }
}

output workspaceId string = law.id           // ARM ID — used by AppIn + ACA env
output customerId string = law.properties.customerId  // GUID — for KQL queries
output workspaceName string = law.name

Step 1.2 — App Insights bound to that LAW

// infra/modules/app-insights.bicep — drop-in
@description('App Insights component. workspace-based (legacy classic mode is deprecated). Auto-grants Monitoring Metrics Publisher to the workload UAMI so OTel traces, logs, and metrics can be ingested keylessly.')
param location string = resourceGroup().location
param name string
param workspaceId string
param uamiPrincipalId string  // Foundry agent UAMI principal — for RBAC

resource appin 'Microsoft.Insights/components@2020-02-02' = {
  name: name
  location: location
  kind: 'web'
  properties: {
    Application_Type: 'web'
    WorkspaceResourceId: workspaceId
    DisableLocalAuth: true        // RBAC-only ingestion — keyless mandate
    publicNetworkAccessForIngestion: 'Enabled'
    publicNetworkAccessForQuery: 'Enabled'
  }
}

// Monitoring Metrics Publisher — required for OTel trace/log/metric ingestion
// when DisableLocalAuth is true. Has both Microsoft.Insights/Metrics/Write and
// Microsoft.Insights/Telemetry/Write dataActions.
// Role GUID: 3913510d-42f4-4e42-8a64-420c390055eb (well-known)
resource dataIngestor 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  scope: appin
  name: guid(appin.id, uamiPrincipalId, '3913510d-42f4-4e42-8a64-420c390055eb')
  properties: {
    principalId: uamiPrincipalId
    principalType: 'ServicePrincipal'
    roleDefinitionId: subscriptionResourceId(
      'Microsoft.Authorization/roleDefinitions',
      '3913510d-42f4-4e42-8a64-420c390055eb'
    )
  }
}

output id string = appin.id
output name string = appin.name
output connectionString string = appin.properties.ConnectionString
output instrumentationKey string = appin.properties.InstrumentationKey

Why DisableLocalAuth: true. Threadlight pilots are keyless by mandate (see azure-tenant-isolation and citadel-spoke-onboarding). AppIn ingestion keys are a back-door around RBAC — disable them and rely on Monitoring Metrics Publisher (GUID 3913510d-42f4-4e42-8a64-420c390055eb) to gate writes. That built-in role is sufficient for OTel trace/log/metric ingestion because it includes Microsoft.Insights/Telemetry/Write. Application Insights Data Ingestor is a common misconception, not a real built-in role. Using the wrong role with DisableLocalAuth: true causes HTTP 400 "Bad Request" from the azure-monitor-opentelemetry-exporter.

Step 1.3 — main.bicep wiring

// infra/main.bicep — observability is ALWAYS-ON
module law 'modules/log-analytics.bicep' = {
  name: 'law-${envName}'
  params: { name: 'log-${envName}' }
}

module appInsights 'modules/app-insights.bicep' = {
  name: 'appin-${envName}'
  params: {
    name: 'appin-${envName}'
    workspaceId: law.outputs.workspaceId
    uamiPrincipalId: uami.outputs.principalId
  }
}

// ACA env binds to the same LAW so console+system logs land alongside traces
module acaEnv 'modules/aca-env-monitoring.bicep' = {
  name: 'env-${envName}'
  params: {
    name: 'env-${envName}'
    workspaceCustomerId: law.outputs.customerId
    workspaceSharedKey: listKeys(law.outputs.workspaceId, '2023-09-01').primarySharedKey  // ACA env still requires shared-key today
    appInsightsConnectionString: appInsights.outputs.connectionString
  }
}

// Every ACA app + ACA job container reads APPLICATIONINSIGHTS_CONNECTION_STRING
// from env. Pass it through `containers[].env`:
//   - { name: 'APPLICATIONINSIGHTS_CONNECTION_STRING', value: appInsights.outputs.connectionString }
//
// MCP server, bot, workspace, deadline-watcher cron — all four get the same env var.
// The Foundry hosted agent does NOT — Foundry auto-injects it (Layer 2).

The shared-key gotcha. Azure Container Apps environment binding to LAW still uses customerId + sharedKey (not RBAC) as of late 2025. This is the one remaining keyed surface in an otherwise-keyless stack. Document the exception in your README; don't fight it.

Layer 2 — Foundry hosted agent (the runtime)

⚡ START HERE if az rest --method PUT returns 400/AAD or GET returns credentials: null: Jump directly to the O-012 workaround below. The auto-injection path does NOT work for all account types — if PUT fails, skip to O-012 immediately.

🚨 START HERE if your AppInsights connection PUT fails or your agent reports server_error after a fresh azd provision.

The "normal" Layer 2 path below assumes the platform auto-injects APPLICATIONINSIGHTS_CONNECTION_STRING after you create an account-level connection. On O-012-affected accounts (recurring in 2026-05-28), the auto-injection silently drops — the connection looks fine but the env var never lands in the container, so traces never reach AppInsights.

Decision tree:

1. Try the Step 2.1 PUT below (authType: ApiKey + credentials.key).
2. If PUT returns HTTP 400 ValidationError "AuthType for AppInsights Connection can only be ApiKey" → you sent authType: AAD; switch to ApiKey and retry. See O-012 row in § Common silent-failure modes.
3. If PUT returns HTTP 200 but GET returns credentials: null → you hit the silent-drop variant of O-012; use the underscored env-var passthrough workaround (APPLICATION_INSIGHTS_CONNECTION_STRING, with underscore — NOT the reserved no-underscore name) from HostedAgentDefinition.environment_variables via create_version(), and call configure_azure_monitor(connection_string=...) explicitly in container.py. See O-012 row for the full forensic.
4. If you see no traces in AppInsights despite a healthy connection record → check agent.yaml did NOT set APPLICATIONINSIGHTS_CONNECTION_STRING (it's a reserved name; setting it blocks auto-injection). Remove and redeploy.

The hosted agent runtime emits traces automatically — but ONLY if the account has an AppInsights connection registered. Without that connection, the runtime silently drops every span.

Architecture (MAF 1.6.0+). The platform (azure.ai.agentserver) manages its OWN OTel pipeline via _tracing.py:_setup_log_export(). This pipeline captures platform log records (HTTP requests, agent lifecycle, message routing) but does NOT export dependency spans. With 1.6.0, the hosting package bundles microsoft-opentelemetry which adds the SpanExporter + all instrumentors (openai-v2, httpx, etc.) — gen_ai dependency spans flow automatically.

Do NOT call standalone configure_azure_monitor() in container.py — it conflicts with the platform's TracerProvider setup, causing duplicate log records and/or lost spans. The only telemetry code you need is the env var passthrough (for O-012 workaround) and optionally client.configure_azure_monitor(). See foundry-hosted-agents § MAF 1.6.0 update.

Step 2.1 — Create the account-level connection (postprovision)

🛣️ Path A (recommended — try this first): PUT with authType: ApiKey

• credentials.key from your AppInsights instrumentation key. Succeeds on most accounts; data lands within 1-2 min after first invocation.

🛣️ Path B (O-012 fallback — only if Path A returns 400/AAD or GET returns credentials: null): skip the PUT entirely, pass APPLICATION_INSIGHTS_CONNECTION_STRING (with underscore between APPLICATION and INSIGHTS) directly in the agent's HostedAgentDefinition.environment_variables, and call configure_azure_monitor(connection_string=...) explicitly in container.py. The reserved no-underscore name is platform-managed; the underscored variant is accepted as a user override. Verified: 88 traces + gen_ai dependency spans landed in AppInsights from a hosted agent on northcentralus. See O-012 row for full forensic.

# infra/scripts/connect_foundry_appinsights.py — see references/postprovision/
uv run infra/scripts/connect_foundry_appinsights.py

The script:

1. Reads ${AZURE_FOUNDRY_ACCOUNT_NAME} and ${AZURE_APPINSIGHTS_RESOURCE_ID} from azd env
2. PUTs to https://management.azure.com/subscriptions/.../accounts/{name}/connections/AppInsights?api-version=2025-04-01-preview
3. Body: { "properties": { "category": "AppInsights", "target": "<armResourceId>", "metadata": { "ApiType": "Azure" } } }
4. The connection is on the account, not the project — this is the trap that bites everyone

After this runs, the platform begins injecting APPLICATIONINSIGHTS_CONNECTION_STRING into every hosted-agent container revision automatically.

Step 2.2 — RBAC for the agent identities

The hosted-agent platform creates two managed identities per agent (AgentService-<agent-name> + Foundry-<workspace>). Both need Monitoring Metrics Publisher (role GUID 3913510d-42f4-4e42-8a64-420c390055eb) on the AppInsights resource, OR they fail to ingest telemetry with HTTP 400 "Bad Request". The Bicep in Step 1.2 grants it to your workload UAMI; the postprovision script extends the same grant to the platform-managed identities.

Step 2.3 — agent.yaml — what NOT to set

This historical heading covers the same reserved-name rule in the selected versioned hosted manifest. Do not regenerate a legacy agent.yaml for a unified-profile consumer. Keep management environment values distinct from user-declared container variables; management PATCH success is not evidence that the serving process loaded the new telemetry configuration. Use a safe actual-runtime discriminator, not an automatic restart or a diagnostic invoke that replays business work.

# agent.yaml
environment_variables:
  - name: COSMOS_ENDPOINT
    value: ${AZURE_COSMOS_ENDPOINT}
  - name: SEARCH_ENDPOINT
    value: ${AZURE_SEARCH_ENDPOINT}
  # ❌ DO NOT add APPLICATIONINSIGHTS_CONNECTION_STRING here
  # The platform injects it from the account-level connection.
  # Setting it manually causes telemetry collisions.

Add it to agent.yaml and the agent runtime errors with APPLICATIONINSIGHTS_CONNECTION_STRING is reserved.

🔑 Env-var naming distinction (O-012 workaround):

• Reserved (platform-managed, NO underscore): APPLICATIONINSIGHTS_CONNECTION_STRING — set by platform from account-level connection; CANNOT be set in agent.yaml or via normal env vars.
• Override name (user-settable, WITH underscore): APPLICATION_INSIGHTS_CONNECTION_STRING — passed ONLY via HostedAgentDefinition.environment_variables in create_version() when using O-012 workaround; platform reserves the no-underscore name but accepts this underscored variant. Make this distinction explicit when configuring O-012 fallback — use the underscored name only in HostedAgentDefinition, never in agent.yaml.

⚠️ Layer 2 caveat — hosted-agent containers MUST guard the init too. The "platform auto-injects APPLICATIONINSIGHTS_CONNECTION_STRING" promise is best-effort, not contractual — we have field evidence that it can silently fail. In some regions, the AppInsights account-level connection can persist as credentials: null (silent-drop on AAD-rejected → ApiKey-fallback PUT — see "Auth-type platform forensic" below). The platform did NOT inject the env var. The hosted-agent container called raw configure_azure_monitor() as the first line of main(). The SDK raised ValueError. The container crashed before ResponsesHostServer bound. Foundry returned server_error/model:"" on every smoke — with ZERO telemetry to debug it (telemetry init was what crashed). The agent itself was fine.

🚨 CRITICAL. If PUT returns 400/AAD or GET returns credentials:null, you HAVE hit O-012 — use the underscored env var passthrough (APPLICATION_INSIGHTS_CONNECTION_STRING, with underscore between APPLICATION and INSIGHTS) + explicit configure_azure_monitor(connection_string=...). DO NOT silently fall through to assuming auto-injection works.

Discipline. Hosted-agent container.py MUST use the same guarded-init pattern as ACA workloads (Layer 3 helper below), NOT raw configure_azure_monitor(). Treat the platform's auto-injection guarantee as best-effort — guard it like Layer 3 does. See gap rows O-011 / O-012 for the full forensic and the canonical inline helper shape.

Layer 3 — ACA workloads + hosted-agent containers (MCP / bot / workspace / cron / agent)

The Foundry runtime auto-instruments hosted agents, but everything else (your MCP server, your bot, your workspace, your cron jobs) is plain Python or Node — you wire OTel yourself. And per the Layer 2 caveat above, hosted-agent container.py belongs in this bucket too when the platform's auto-injection of APPLICATIONINSIGHTS_CONNECTION_STRING fails — they MUST use the same init_telemetry() helper as every other workload, just defensively guarded so a missing env var doesn't crash startup.

Step 3.1 — Python init (one line of init code per workload)

🔗 Version compatibility note (azure-monitor-opentelemetry >= 1.6.0): configure_azure_monitor() from azure-monitor-opentelemetry>=1.6.0 is NOT compatible with the hosted-agent platform's TracerProvider setup unless guarded. The hosted-agent path differs from ACA-workload path starting MAF 1.6.0. For hosted agents on FoundryChatClient + MAF >= 1.6.0, reference foundry-hosted-agents § MAF 1.6.0 update for the guarded init pattern. For ACA workloads, the pattern below applies directly.

# src/mcp/server.py (or src/bot/app.py, or src/jobs/deadline-watcher/main.py)
from azure.monitor.opentelemetry import configure_azure_monitor

# Reads APPLICATIONINSIGHTS_CONNECTION_STRING from env automatically.
# Call ONCE at module import / before any FastMCP / FastAPI / app code.
# Wraps OTel SDK + Azure Monitor exporter + standard library instrumentations
# (logging, urllib3, requests, fastapi, httpx, redis, postgres, ...) in one shot.
configure_azure_monitor(
    logger_name="<process>-mcp",          # show up as cloudRoleName in AppIn
    instrumentation_options={
        "azure_sdk": { "enabled": True },   # capture Cosmos / AOAI / Search SDK calls
        "fastapi":   { "enabled": True },   # MCP server is FastAPI under the hood
        "django":    { "enabled": False },
        "flask":     { "enabled": False },
        "psycopg2":  { "enabled": False },
        "requests":  { "enabled": True },
        "urllib3":   { "enabled": True },
        "urllib":    { "enabled": True },
    },
)

See references/python/otel_init.py for the full version (handles missing env var gracefully — important for local-test pattern).

Step 3.2 — requirements.txt addition

azure-monitor-opentelemetry>=1.6.0

That's the only dep. It transitively pulls in opentelemetry-api, opentelemetry-sdk, the AzureMonitor exporter, and the auto-instrumentors. Don't pin individual opentelemetry-* packages — they drift fast and the Azure-Monitor wrapper version-locks the compatible set for you.

Step 3.3 — ACA env wiring

// infra/modules/<svc>-aca.bicep — for MCP, bot, workspace
containers: [
  {
    name: 'mcp'
    image: image
    env: [
      // ... cosmos, search, etc ...
      {
        name: 'APPLICATIONINSIGHTS_CONNECTION_STRING'
        value: appInsightsConnectionString  // passed from main.bicep
      }
    ]
  }
]

For ACA Jobs (cron deadline-watcher):

// infra/modules/aca-scheduled-job.bicep
configuration: {
  triggerType: 'Schedule'
  scheduleTriggerConfig: {
    cronExpression: '*/15 * * * *'
  }
}
template: {
  containers: [
    {
      name: 'deadline-watcher'
      image: image
      env: [
        // ... cosmos endpoint, etc ...
        {
          name: 'APPLICATIONINSIGHTS_CONNECTION_STRING'
          value: appInsightsConnectionString
        }
      ]
    }
  ]
}

The scheduled/event-trigger module family is owned by threadlight-event-triggers; azd-patterns/references/bicep/aca-job.bicep stays the generic Manual module.

The silent-cron lesson (from recent pilot retrospectives — see azd-patterns § ACA Job silent-failure playbook). When the cron exits with non-zero before any print() reaches stdout, ACA's default LAW logging routes nothing — but if configure_azure_monitor() ran successfully before the crash, the exception trace lands in AppIn exceptions table even though ContainerAppConsoleLogs_CL is empty. Initialize OTel as the very first thing in __main__ — before any I/O, any imports of Cosmos SDK, any environment lookups.

Diagnostic queries (KQL)

Drop these in your repo's docs/queries/ and reference from the runbook. See references/queries/ for the full set.

"Did ANY trace land in the last 5 minutes?" — the smoke probe

union traces, requests, dependencies, exceptions
| where timestamp > ago(5m)
| summarize count() by table=$table, cloud_RoleName
| order by count_ desc

If count_ == 0 for everything, the AppInsights connection is broken at one of the three layers above. Most common causes:

Hosted-agent traces, last hour

// Platform log traces (HTTP, messages, agent lifecycle)
traces
| where timestamp > ago(1h)
| where cloud_RoleName startswith "AgentService-" or cloud_RoleName startswith "agent-"
| project timestamp, severityLevel, message, cloud_RoleName
| order by timestamp desc

Hosted-agent gen_ai spans (model, tokens, latency)

Note (MAF 1.6.0+): gen_ai dependency spans use cloud_RoleName == "agent_framework" — NOT the agent name prefix. Queries filtering startswith "agent-" will miss these spans.

dependencies
| where timestamp > ago(1h)
| where cloud_RoleName == "agent_framework"
| project timestamp, name,
    model=tostring(customDimensions['gen_ai.response.model']),
    op=tostring(customDimensions['gen_ai.operation.name']),
    input_tokens=tostring(customDimensions['gen_ai.usage.input_tokens']),
    output_tokens=tostring(customDimensions['gen_ai.usage.output_tokens']),
    duration
| order by timestamp desc

MCP tool-call breakdown

dependencies
| where timestamp > ago(1h)
| where cloud_RoleName has "mcp"
| where name has "tools/call"
| extend tool = tostring(customDimensions.['mcp.tool.name'])
| summarize count(), avg_ms=avg(duration), p95_ms=percentile(duration, 95) by tool
| order by count_ desc

Silent-cron debug — exec failures with no console output

let job_name = "ca-job-deadline-watcher";
let window = 1h;
union
  (exceptions
    | where timestamp > ago(window)
    | where cloud_RoleName has job_name
    | project timestamp, table="exceptions", problem=type, outerMessage),
  (ContainerAppConsoleLogs_CL
    | where TimeGenerated > ago(window)
    | where ContainerAppName_s has job_name
    | project timestamp=TimeGenerated, table="console", problem=Log_s, outerMessage="")
| order by timestamp desc

If exceptions returns rows but console is empty, the cron initialized OTel before crashing — you have a stack trace in AppIn even though logs are silent. This is the single most useful query when an ACA Job is "failing without telemetry."

Resource Health — availability vs config/provisioning state

When a workload is degraded but App Insights is silent, check Resource Health — the platform's own availability signal. Two CLI shapes look interchangeable but answer different questions; mixing them up is a common diagnostic dead-end:

# Current availability status (Available / Unavailable / Degraded) — what you want
az rest --method get --url \
  "https://management.azure.com/<RESOURCE_ID>/providers/Microsoft.ResourceHealth/availabilityStatuses/current?api-version=2025-05-01"

# Config + provisioning state ONLY — NOT availability
az resource show --ids <RESOURCE_ID>

az resource show returns ARM config and provisioningState: Succeeded even when the resource is unavailable — it never reports a platform outage. For "is it actually up right now?" use the Microsoft.ResourceHealth/availabilityStatuses/current data plane (GA 2025-05-01). Pair with az monitor activity-log list -g <rg> --max-events 20 for health-impacting control-plane events.

Reusable KQL probe helpers

When you need a programmatic read of OTel telemetry health for a Foundry workload (trace freshness, exception rate, RAI denials, AGT denials, rate limit events), call the canonical helpers:

from foundry_observability.kql_probes import (
    trace_freshness, exception_rate, rai_denials,
    agt_denials, rate_limit_events,
)

result = trace_freshness(workspace_id="<law-workspace-id>",
                          app_name="my-agent", since="1h")
# result["result"]         → int minutes (None on error)
# result["confidence"]     → 0.0..1.0
# result["last_probe_at"]  → ISO8601 UTC
# result["error"]          → None on success, str on failure

For async callers (FastAPI, Quart, etc.) use the mirrored aio module:

from foundry_observability.kql_probes_aio import trace_freshness
result = await trace_freshness(workspace_id=..., app_name=...)

MUST: Copy verbatim from references/python/kql_probes.py and references/python/kql_probes_aio.py. Do NOT redefine inline — the validator enforces single-source-of-truth.

Every helper NEVER raises. On any exception (auth failure, KQL syntax error, transient outage) the helper returns {result: None, confidence: 0.0, error: "<reason>"}.

Threadlight integration

Common silent-failure modes (the gap catalog)

Auth-type platform forensic (some regions)

The Layer 2 caveat callout above and gap rows O-011 / O-012 are anchored on this forensic. Three remediation paths attempted, all failed with documented evidence:

Re-confirmation guidance. If the same three failure paths reproduce on a separate Foundry account with a clean install + upgraded agent dependencies (agent-framework-foundry-hosting >= 1.0.0a260421), treat the behavior as platform-region, not project- or dependency- specific. For affected regions, ship Layer 3 telemetry and skip the AppIn account connection until the platform fix lands.

Lesson. The Layer 2 promise ("platform auto-injects when you have an account-level AppInsights connection") cannot be relied upon as a runtime contract. Layer 2 is best-effort observability; Layer 3 (init_telemetry() everywhere — including in the hosted-agent container) is the contract that holds. Build for Layer 3; treat Layer 2 as a bonus.

O-012 workaround found (May 2026). Pass APPLICATION_INSIGHTS_CONNECTION_STRING (underscored variant) directly in HostedAgentDefinition.environment_variables via create_version(). The platform reserves APPLICATIONINSIGHTS_CONNECTION_STRING (no underscore) but accepts the underscored name. In the container entrypoint:

def _valid_cs(cs: str | None) -> str:
    cs = (cs or "").strip()
    return cs if cs.startswith("InstrumentationKey=") else ""

_CS = _valid_cs(os.getenv("APPLICATION_INSIGHTS_CONNECTION_STRING")) \
    or _valid_cs(os.getenv("APPLICATIONINSIGHTS_CONNECTION_STRING"))
if _CS:
    os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"] = _CS
    configure_azure_monitor(connection_string=_CS, logger_name="hosted-agent-maf")

This bypasses both the broken platform auto-injection AND the broken SDK FoundryChatClient.configure_azure_monitor() path (which also hits O-012 via _get_with_credentials → empty credentials). Verified: 88+ traces from a hosted agent on northcentralus.

Also requires: no_cache=True on DockerBuildRequest + a used ARG BUILD_TS (RUN echo $BUILD_TS > /dev/null) in the per-job Dockerfile, otherwise ACR layer caching produces identical digests and Foundry deduplicates create_version — the new code never reaches the container. See foundry-hosted-agents § Image-tag staleness trap.

Validation checklist

Before declaring a pilot "deploy-complete":

• infra/modules/log-analytics.bicep exists and is referenced from main.bicep
• infra/modules/app-insights.bicep exists and is referenced from main.bicep
• infra/scripts/connect_foundry_appinsights.py exists and is called from azure.yaml postprovision hook
• Every ACA service in azure.yaml (MCP, bot, workspace, cron) passes APPLICATIONINSIGHTS_CONNECTION_STRING env from bicep outputs
• Every Python entry-point under src/ calls configure_azure_monitor() before any other init
• agent.yaml does NOT include APPLICATIONINSIGHTS_CONNECTION_STRING in environment_variables
• Monitoring Metrics Publisher granted to workload UAMI in app-insights.bicep
• Postprovision hook also grants Monitoring Metrics Publisher to AgentService-* and Foundry-* platform identities
• After azd up smoke + first agent invocation, the smoke probe first-trace-probe.kql returns ≥ 1 row from each of: hosted-agent (AgentService-* cloudRoleName), MCP (mcp cloudRoleName), bot (bot cloudRoleName)
• threadlight-safe-check Step 5.6 (App Insights existence) PASSES

If any box is unchecked, the pilot is not deploy-complete. The visible "agent works in the portal" smoke does not catch this gap — the agent works exactly the same with or without telemetry. Only the KQL probe catches it.

Verification before claiming telemetry is wired

DO NOT report "observability is configured" until you have run the KQL probe and seen actual traces. Bicep deploying successfully does NOT mean traces are flowing.

After wiring all 3 layers, execute this gate:

1. Invoke the agent once via the Foundry playground or a curl to the ACA endpoint — this generates the first trace.
2. Wait 2 minutes for ingestion (App Insights has ~90-second ingestion delay).

3. Run the first-trace-probe KQL and verify ≥ 1 row per workload role:

   union traces, requests, dependencies
   | where timestamp > ago(10m)
   | summarize count() by cloud_RoleName
   | order by count_ desc

4. If zero rows: check (a) connection string propagation, (b) RBAC assignment, (c) configure_azure_monitor() import order — in that sequence. Do NOT report "wiring looks correct" without trace evidence.

References

• App Insights workspace-based components: https://learn.microsoft.com/azure/azure-monitor/app/create-workspace-resource
• azure-monitor-opentelemetry Python SDK: https://learn.microsoft.com/python/api/overview/azure/monitor-opentelemetry-readme
• Foundry account-level connections (AppInsights category): https://learn.microsoft.com/azure/ai-foundry/concepts/connections
• Monitoring Metrics Publisher role: https://learn.microsoft.com/azure/role-based-access-control/built-in-roles/monitor#monitoring-metrics-publisher
• ACA env LAW + AppIn binding: https://learn.microsoft.com/azure/container-apps/observability
• KQL reference: https://learn.microsoft.com/azure/data-explorer/kusto/query/

Related skills: threadlight-deploy, threadlight-safe-check, foundry-hosted-agents, foundry-mcp-aca, foundry-teams-bot, threadlight-event-triggers, azd-patterns, foundry-evals.