Modal Observability

Domain skill for runtime logs, debugging, metrics, and integrations.

For the full logs/debug module, read logging-debugging.md — it is self-contained (decision tree, all log CLI flags, shell/exec, interact, env vars, symptom table).

When to Use This Skill

• Fetching or streaming App / container / Sandbox logs
• Missing logs, failures, OOMs, timeouts, hung containers
• modal shell, modal container exec, modal run -i / modal.interact()
• modal.enable_output(), MODAL_LOGLEVEL, MODAL_TRACEBACK
• Opening dashboard URLs for objects from code / CLI
• Datadog, OpenTelemetry, Slack alerts
• GPU metrics / health
• Enterprise audit logs (workspace actions, not stdout)

Logs and debugging

→ logging-debugging.md (complete module)

Agent defaults (summary)

1. Finite fetch by default — do not pass -f unless streaming is required.
2. CLI default is last 100 lines then exit (changed in SDK 1.4+).
3. Prefer modal container logs <ta-*|sb-*> --all once you have a container/sandbox ID.
4. --source stderr|stdout|system; add --timestamps and --show-*-id when correlating.

Quick commands

modal app logs my-app --since 2h --source stderr --timestamps \
  --show-function-id --show-function-call-id --show-container-id

modal container list --app-id ap-… --json
modal container logs ta-… --all --timestamps
modal container logs sb-… --all

modal shell ta-…                 # running container
modal shell script.py::fn        # fresh image/spec (not live task)
modal container exec ta-… -- ps aux

MODAL_LOGLEVEL=DEBUG modal run hello.py

with modal.enable_output():
    with app.run():
        fn.remote()

Dashboard URLs

Most Modal objects expose .get_dashboard_url() for deep links (logs, metrics, browsers):

fc = f.spawn(arg)
print(fc.get_dashboard_url())

CLI helpers:

modal dashboard                  # Apps homepage for current environment
modal dashboard <object-id>
modal app dashboard <app-name>
modal volume dashboard <volume-name>

GPU Metrics

GPU utilization % ≠ compute efficiency; prefer power utilization when judging useful GPU work.

Details: gpu-monitoring.md.

GPU Health

Modal detects ECC / hardware faults and retries on healthy GPUs automatically.

Integrations

Setup detail: integrations.md.

• Datadog — dashboard Settings → Integrations → Datadog (metrics, lifecycle, Function logs)
• OpenTelemetry — OTLP endpoint; spans for invocations, container lifecycle, image builds
• Slack (Beta) — scheduled Function failure / timeout / OOM alerts

Audit Logs (Enterprise)

Workspace actions only (app.deploy, secret.create, …) — not Function stdout or Sandbox exec. Filters: key:value, negate with -.

→ audit-logs.md

Go / TypeScript client telemetry

gRPC unary interceptors (Go) or middleware (TS) on ModalClient for SDK RPC tracing. Runtime App logs still come from Python Functions + the log CLI above.

Crash / performance triage

Overlay deployments on dashboard metric charts to correlate regressions with deploys.

References (this skill)

• logging-debugging.md — logs + debug CLI (complete)
• integrations.md — Datadog, OTel, Slack
• audit-logs.md — audit actions / filters
• gpu-monitoring.md — GPU metrics detail

Guardrails

• Log CLI does not follow by default — last 100 then exit; pass --follow for live streams
• --all is container/sandbox only, not modal app logs
• Audit logs ≠ application logs (workspace actions only)
• Historical CLI logs respect plan retention
• GPU util % is occupancy, not efficiency
• Slack notifications require Beta access
• OpenTelemetry exports add minor overhead; keep sampling rate reasonable