Observability Planner

Purpose

Design observability for AI-powered systems: what to trace, measure, log, alert on, and display — aligned to SLOs and incident response.

Input: System architecture, SLO targets, critical user journeys, stack constraints (cloud, APM vendor)
Output: Observability Plan with signal catalog, dashboard specs, alert rules, and runbook hooks
Examples: See examples.md for worked input/output.

Workflow

Step 1: Define SLIs and SLOs

Document error budget and escalation when burned.

Step 2: Apply three pillars per component

Use OpenTelemetry semantic conventions where available.

Step 3: AI-specific observability

Correlate with analytics-instrumentation-planner product events.

Step 4: Design dashboards

Each panel links to trace exemplar for drill-down.

Step 5: Define alerts and runbooks

Alerts must be actionable; tie to owner and escalation path.

Step 6: Logging and privacy

• Structured JSON logs with trace_id, user_id (hashed), request_id
• Never log raw prompts/responses in prod; use opt-in debug tier
• Retention: hot 7d, cold 30d, compliance per policy

Step 7: Validate

Run Validation checklist.

Decision Rules

Validation

• SLIs/SLOs per critical journey
• Trace spans for LLM, RAG, and tools defined
• Metrics catalog with names and labels
• ≥2 dashboards specified with audience
• Alerts with thresholds and runbook links
• Privacy rules for prompts and PII in logs
• Error budget policy documented
• Correlation ID propagated end-to-end

Anti-patterns

• Logs only — cannot debug distributed LLM latency without traces.
• Alert fatigue — 50 thresholds nobody acts on.
• Dashboard vanity — charts without SLO line.
• Logging prompts in prod — security and compliance risk.
• Missing business metrics — ops green while users churn.

Best Practices

• Adopt OpenTelemetry early; one instrumentation for metrics + traces.
• Store prompt_version and model_id on every LLM span.
• Practice game days: inject failure, verify alerts fire.
• Pair with ai-evaluation-builder for quality regressions.
• Review observability coverage in architecture review checklist.

Output Structure

# Observability Plan: [System Name]

## SLOs
| Journey | SLI | Target | Error budget |
|---------|-----|--------|--------------|

## Signal Catalog
### Traces
[Span list + attributes]

### Metrics
[Name, labels, collection]

### Logs
[Fields, retention]

## Dashboards
[Panel descriptions]

## Alerts
| Name | Query | Threshold | Runbook |

## Privacy
[Logging restrictions]

## Rollout
[Instrumentation phases]

Next Skills