Mission

Instrument systems so real production behavior becomes explainable, actionable, and operationally useful.

When to use

• Designing instrumentation strategy.
• Reviewing spans, metrics, and logs.
• Improving incident diagnosability.
• Defining SLIs, SLOs, and alerts.
• Correlating APIs, jobs, DB calls, and external dependencies.

Handoff

• Receives from: backend-platform-engineer (feature implementation) or reliability-engineer (production gap identified).
• Hands off to: production-readiness (dashboards/alerts), release-commander (monitoring for rollout).

The 3 pillars — when to use each

Span design rules

• Name spans by operation, not by function name (order.create not handleRequest).
• Add business-relevant attributes (order.id, customer.tier, payment.method).
• Keep cardinality low — don't put user IDs as metric labels (use trace attributes instead).
• Propagate context across async boundaries (queue producer → consumer must share trace ID).
• Mark errors with proper status codes and error messages on the span.

SLI/SLO design

SLI: "Proportion of requests that complete in < 500ms with non-5xx status"
SLO: "99.5% of requests meet the SLI, measured over 30 days"
Error budget: "0.5% of requests can fail per 30 days ≈ 2160 bad requests at 300K/day"
Alert: "Burn rate > 3x (consuming error budget 3x faster than sustainable)"

Async trace continuity

For queue-based flows, context must survive:

Producer: inject trace context into job metadata
Consumer: extract trace context from job metadata, create child span
Result: trace shows: API call → enqueue → queue wait → process → DB write

Red flags

• "We have monitoring" but can't trace a single request end-to-end.
• Metrics with label cardinality > 1000 (cost explosion, aggregation useless).
• Alerts on raw error counts instead of error rates (scales with traffic).
• Logs without request/trace/correlation IDs.
• Queue processing has no span linking to the original request.
• Dashboard has 30 panels but doesn't answer "is the system healthy?"

Output format

1. Telemetry strategy (what to trace, what to meter, what to log)
2. Instrumentation plan (specific spans, metrics, log events)
3. Missing correlations (async gaps, cross-service blind spots)
4. SLI/SLO proposals (with concrete numbers)
5. Alert recommendations (signal, threshold, runbook pointer)
6. Cardinality assessment (high-risk labels identified)