Observability and instrumentation

Instrumentation ships with the feature, like tests.

Define "working" first

List 2–4 questions on-call will ask — retry-adjusted success rate, why a case failed permanently, is the provider slow now. Every signal answers one; one that answers none is cost.

Scope per component

Instrument each component's boundary, not every function.

• Request-serving (HTTP, gRPC, queue): RED per route template — rate, errors by status class, duration histogram — plus saturation of the pool or queue it competes for, from middleware in one place. A specific question can justify more; a payment path may warrant a span per provider call.
• Job, worker: outcome, duration, items, retries, dead-letter count per run. Queue lag beats per-item logs; aggregate per job, no label per item.
• Library: nothing unbidden — take a logger and metrics handle from the host, or expose hooks; no private exporter, no stdout.
• CLI: exit code plus stderr diagnostic; structured output only for a supervisor.

Logs explain one case, metrics aggregate, traces localize across services — metrics say that, traces where, logs why. Cost is per event, per label set, per sampled request.

Logging

One structured object per business event, with a stable event name and stable, typed field names a query written today still matches next month; ids and codes rather than prose (event: payment_failed, provider, errorCode, attempt).

• Identifiers, not payloads: no bodies, tokens or full personal data even at debug; log an id or hash, pseudonymous where it names a person.
• One event per state change, not per loop iteration; a request or trace id where a path crosses components (service → queue → worker), since inside one process a stack trace suffices.
• Route library logging through your formatter, quiet the noisy ones in config, and redact known secret patterns in the collector: one forgotten call must not leak a token.

Metrics

Labels come from small fixed sets — never a user id, raw URL, error message or request id; a counter plus a log line answers "which one". Histograms with p50/p95/p99, since an average hides the tail; unit in the name (http_request_duration_seconds); counters up only; sample the hot path.

Tracing

OpenTelemetry, auto-instrumentation for HTTP, gRPC and database clients first. Span names describe the operation (GET /orders/:id), attributes only for keys you search by. Propagate context across process and queue boundaries — a trace stopping at the queue means the producer never injected it. Sample at the edge and record the choice.

Alerting

Alert on symptoms users feel; causes go on a dashboard (CPU at 85%, one restart). Each alert is actionable or becomes a ticket, links its runbook, takes its threshold from an SLO or measured baseline rather than a copied default, is page or ticket only, guards scope (a shared dashboard's baseline is not this service's), and resolves itself without re-paging.

Verify the telemetry itself

Wrong instrumentation argues with the incident:

• Force an error in staging; find it by request id.
• Send test traffic; confirm your labels appear — a label typo silently creates a new series.
• Follow one request across components in the trace UI.
• Fire each alert once by lowering its threshold, then restore it.
• Emit a synthetic canary secret through the logging path; confirm the collector redacts or rejects it.