speckit.sre-observability

🎯 Mission

Design, configure, and enforce enterprise-grade Observability and Site Reliability Engineering (SRE) standards across services. Ensure system state is fully transparent through the Three Pillars (Logs, Metrics, Traces), actionable alerts, and battle-tested resilience patterns.

📥 Inputs and Outputs

• Input: Feature spec, plan, tasks, service topology, SLA/SLO commitments, and container/infrastructure manifests.
• Output: OpenTelemetry collector configs and SDK setup scripts, Prometheus scraping configs and Grafana dashboards, /livez and /readyz endpoints with graceful shutdown handlers, and incident runbooks.

📋 Protocol

1. Distributed Tracing & OpenTelemetry (OTel)

• Trace Context Propagation: Propagate W3C TraceContext headers (traceparent, tracestate) across all HTTP, gRPC, and message queue boundaries.
• Span Granularity: Instrument high-level operations: incoming HTTP request, external API calls, DB transactions, background jobs, and cache accesses.
• Baggage & Correlation: Attach trace_id, span_id, service.name, and environment to all telemetry data.

2. Metrics Architecture (RED & USE Methods)

• RED Method (Request-driven Services):
  • Rate: Requests per second (Counter).
  • Errors: Failed requests count partitioned by HTTP status code / error type (Counter).
  • Duration: Request latency distribution (Histogram with standardized buckets: 0.005s, 0.01s, 0.025s, 0.05s, 0.1s, 0.25s, 0.5s, 1s, 2.5s, 5s, 10s).
• USE Method (Resources - CPU, RAM, Disk, DB Pools):
  • Utilization: Percent time busy.
  • Saturation: Queue length or waiting threads.
  • Errors: Device or connection error count.

3. Structured Logging Standard

• Format: 100% Structured JSON logs emitted to stdout / stderr.
• Mandatory Fields: timestamp (ISO 8601 UTC), level (DEBUG, INFO, WARN, ERROR), message, trace_id, span_id, service, duration_ms.
• Zero Sensitive Data: Automatically scrub passwords, tokens, API keys, card numbers, and PII from log payloads.

4. Health Checks & Lifecycle Management

• Liveness Probe (/livez): Returns 200 OK if the process is running. NEVER check downstream dependencies here (prevents cascading restart loops).
• Readiness Probe (/readyz): Returns 200 OK ONLY if database, Redis, and critical downstream dependencies are connected and ready to accept traffic.
• Graceful Shutdown: Intercept SIGTERM/SIGINT, stop accepting new requests, drain existing in-flight connections (15–30s timeout), and close DB/queue connections safely.

5. Resilience & Fault Tolerance

• Circuit Breaker: Wrap external third-party calls with circuit breakers (e.g. Resilience4j, Tenacity) to fast-fail when error rate exceeds 50%.
• Rate Limiting: Enforce Token Bucket / Leaky Bucket rate limits at ingress and per-tenant boundaries.
• SLO Error Budgets: Define Service Level Objectives (SLOs) and establish automated alerting when Error Budget consumption rate spikes.

🏁 Completion Gate

OpenTelemetry distributed tracing propagates trace context across all service boundaries, RED and USE metrics are fully scraped with standardized buckets, 100% structured JSON logs scrub sensitive data, and liveness/readiness probes with graceful shutdown handlers pass resilience tests.

🚫 Guard Rails

• FORBIDDEN: Logging secrets, credentials, auth tokens, or PII in plaintext.
• FORBIDDEN: Checking DB connections in /livez liveness probes.
• FORBIDDEN: Running services without graceful shutdown signal trapping.
• FORBIDDEN: Creating metrics with unbounded label cardinality (e.g. user IDs or UUIDs as metric labels).