observability-and-slo-architect
Skill4 starsChanged 31 days agoMITView on GitHub
Transforms observability from passive logger.error statements into a Google Monarch & Dapper proactive telemetry architecture—mandating OpenTelemetry by default with unbroken W3C traceparent propagation across every I/O operation, RPC, message queue, and worker; emitting automated RED metrics for endpoints and USE metrics for resources without metric cardinality explosions; and gating production alerts on Google SRE Multi-Window Multi-Burn-Rate error budgets. Use when designing observability architectures, instrumenting APIs, background queues, outbox workers, or database drivers, adding Prometheus/OpenTelemetry/Monarch metrics, replacing high-cardinality metric labels (user_id in labels), defining SLIs/SLOs, or auditing code for broken trace contexts and alert noise.
In this repository47 other files it gives its agents.
- AGENTS.mdAGENTS.md
- anti-slop-engineeringSkill
- anti-slop-engineeringSkill
- api-ergonomics-and-contract-governorSkill
- apple-craft-and-frame-budgetSkill
- apple-craft-and-interaction-physicsSkill
- asset-provenance-guardSkill
- backprop-guardSkill
- chaos-and-degradation-choreographerSkill
- chromatic-drift-guardSkill
- cognitive-load-and-attention-governorSkill
- comment-budget-guardSkill
- context-window-and-token-economics-architectSkill
- cryptographic-agility-and-secret-lifecycle-gateSkill
- decision-guardSkill
- demock-guardSkill
- design-system-drift-guardSkill
- deterministic-simulation-and-property-testingSkill
- distributed-consensus-and-concurrency-gateSkill
- finops-and-unit-economics-governorSkill
- future-scale-gateSkill
- grumpy-senior-staff-engineerSkill
- honeypot-trap-guardSkill
- house-tell-guardSkill
- install-anti-slopSkill
- llm-eval-and-prompt-injection-fortressSkill
- mandela-guardSkill
- mechanical-sympathy-and-perf-gateSkill
- mechanical-sympathy-profilerSkill
- motion-runtime-guardSkill
- observability-and-slo-gateSkill
- paranoid-principal-architectSkill
- pr-boundary-guardSkill
- privacy-and-data-lineage-gateSkill
- re0-handoff-guardSkill
- reduction-guardSkill
- repo-memory-and-invariant-stewardSkill
- root-fix-gateSkill
- safe-evolution-and-rollout-choreographerSkill
- scientific-fault-isolationSkill
- styleseed-guardSkill
- supply-chain-and-sbom-sentinelSkill
- tagore-guardSkill
- type-level-and-ast-slop-gateSkill
- ui-and-copy-disciplineSkill
- verification-oracle-architectSkill
- zero-downtime-migration-gateSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 4 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 17
OpenTelemetry by Default: Every I/O operation (HTTP, gRPC, database query, Redis command, message queue enqueue/dequeue, background worker) …
- Distinctive: line 38
Ban Non-Impact Alerts: No alert fires on raw resource thresholds (e.g.
- Distinctive: line 61
Every network call, database query, cache access, and queue message must execute within an active OpenTelemetry Span.
- Very specific: line 67
Never attach dynamic or unbounded values (user_id, request_id, order_id, email, ip_address, raw URL paths) to metric attributes.
- Very specific: line 83
Exemplar Binding: When recording latency Histograms in sampled spans, attach the active trace_id and span_id as a Metric Exemplar.
- Distinctive: line 85
Pre-Enqueuing Resource Backpressure: Check resource Saturation before enqueuing or accepting new transactions.
- Very specific: line 104
Causality Audit: When this message is published to the queue, how does the background consumer retrieve the parent traceId?
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Observability & SLO Architect (Google Monarch & Dapper School)
Telemetry without distributed context is blindness; high-cardinality identifiers in time-series metrics are a denial-of-service attack on your monitoring infrastructure; alerts not tied to user impact or error budget burn rate are operational noise. Transform observability into a proactive engineering contract: OpenTelemetry by default, automated RED and USE metrics, and Multi-Window Multi-Burn-Rate SLO alerts.
Target Architecture: Google Monarch & Dapper Imprint
- Google Dapper (Distributed Tracing & Context Causality):
- OpenTelemetry by Default: Every I/O operation (HTTP, gRPC, database query, Redis command, message queue enqueue/dequeue, background worker) is bound to an active Span.
- Unbroken Context Propagation: Pass W3C TraceContext (
traceparent: 00-<trace_id>-<span_id>-<flags>,tracestate) andx-correlation-idacross every network boundary, asynchronous queue, and database statement (via SQLcommenter). - Forensic Isolation: High-cardinality identifiers (
user_id,tenant_id,order_id,idempotency_key,request_id) live exclusively within Span Attributes and structured JSON logs.
- Google Monarch (Planetary Low-Cardinality Metric Stream Engine):
- Strict Low-Cardinality Metric Schemas: Time-series metric dimensions use bounded allowlists ($\prod |L_i| \le 1,000$ per metric family). Dynamic identifiers in metric labels are strictly prohibited.
- 404 Route Collapsing: Unmatched URL paths collapse to
"__unmatched__"to neutralize scanner-induced cardinality explosions. - Histograms over Averages: Durations are recorded exclusively in explicit-bucket Histograms to compute P50, P95, and P99 quantiles. Arithmetic averages and Gauge percentiles are mathematically invalid and prohibited.
- Trace Exemplars: Span
trace_idandspan_idattach directly to Histogram bucket recordings, enabling 1-click jumps from latency spikes to causal traces.
- RED & USE Metric Instrumentation:
- RED for Endpoints (Request-Driven Work):
- Rate: Throughput in requests/sec (
http.server.request.count,rpc.server.request.count). - Errors: Failure rate (
http.server.error.countpartitioned by status class or error code). - Duration: Latency distribution recorded via explicit-bucket Histograms (
http.server.duration).
- Rate: Throughput in requests/sec (
- USE for Resources (Capacity & Hardware/Software Limits):
- Utilization: Fraction of time busy or capacity consumed ($0.0 \le U \le 1.0$) for CPU, Memory, DB Connection Pools, Worker Pools.
- Saturation: Depth of queued work waiting for capacity (Queue backlog, connection waiters, event loop lag).
- Errors: Count of resource-level failure events (pool checkout timeouts, worker OOMs, dropped messages).
- RED for Endpoints (Request-Driven Work):
- SLI/SLO-Driven Alerts (Google SRE Multi-Burn-Rate):
- Ban Non-Impact Alerts: No alert fires on raw resource thresholds (e.g. CPU > 80%) unless there is measurable user impact or Error Budget exhaustion.
- Multi-Window Multi-Burn-Rate Matrix:
- 14.4x Fast Burn (2% budget in 1 hour): Long window = 1 hour, Short window = 5 minutes $\rightarrow$ Immediate PagerDuty page & automated canary rollback.
- 6.0x Slow Burn (5% budget in 6 hours): Long window = 6 hours, Short window = 30 minutes $\rightarrow$ Incident remediation ticket for next business day.
Activation and Boundaries
| Situation | Behavior |
| Instrumenting HTTP endpoints, gRPC handlers, or RPC clients | Emit RED metrics (Rate, Errors, Histogram Duration), bind to OpenTelemetry Server Spans, propagate W3C traceparent |
| Designing or instrumenting background queues, event buses, outbox workers, or DB queries | Inject traceparent and correlation_id on producer, extract on consumer, record USE metrics on workers and connection pools |
| Writing logging middleware or error handling layers | Ban raw console.log and logger.error("..."); emit structured JSON logs correlated with trace_id, span_id, and redacting secrets |
| Establishing alerting policies, service readiness checklists, or SLO gates | Enforce Multi-Window Multi-Burn-Rate alerts on availability and P95/P99 latency SLIs; reject static threshold alerts |
| Auditing existing services or PR diffs | Scan for cardinality leaks (user_id in labels), gauge-averaged latencies, broken async trace trees, and noisy threshold alerts |
| Static documentation edits, CSS token changes, or local offline scripts | Do not activate; use anti-slop-engineering |
The Six Absolute Laws
- The OpenTelemetry by Default Law:
Every network call, database query, cache access, and queue message must execute within an active OpenTelemetry Span. Handlers must never catch errors without recording the exception (
span.recordException(err)) and setting error status (span.setStatus({ code: SpanStatusCode.ERROR, message: err.code })). - The Unbroken W3C Context Propagation Law:
Context must never drop across synchronous or asynchronous boundaries. Message payloads (Kafka, SQS, RabbitMQ, BullMQ) and Transactional Outbox rows must contain explicit
traceparentandcorrelation_idheaders. Workers must extract parent context before executing task logic. - The Google Monarch Low-Cardinality Law:
Never attach dynamic or unbounded values (
user_id,request_id,order_id,email,ip_address, raw URL paths) to metric attributes. Label products must remain $\prod |L_i| \le 1,000$. Unmatched 404 paths must be collapsed into"__unmatched__". - The RED & USE Dual-Matrix Law: Endpoints must expose Rate, Errors, and Duration (RED) via Histograms. Resources (CPU, Memory, DB Connection Pools, Thread Pools, Worker Queues) must expose Utilization, Saturation, and Errors (USE).
- The Histogram Quantile Law:
Measure durations and payload sizes exclusively with Histograms configured with explicit, domain-calibrated bucket boundaries (e.g.
[5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000]ms). Never average latencies across horizontal instances. Attach Trace Exemplars to bucket observations. - The Multi-Window Multi-Burn-Rate Alerting Law: Ban alerts on non-user-impacting symptoms. Alerting must be driven by Error Budget Burn Rate evaluated simultaneously across long and short lookback windows (14.4x 1h/5m for paging; 6.0x 6h/30m for ticketing).
Invariant Rules
- Mandatory Envelope Schema: Async message schemas must define mandatory
traceparent: stringandcorrelationId: stringmetadata. - Exemplar Binding: When recording latency Histograms in sampled spans, attach the active
trace_idandspan_idas a Metric Exemplar. - Deterministic Log Redaction: Structured logs must automatically sanitize fields matching
authorization,cookie,password,secret,token, andapi_key. - Pre-Enqueuing Resource Backpressure: Check resource Saturation before enqueuing or accepting new transactions.
- In-Memory Verification Tests: Automated test suites must verify telemetry using in-memory exporters, asserting zero cardinality leaks and unbroken trace trees across queue boundaries.
Execution Router
| Task Shape | Primary Workflow | Reference Contract |
| Instrumenting HTTP/gRPC endpoints with RED metrics & OTel spans | Emit RED & USE Metrics | RED & USE Metrics Matrix |
| Propagating W3C context across queues, outboxes, workers & DB | Enforce OpenTelemetry I/O Spans | W3C traceparent & Async Context |
| Formulating SLIs, SLOs & Multi-Burn-Rate alerting policies | Define SLO Burn-Rate Alerts | SLI/SLO & Multi-Burn Alerting |
| Auditing records and source code with automated guard | Record validation and source scan | scripts/observability_guard.py |
Pre-Instrumentation Interrogation Matrix
- Cardinality Audit: What is the maximum cardinality product of all labels on this metric family? Could any tenant or user input explode the TSDB?
- Causality Audit: When this message is published to the queue, how does the background consumer retrieve the parent
traceId? - Resource Saturation Audit: What happens when the connection pool or queue saturates? Does USE metric instrumentation surface saturation before requests fail?
- Burn-Rate Audit: Does this alert page on a temporary 10-second blip, or does it require sustained budget consumption across both 1-hour and 5-minute windows?
- Exemplar Linkage: Can an operator click from the 99th-percentile latency bucket in Grafana directly to the offending Dapper trace?
Prohibited Anti-Patterns
- Dynamic Identifiers in Metric Labels: Passing
user_id,userId,account_id,order_id,email,ip, orreq.urlinto Prometheus/OTel metric attributes. - Unstructured
logger.error/console.log: Emitting string-interpolated logs without structured JSON fields and trace correlation. - Gauge Latency Averages: Recording latency averages or percentiles using a Gauge instead of a Histogram.
- Context Dropping in Queues: Enqueuing tasks to Kafka/SQS/BullMQ/Outbox without injecting
traceparent. - Noisy Single-Threshold Alerts: Paging on-call when CPU > 80% without user-facing latency or error rate degradation.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

