observability-and-slo-gate
Skill4 starsChanged 31 days agoMITView on GitHub
Enforces Google Dapper and OpenTelemetry production observability architecture—strictly separating low-cardinality metrics (bounded label sets and P50/P95/P99 latency Histograms) from high-cardinality traces and structured logs, guaranteeing unbroken end-to-end W3C traceparent and correlation_id context propagation across HTTP, gRPC, message queues, outbox workers, and database queries, and gating releases with Multi-Window Burn-Rate SLOs. Use when instrumenting APIs, background jobs, queues, or database layers, adding Prometheus/OpenTelemetry/Datadog metrics, structured logging, distributed tracing, defining SLIs/SLOs, or auditing code for metric cardinality explosions (user_id in labels) and broken trace trees — even if the user doesn't explicitly say "observability gate". Do NOT use for purely static markdown formatting or offline one-off scripts with no runtime telemetry requirements.
In this repository47 other files it gives its agents.
- AGENTS.mdAGENTS.md
- anti-slop-engineeringSkill
- anti-slop-engineeringSkill
- api-ergonomics-and-contract-governorSkill
- apple-craft-and-frame-budgetSkill
- apple-craft-and-interaction-physicsSkill
- asset-provenance-guardSkill
- backprop-guardSkill
- chaos-and-degradation-choreographerSkill
- chromatic-drift-guardSkill
- cognitive-load-and-attention-governorSkill
- comment-budget-guardSkill
- context-window-and-token-economics-architectSkill
- cryptographic-agility-and-secret-lifecycle-gateSkill
- decision-guardSkill
- demock-guardSkill
- design-system-drift-guardSkill
- deterministic-simulation-and-property-testingSkill
- distributed-consensus-and-concurrency-gateSkill
- finops-and-unit-economics-governorSkill
- future-scale-gateSkill
- grumpy-senior-staff-engineerSkill
- honeypot-trap-guardSkill
- house-tell-guardSkill
- install-anti-slopSkill
- llm-eval-and-prompt-injection-fortressSkill
- mandela-guardSkill
- mechanical-sympathy-and-perf-gateSkill
- mechanical-sympathy-profilerSkill
- motion-runtime-guardSkill
- observability-and-slo-architectSkill
- paranoid-principal-architectSkill
- pr-boundary-guardSkill
- privacy-and-data-lineage-gateSkill
- re0-handoff-guardSkill
- reduction-guardSkill
- repo-memory-and-invariant-stewardSkill
- root-fix-gateSkill
- safe-evolution-and-rollout-choreographerSkill
- scientific-fault-isolationSkill
- styleseed-guardSkill
- supply-chain-and-sbom-sentinelSkill
- tagore-guardSkill
- type-level-and-ast-slop-gateSkill
- ui-and-copy-disciplineSkill
- verification-oracle-architectSkill
- zero-downtime-migration-gateSkill
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 4 are distinctive, and 6 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 22
Unbroken End-to-End W3C Context Propagation: Every synchronous RPC and asynchronous queue/outbox/worker hop propagates traceparent …
- Distinctive: line 23
Actionable SLIs & Multi-Window Burn-Rate SLOs: Every critical service boundary defines latency (P95/P99) and availability SLIs backed by …
- Very specific: line 43
Never: place unbounded or high-cardinality values (user_id, customer_id, account_id, request_id, trace_id, span_id, correlation_id, …
- Very specific: line 50
Every request entering at the edge must extract or mint a W3C Trace Context (traceparent: …
- Very specific: line 57
Every log record must be emitted as structured JSON containing at minimum: timestamp, level, message, service.name, trace_id, span_id, …
- Distinctive: line 59
Every critical path must define quantitative Service Level Indicators (SLIs) covering Latency (Histogram P95 and P99), Availability …
- Very specific: line 67
Always normalize HTTP routes before recording metrics: Use the matched router template (/api/v1/accounts/:accountId/transfers) for …
- Distinctive: line 88
Before adding a metric, span, log line, or background job handler, answer these five questions:
- Very specific: line 116
In-Memory Telemetry Verification & Guard Audit: Verify in tests that emitted metrics contain zero high-cardinality labels and that traceId …
- Distinctive: line 118
When a durable observability and SLO assessment record is warranted, save a JSON assessment following the Observability & SLO Record …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Skill: Observability & SLO Gate (Google Dapper / OpenTelemetry & Multi-Burn-Rate SRE School)
Production code that executes without telemetry is blind; production code that puts
user_idinto a Prometheus metric label is a financial time bomb. Enforce Low-Cardinality Metrics with P50/P95/P99 Histograms, High-Cardinality Traces/Logs, and Unbroken W3CtraceparentContext Propagation from the first frontend click to the final database statement.
Why This Skill Is Needed & Target State
AI-generated services typically pass local functional tests while introducing two catastrophic production observability failures:
- Metric Cardinality Explosion (The Datadog/Prometheus Bankruptcy Trap): Agents routinely attach unbounded dynamic identifiers (
user_id,request_id,order_id,email,ip_address,trace_id, or unrouted raw URLs like/orders/9481-a8f2/pay) as Prometheus or OpenTelemetry Metric labels/attributes. Because every unique label combination creates a new time-series in the TSDB, 100,000 active users across 10 routes and 5 status codes instantly spawn 5,000,000 active time-series—OOM-killing Prometheus collectors and exploding Datadog/CloudWatch custom metric bills. - Broken Async Trace Causality & Blind Production Failures: Agents emit unstructured
console.log("Failed", err)or start HTTP spans that vanish the moment work is enqueued to Kafka, SQS, RabbitMQ, BullMQ, Redis Streams, an Outbox table, or a worker pool. Without explicit W3Ctraceparentandcorrelation_idinjection and extraction across async boundaries, production incidents become impossible to trace from the user's frontend click down to the failing SQL query.
The target state is Google Dapper & OpenTelemetry Grade Observability:
- Strict Cardinality Split: Metrics (
Counter,Histogram,UpDownCounter) use strictly bounded, low-cardinality label sets ($\prod |L_i| \le 1,000$ per metric family) and record latency distributions via Histograms for accurate P50, P95, and P99 quantile calculation. All high-cardinality diagnostic identifiers (user_id,tenant_id,order_id,idempotency_key,request_id) live exclusively inside Trace Span Attributes and Structured JSON Logs. - Unbroken End-to-End W3C Context Propagation: Every synchronous RPC and asynchronous queue/outbox/worker hop propagates
traceparent(00-<trace_id>-<span_id>-<flags>),tracestate, andcorrelation_id, maintaining a single causal trace tree from client UI to database query. - Actionable SLIs & Multi-Window Burn-Rate SLOs: Every critical service boundary defines latency (P95/P99) and availability SLIs backed by Google SRE Multi-Window, Multi-Burn-Rate error-budget alerts.
Activation and Boundaries
| Situation | Behavior |
| Adding Prometheus, OpenTelemetry, StatsD, or Datadog metrics to HTTP routes, RPC handlers, or workers | Enforce the Low-Cardinality Metric Law (allowlist bounded labels only; ban user_id/request_id/raw URLs) and use Histograms for latency |
| Implementing message producers/consumers (Kafka, SQS, BullMQ, RabbitMQ), transactional outboxes, or background workers | Require explicit W3C traceparent and correlation_id injection on publish/enqueue and extraction before worker execution |
| Authoring error handlers, logging middleware, or database query wrappers | Enforce structured JSON logs automatically correlated with trace_id, span_id, and correlation_id, with PII/secret redaction |
| Designing service readiness, production launch checklists, or alerting rules | Define Golden Signal SLIs (P50/P95/P99 Histogram quantiles + Error Rate) and Multi-Window Burn-Rate SLO gates |
| Auditing PR diffs or existing services for observability defects or metric cost spikes | Scan for high-cardinality metric labels, console.log noise, unpropagated async contexts, and gauge-averaged latencies |
| Purely static documentation edits, local one-off CLI formatting scripts, or CSS token tweaks | Do not activate; use anti-slop-engineering |
The Five Absolute Laws
- The Low-Cardinality Metrics & High-Cardinality Traces/Logs Split Law:
- Never place unbounded or high-cardinality values (
user_id,customer_id,account_id,request_id,trace_id,span_id,correlation_id,session_id,order_id,idempotency_key,email,ip_address,error.message, or raw unparameterized URL paths likereq.url/req.path) inside Metric labels/attributes. - Restrict Metric attributes to a compile-time bounded allowlist of low-cardinality dimensions (
http.request.method, parameterizedhttp.routelike/v1/orders/:id,http.response.status_codeorstatus_class,rpc.system,error.codeenum,queue.name,worker.outcome). The Cartesian product of all label values for any single metric family must remain $\prod_{i} |L_i| \le 1,000$. - Attach high-cardinality forensic attributes (
app.user_id,app.tenant_id,app.order_id,app.idempotency_key,http.request_id) exclusively to OpenTelemetry Span Attributes (span.setAttribute(...)) and Structured JSON Log fields.
- Never place unbounded or high-cardinality values (
- The Histogram Quantile Law (Measure P50 / P95 / P99, Never Averages):
- Arithmetic averages (
avg_latency_ms = total_ms / count) hide tail-latency outages where 1%–5% of users experience 30-second timeouts. Pre-computed per-pod percentiles (gauge.set(p99)) are mathematically invalid because percentiles cannot be averaged or summed across horizontal replicas. - Record all request durations, queue dwell times, and database query latencies using Histograms (
Histogramwith explicit domain-calibrated bucket boundaries, e.g.,[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]seconds, or native OpenTelemetry Exponential Histograms) so P50, P95, and P99 quantiles can be aggregated accurately across any fleet dimension.
- Arithmetic averages (
- The Unbroken W3C Context Propagation Law (
traceparent+correlation_id):- Every request entering at the edge must extract or mint a W3C Trace Context (
traceparent: 00-<32-hex-trace-id>-<16-hex-span-id>-<2-hex-flags>,tracestate) and a deterministiccorrelation_id. - Context must never drop across asynchronous boundaries:
- HTTP / gRPC Egress: Inject
traceparent,tracestate, andx-correlation-idheaders into every outbound request. - Queues, Event Buses & Transactional Outboxes: Serialize
traceparentandcorrelation_idinto the message headers or outbox row columns at enqueue time (propagation.inject), and extract them inside the worker (propagation.extract) to start a child or linkedCONSUMERspan before executing business logic. - Database Queries: Propagate active context to the database driver and annotate query spans with low-cardinality operation names (
SELECT orders.by_id) or SQLcommenter metadata (/*traceparent='00-...'*/) without leaking literal SQL parameter values.
- HTTP / gRPC Egress: Inject
- Every request entering at the edge must extract or mint a W3C Trace Context (
- The Structured Log-Trace Correlation & Redaction Law:
- Unstructured
console.log,console.error,print(), orfmt.Printlncalls in production service paths are prohibited. - Every log record must be emitted as structured JSON containing at minimum:
timestamp,level,message,service.name,trace_id,span_id,correlation_id, and high-cardinality domain context (user_id,order_id,error.code), while enforcing deterministic redaction of credentials (Authorization,cookie,password,secret,token,api_key, full PAN/SSN) prior to serialization.
- Unstructured
- The Golden Signals & Multi-Window Multi-Burn-Rate SLO Law:
- Every critical path must define quantitative Service Level Indicators (SLIs) covering Latency (Histogram P95 and P99), Availability (non-5xx / valid outcome ratio), and Saturation (queue depth / connection pool utilization / consumer lag).
- Gate production operations on an explicit Error Budget ($1 - \text{SLO}$) using Google SRE Multi-Window, Multi-Burn-Rate Alerts (e.g., Fast Burn
14.4xover1h+5mconsuming 2% budget -> Page; Slow Burn6xover6h+30mconsuming 5% budget -> Ticket).
Invariant Rules
- Enforce a Metric Attribute Allowlist Decorator/Wrapper. Wrap metric recorders (
counter.add,histogram.record) in a strongly typed helper that accepts only compile-time bounded attribute keys and strips or rejects forbidden high-cardinality keys (user_id,request_id,trace_id,email,ip,url). - Always normalize HTTP routes before recording metrics. Use the matched router template (
/api/v1/accounts/:accountId/transfers) forhttp.route—never the raw request URL (/api/v1/accounts/acc_991823/transfers). If a request hits an unmatched404path, maphttp.routeto the constant"__unmatched__"to prevent 404 scanner attacks from exploding metric cardinality. - Require
traceparentandcorrelation_idfields in every Queue/Outbox envelope schema. Define async job/event TypeScript/Pydantic/Protobuf schemas with mandatorytraceparent: stringandcorrelationId: stringmetadata fields so a producer cannot compile without passing trace context. - Bind OpenTelemetry Exemplars to Histogram recordings. When recording a Histogram measurement inside an active sampled span, attach the span's
trace_idandspan_idas a Metric Exemplar so engineers can jump directly from a P99 latency spike on a Grafana histogram bucket to the exact offending Dapper trace. - Record exceptions on the active span before rethrowing or mapping. Whenever an operation catches an error, call
span.recordException(err)andspan.setStatus({ code: SpanStatusCode.ERROR, message: err.code })so trace trees surface the exact failure boundary. - Verify telemetry in unit/integration tests with In-Memory Exporters. Test suites for instrumented services must assert against an
InMemorySpanExporterandInMemoryMetricReaderthat (a) zero banned high-cardinality keys appear in emitted metric attributes, and (b) producer and consumer spans share the exact sametraceIdacross async queue/outbox hops.
Execution Router
| Task Shape | Primary Workflow | Reference Contract |
| Instrumenting metrics, replacing high-cardinality labels, or configuring P50/P95/P99 latency Histograms | Enforce Low-Cardinality Metrics & Histograms | Low-Cardinality Metrics & Histograms |
| Instrumenting HTTP/gRPC boundaries, background workers, message queues, outboxes, and DB spans | Propagate W3C Trace Context End-to-End | W3C traceparent & Async Context Propagation |
| Setting up structured trace-correlated logging, Golden Signal SLIs, and Multi-Burn-Rate SLOs | Define Golden Signals & Burn-Rate SLOs | Structured Logs, SLIs & Burn-Rate SLOs |
| Durable observability & SLO assessment record generation and source hazard audit | Record Contract | scripts/observability_guard.py |
Mandatory Pre-Instrumentation Interrogation Matrix
Before adding a metric, span, log line, or background job handler, answer these five questions:
- The Cardinality Bomb Check: What is the maximum number of unique values for every attribute on this metric? Could
user_id,tenant_id,order_id,request_id,email,ip,error.message, or an unparameterized URL (req.url) ever enter this label? What happens to the TSDB if 1,000,000 malicious 404 requests hit random URLs? - The Quantile Aggregation Check: Am I measuring latency or payload size with a
Histogramconfigured with explicit buckets (to compute P50, P95, P99 across pods), or did I mistakenly emit aGaugeof averages/percentiles that cannot be aggregated horizontally? - The Async Context Hop Check: When this HTTP handler inserts a job into a queue, Redis stream, or transactional outbox table, where are
traceparentandcorrelation_idserialized? When the worker wakes up 10 seconds later in another process, how does it extracttraceparentso its database span appears under the original frontend trace? - The 3:00 AM On-Call Correlation Check: If thisP99 latency SLO alert fires at 3:00 AM, can the on-call engineer go from the Histogram Exemplar -> to the exact
trace_id-> to all structured JSON logs across the API gateway, queue worker, and DB layer using a singletrace_idorcorrelation_idquery? - The Secret & PII Redaction Check: Could any span attribute or structured log field inadvertently capture an
Authorizationheader, session token, webhook signature, password, or unmasked customer PII?
Prohibited Anti-Patterns (Instant Rejection List)
- High-Cardinality Identifiers in Metric Labels: Passing
user_id,userId,customer_id,request_id,trace_id,span_id,correlation_id,order_id,session_id,email,ip, orerror.messageinto Prometheus/OpenTelemetry metric attributes (counter.add(1, { user_id })). - Unnormalized
req.url/req.originalUrl/req.pathin Metric Labels: Recording raw URL strings containing path parameters or query strings in metrics instead of parameterized route templates (http.route = "/v1/orders/:id") and"__unmatched__"for 404s. - Fire-and-Forget Async Enqueue Without Context Injection: Pushing payloads to queues, worker pools,
setImmediate, or outbox tables without injecting W3Ctraceparentandcorrelation_idinto the envelope and extracting them on the consumer side. - Gauge-Averaged Latency or Pre-Aggregated Pod Percentiles: Tracking latency by writing
latencyGauge.set(durationMs)or computingsum/countaverages without aHistogramsupporting P50/P95/P99 bucket quantiles. - Unstructured
console.log/printin Production Paths: Emitting raw text logs lackingtrace_id,span_id, andcorrelation_idJSON fields, or logging raw error/request objects without PII/secret redaction. - Silent Span Error Swallowing: Catching an exception inside a traced block without calling
span.recordException(err)and settingSpanStatusCode.ERROR.
Required Output & Review Workflow
Structure all observability implementations, telemetry refactors, and production readiness audits through this 4-phase discipline:
- Cardinality Budget & Telemetry Contract: List every metric family, its instrument type (
counter,histogram,up_down_counter), its bounded label allowlist, and the mathematical upper bound of its label cardinality ($\prod |L_i| \le 1,000$). Map all high-cardinality identifiers to Trace Span Attributes and Structured Log fields. - End-to-End W3C Context Propagation Wiring: Implement W3C
traceparent(00-<traceId>-<spanId>-<flags>),tracestate, andcorrelation_idextraction at ingress, propagation throughAsyncLocalStorage/Context, header injection into queues/outboxes/HTTP clients, and extraction in background workers down to database query spans. - Histogram SLIs & Multi-Burn-Rate SLO Gate: Configure latency Histograms with explicit bucket boundaries for P50/P95/P99 computation, bind trace Exemplars, wire structured JSON logging with secret redaction, and specify the Multi-Window Burn-Rate SLO alerts (
14.4xfast burn /6xslow burn). - In-Memory Telemetry Verification & Guard Audit: Verify in tests that emitted metrics contain zero high-cardinality labels and that
traceIdis preserved across async producer->consumer hops, then audit the record and source code withscripts/observability_guard.py.
When a durable observability and SLO assessment record is warranted, save a JSON assessment following the Observability & SLO Record Contract and audit it with:
npx anti-slop-engineering observability audit <record.json> --json
# or: python3 scripts/observability_guard.py audit <record.json> [--source <instrumented_file.ts>] --jsonDomain References
- Observability & SLO Gate Playbook Suite: Central orchestration index for Google Dapper distributed tracing, OpenTelemetry semantic conventions, cardinality control, and SRE SLO alerting.
- Low-Cardinality Metrics, Histograms & Multi-Burn-Rate SLOs Playbook: Cardinality math, compile-time attribute allowlists, Histogram bucket design for P50/P95/P99, Exemplars, and Google SRE Multi-Window Burn-Rate formulas.
- Google Dapper & OpenTelemetry End-to-End Context Propagation Playbook: W3C
traceparentwire format, HTTP/gRPC/Queue/Outbox producer-consumer injection and extraction, database SQLcommenter spans, and trace-correlated structured JSON logging. - Low-Cardinality Metrics & Histograms Reference: Production TypeScript, Python, Go, and Rust patterns for type-safe metric wrappers, route normalization, and Histogram buckets.
- W3C
traceparent& Async Context Propagation Reference: End-to-end context carrier propagation across HTTP, BullMQ/Kafka/SQS queues, transactional outboxes, and database queries. - Structured Logs, SLIs & Burn-Rate SLOs Reference: Trace-correlated JSON log serializers with secret redaction, Golden Signals, and PromQL/OTel Multi-Window Burn-Rate alert specifications.
- Observability & SLO Record Contract: JSON schema and automated verification rules for
scripts/observability_guard.py.
Failure Taxonomy
| Failure Mode | Observable Signal | Remediation |
| Metric cardinality explosion | Prometheus OOM or massive Datadog custom metrics bill; user_id, request_id, or req.url in metric labels | Strip high-cardinality keys from metrics via compile-time allowlist; move user_id/request_id to Span Attributes and Structured Logs |
| 404 scanner cardinality attack | Millions of unique http.route time-series from bots probing random URLs | Record matched route template (/v1/items/:id) only; collapse all unmatched routes to "__unmatched__" |
| Orphaned worker spans | Queue consumer or cron worker starts a brand-new root trace disconnected from the API request | Inject traceparent and correlation_id into the job/outbox payload on publish and extract parent context before starting the worker span |
| Blind tail-latency outages | Dashboard shows avg_latency = 45ms while P99 users time out at 15,000ms | Replace average/gauge latency metrics with a bucketed Histogram and alert on P95/P99 quantiles |
| Uncorrelated log grep | Production error logged via console.error(err) with no trace_id or correlation_id | Emit structured JSON logs enriched automatically from active span context (trace_id, span_id, correlation_id) |
| Telemetry secret leak | Raw headers or payloads logged into span attributes or JSON logs | Enforce a deterministic field redaction allowlist/blocklist ([REDACTED]) before log/span emission |
Delegated Skills
anti-slop-engineering: Enforce core SRE discipline (KISS, DRY, YAGNI, Fix Issues at Scale) alongside telemetry instrumentation.paranoid-principal-architect: Pair W3Ccorrelation_idandtraceparentpropagation with idempotency keys, boundary schema validation, and deterministic error codes.mechanical-sympathy-profiler: Pair P50/P95/P99 Histogram latency telemetry with pprof/FlameGraph profiling to eliminate tail-latency CPU/allocation bottlenecks.scientific-fault-isolation: Use distributed traces, span events, and trace-correlated structured logs as primary diagnostic evidence during bug reproduction.verification-oracle-architect: Verify telemetry exporters (InMemorySpanExporter,InMemoryMetricReader) using deterministic contract assertions without module mocks.
Completion and Limits
The task is complete when all emitted metrics enforce a strict low-cardinality attribute allowlist ($\prod |L_i| \le 1,000$ with zero dynamic IDs or raw URLs), latency distributions are captured via explicit Histograms supporting P50/P95/P99 quantiles with Exemplars, W3C traceparent and correlation_id propagate unbroken across HTTP, async queues/outboxes, background workers, and database spans, structured JSON logs correlate with trace_id/span_id and redact secrets, Multi-Window Burn-Rate SLOs are defined, and scripts/observability_guard.py reports zero findings. A review may validly output refuse or pause if code injects high-cardinality IDs into metric labels or drops trace context across async worker hops.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

