agentleFS
kapadiaskapadias/nonna/.claude/skills/observability/SKILL.md

observability

Skill0 starsChanged 46 days agoMITView on GitHub

Adding or changing logging, metrics, tracing, or alerting; instrumenting a service. SLIs/SLOs, structured logs, metric cardinality, trace IDs, alerts that page.

90 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use gitops, background-jobs and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • kcns008Skill

    gitops

    GitOps Agent (Flow) — manages ArgoCD applications, Helm charts, Kustomize overlays, deployment strategies (canary, blue-green, rolling), multi-cluster GitOps, and drift detection for Kubernetes and OpenShift clusters.

    Also in 4 projects that use this

  • YepAPISkill

    background-jobs

    Queue workers with BullMQ/Inngest/Trigger.dev, job retries, dead letter queues, concurrency.

    Also in 4 projects that use this

  • Mushfiqur-Rahman-RobinSkill

    rag-systems

    Best practices for building Retrieval-Augmented Generation systems — chunking, embedding, hybrid retrieval, reranking, indexing, caching, evaluation, observability — plus a catalog of RAG patterns and when each applies.

    Also in 3 projects that use this

  • YepAPISkill

    data-pipelines

    ETL patterns, cron scheduling, queue-based processing, and batch jobs with Inngest/BullMQ.

    Also in 4 projects that use this

  • wpankSkill

    caching

    Caching strategies, invalidation, eviction policies, HTTP caching, distributed caching, and anti-patterns. Use when designing cache layers, choosing eviction policies, debugging stale data, or optimizing read-heavy workloads.

    Also in 5 projects that use this

  • JohnJBorenSkill

    ci-cd

    CI/CD pipeline design, GitHub Actions, testing automation, and deployment strategies

    Also in 13 projects that use this

The file

We split this skill into the rules it gives an agent

3 rules. 0 are common, 2 are distinctive, and 1 is very specific to this project.

  • 0

    Common

    Most of the 73 edited versions we read keep these.

  • 2

    Distinctive

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Don't reach for all three reflexively.

    line 16
  2. Distinctive:

    Fail closed in the alerting too: if telemetry stops arriving, _that_ should page (dead-man's switch) — silence can mean the system is down, …

    line 85
  3. Very specific:

    When chasing a live incident, the goal is to _extract_ what's already emitted before adding more: filter logs by the correlation id, read …

    line 98

“Common” is measured: kept by at least 6 in 10 of the 73 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Observability

You instrument so that when prod breaks at 3am, the system can already answer what is broken, for whom, and how badly — without a redeploy to add a log line. The goal is not "more telemetry"; it is the fewest signals that let you debug a failure you haven't seen yet. Telemetry is also a data surface: never log secrets or PII — a log store is a breach waiting to be exfiltrated (.claude/rules/engineering.md, .claude/rules/safety.md).

The three signals — and what each is for

Don't reach for all three reflexively; each answers a different question.

  • Logs — discrete events with context. Answer "what exactly happened for this one request?" Use for errors, state transitions, and audit trail. Expensive at volume; the most detail per event.
  • Metrics — cheap aggregate numbers over time. Answer "is the system healthy, and is it getting worse?" Use for rates, latencies, saturation, and alerting. Pre-aggregated, so nearly free — but bounded by cardinality (below).
  • Traces — one request's path across services with timing per span. Answer "where did the latency go / where did it fail?" in a distributed call. The tool for tail latency and cross-service blame.

Structured logs — machine-parseable, never secret-bearing

Log key-value JSON, not interpolated prose. log.info("charge failed", user_id=u.id, amount_cents=a, reason="card_declined") is queryable; log.info(f"charge failed for {u}...") is a grep tax forever.

  • Every log line carries the correlation/trace id (below) so events join across services.
  • Levels mean things: ERROR = a human must look / something failed; WARN = degraded but handled; INFO = notable state change; DEBUG = off in prod. If everything is ERROR, nothing is.
  • NEVER log: passwords, tokens, keys, full card numbers, auth headers, request/response bodies containing PII, or whole user objects. Log identifiers and outcomes, not payloads. Redact at the logger, not at each call site (one missed call site is a breach). This is a hard line, not a nicety.
  • Sample high-volume info logs, but never sample errors — the one you drop is the one you need.

Metrics — measure with RED and USE, not vibes

Two complementary lenses; instrument the right one for the thing:

  • RED — for request-driven services (the user-facing view): Rate (requests/sec), Errors (failures/sec or error ratio), Duration (latency distribution). This is what tells you the service is hurting users.
  • USE — for resources (queues, pools, CPU, disk, connections): Utilization, Saturation (queue depth / wait time), Errors. This is what tells you why — the saturated pool behind the slow endpoint.

Latency is a distribution — alert and report on percentiles, never the mean. p50/p95/p99; the mean hides the tail where users actually suffer. Use histograms, not averages, so percentiles are computable after the fact.

Cardinality discipline — the #1 way to blow up a metrics bill

A metric's cost is its time series count = product of all label value combinations. Labels must be bounded, low-cardinality sets (status code, method, region, endpoint template). Never label with unbounded values — user_id, request_id, raw path with ids in it, email, full URL, error message. One user_id label on a busy service = millions of series = a melted metrics backend and a shocking invoice. High-cardinality identifiers belong on logs and traces (which are indexed for exactly that), never on metric labels. Pre-register the label set; reject anything per-request.

Correlation & trace IDs — make a request followable end to end

  • Generate a request/correlation id at the edge (or adopt the inbound one) and propagate it through every call, log line, and outgoing request header (W3C traceparent / your tracing SDK).
  • One request → one trace id appears in the gateway log, the service logs, the DB-slow log, and the downstream call. That join is what turns "errors are up" into "this code path, this dependency."
  • Propagate context across async boundaries too (queue messages, background jobs) or the trace snaps at the handoff and you lose the causal chain.

Alerts — page on symptoms, and only when a human must act

An alert that doesn't require human action is noise that trains people to ignore the pager — and the ignored page is the one that was real. Ruthlessly prune.

  • Alert on user-facing symptoms (SLO burn), not internal causes. "p99 latency > 1s for 5m" or "error ratio > 2%" pages; "CPU > 80%" usually shouldn't — high CPU with healthy latency is fine, and the cause belongs in a dashboard you consult after the symptom fires.
  • Every alert is actionable and has a runbook: what it means, how to confirm, first mitigation. An alert with no runbook is a riddle handed to someone half-asleep.
  • Tune for signal: require a duration ("for 5m") to kill flaps; alert on error rate/ratio, not a single failure. A noisy alert is a broken alert — fix or delete it, exactly as with a flaky test.
  • Fail closed in the alerting too: if telemetry stops arriving, that should page (dead-man's switch) — silence can mean the system is down, not healthy.

SLIs / SLOs — define "good enough" as a number, then defend it

  • SLI = the measured indicator of user happiness (success ratio, p99 latency, freshness). Measure it from the user's edge, not deep internals.
  • SLO = the target (e.g. 99.9% of requests succeed over 30 days). It defines the error budget — the allowed failure. Alerts should fire on budget burn rate, so you page fast on a sharp outage and slow on a gentle one, instead of on every blip.

Instrument to debug prod — without a redeploy

When chasing a live incident, the goal is to extract what's already emitted before adding more: filter logs by the correlation id, read the trace for the slow span, check RED/USE dashboards for the inflection point (.claude/skills/debugging for the root-cause method). If you must add telemetry, add a structured log with the trace id (cheap, safe) before a new metric (cardinality risk). Then fold the signal that would have caught it sooner into the permanent instrumentation.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.