agentleFS
Sign inStart free
juandelossantosjuandelossantos/another-agent-skills/skills/observability-and-instrumentation/SKILL.md

observability-and-instrumentation

Skill1 starsChanged 11 days agoMITView on GitHub

Instrument code so production behavior is visible: structured logging, metrics, tracing, alerting. Use when shipping features that need evidence. Do NOT use for local development.

We split this skill into the rules it gives an agent

7 rules. 0 are common, 7 are distinctive, and 0 are very specific to this project.

  • 0

    Common

    Most of the 42 edited versions we read keep these.

  • 7

    Distinctive

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    After an incident postmortem (filling observability gaps discovered during analysis)

    line 28
  2. Distinctive:

    Before a public launch or major feature release (ensuring launch readiness)

    line 29
  3. Distinctive:

    Every production log entry should be a structured event (JSON), not a free-text line.

    line 44
  4. Distinctive:

    Never log secrets, PII, or full request bodies

    line 50
  5. Distinctive:

    Every alert must have a runbook.

    line 95
  6. Distinctive:

    No correlation ID — logs that cannot be linked to a specific request.

    line 100
  7. Distinctive:

    No runbook — an alert with no documented response is noise.

    line 103

“Common” is measured: kept by at least 6 in 10 of the 42 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Observability and Instrumentation

Make production behavior visible and diagnosable.

Complementary to shipping-and-launch. Where shipping handles deployment and rollback, observability handles the ongoing production lifecycle: knowing what your system is doing, detecting when it's wrong, and having the data to diagnose it.

Sources: Google SRE Workbook (sre.google/workbook/monitoring) — defines monitoring data types, RED method, and alert severity. OpenTelemetry Specification (opentelemetry.io/docs/specs/semconv) — semantic conventions for metrics, logs, and traces. OpenTelemetry Span Metrics Connector (opentelemetry.io/docs/collector/registry) — RED metrics from trace spans. SRE Prodcast (sre.google/prodcast) — three pillars of observability.

When to Use

  • Adding logging, metrics, tracing, or alerting
  • Shipping features that run in production
  • Production issues reported but you can't tell what happened
  • Setting up monitoring dashboards
  • After an incident postmortem (filling observability gaps discovered during analysis)
  • Before a public launch or major feature release (ensuring launch readiness)

Output Contract

ArtifactFormatLocationQuality Criteria
Instrumented source code + monitoring configSource code changes (any language) + config files (YAML, JSON)Existing source tree + monitoring/ or otel/JSON structured logs with correlation IDs on every entry, RED metrics (rate/errors/duration) per service, USE metrics (utilization/saturation/errors) per resource, distributed tracing on every external call with OpenTelemetry span attributes, symptom-based alerts with documented runbooks, no secrets or PII in log output, monitoring configuration treated as code

When NOT to Use

  • Deployment or rollback planning (use shipping-and-launch)
  • Pre-production work

Structured Logging

Every production log entry should be a structured event (JSON), not a free-text line. Structured logs are machine-parseable, searchable, and carry context (request ID, user ID, latency) that free-text cannot.

Rules:

  • Log in JSON format
  • Include correlation ID on every entry
  • Log levels: DEBUG, INFO, WARN, ERROR, FATAL
  • Never log secrets, PII, or full request bodies

See guides/STRUCTURED-LOGGING.md for format templates and correlation ID patterns.

Metrics — RED Method

For every service, instrument three metrics per the Google SRE RED method:

MetricWhat It Measures
RateRequests per second
ErrorsFailed requests per second
DurationLatency distribution (histogram)

Metrics — USE Method

For every resource (CPU, memory, disk, network), instrument per the USE method:

MetricWhat It Measures
UtilizationPercent of time resource is busy
SaturationQueue depth or pressure
ErrorsError count

RED is for services. USE is for resources. Both are needed.

See guides/METRICS-AND-TRACING.md for metric naming conventions per OpenTelemetry semantic conventions.

Distributed Tracing

Traces follow a request across service boundaries. Every trace has a Trace ID, spans, and span context. Trace every external call (HTTP, database, queue, cache). Without tracing, you cannot diagnose latency in distributed systems.

See guides/METRICS-AND-TRACING.md for span attributes per OpenTelemetry semantic conventions.

Alerting

Symptom-based alerts, not cause-based. Alert on what users experience (latency, errors), not on internal details (CPU at 80%). Per Google SRE:

LevelMeaningResponse Time
P0Service down or data lossImmediately
P1Severe degradationWithin 1 hour
P2Partial degradationWithin 1 day
P3Minor issueNext sprint

Every alert must have a runbook. Without a documented response, the alert will be ignored.

Anti-Patterns

  1. Log spam — logging every line of execution instead of at boundaries.
  2. No correlation ID — logs that cannot be linked to a specific request.
  3. Dashboard-only monitoring — a dashboard without alerts means nobody watches it.
  4. Cause-based alerting — alerting on CPU instead of latency.
  5. No runbook — an alert with no documented response is noise.

Verification

  • Every service has RED metrics
  • Every resource has USE metrics
  • Logs are structured (JSON) with correlation IDs
  • External calls have distributed tracing instrumentation
  • Alerts are symptom-based with documented runbooks
  • No secrets or PII in log output

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free