Observability scan

A codebase's telemetry code is a statement about what its authors expect to go wrong and what they can afford to see. This scanner reads that statement: which signals the code emits (logs, metrics, traces, crash reports, health states), how disciplined the emission is, where the data flows, and — the inferential step — what monitoring the emitted signals can and cannot support. You cannot see the team's dashboards from the repo, but you can say "these metrics exist, so alerting on X is possible; nothing measures Y, so Y is invisible in production."

First read sokrates-scan-core/SKILL.md (sibling skill) — it defines the output format, evidence rules, validate/render scripts, and the _sokrates data layout. This file adds only what is specific to observability scanning.

Workflow

1. Orient per the core skill; use the tech-stack findings (_sokrates/reports/ai-insights/tech-stack-scan.json, if present) to learn the telemetry SDKs already identified rather than rediscovering them.
2. Inventory the emission surface by grep, then read the hits that matter. Search for the ecosystem's signal vocabulary — for Rust: tracing::/info!/warn!/error!, metrics/counter/histogram/Gauge, #[instrument], sentry, otel/opentelemetry, panic::set_hook; for JS/TS: console., pino, winston, debug(, Sentry., @opentelemetry; for Python: logging., structlog, sentry_sdk, opentelemetry, prometheus_client. Record rough counts (they go in stats), then read the initialization and export code closely — where the subscriber/provider/exporter is configured is where most of the truth lives.
3. Follow the data out. For each signal type, find where it leaves the process: exporter endpoints, backend SDKs, env vars that enable/point telemetry, file sinks, "no-op unless configured" defaults. Note sampling, batching, and buffering decisions. This is where privacy questions live too: what user content, paths, or identifiers ride along, and what redaction exists.
4. Probe the dark spots. Pick a handful of load-bearing paths (from Sokrates hotspots or the architecture) and check what they emit on failure. Grep for error-swallowing shapes (let _ =, empty catch, .ok() on fallible calls, logged-then-dropped errors) in those paths specifically — a repo-wide count is noise, a swallowed error on the payment path is a finding.
5. Infer the monitoring posture. From the assembled evidence, state what a production operator of this system could alert on today, and what they could not. Mark these likely/possible per the confidence rules — they are inferences, and saying so is the point.
6. Write findings, validate, render, and report per the core workflow. Scanner id: observability-scan, version 1.0.

Group taxonomy

What a good finding looks like

Anchor every claim in emission or configuration code you read: the subscriber setup line, one representative counter! call, the exporter endpoint construction. For hygiene findings, the evidence is the offending line itself (the error! that logs a full request body; the let _ = on the flush). For monitoring-posture findings, evidence cites the signals that do exist while the description reasons about what they enable or miss. That combination — real, validating evidence with likely/possible confidence — is correct, not contradictory: confidence rates the claim (the inference about downstream monitoring), not the citations. Example: evidence cites the OTLP exporter setup and two metric definitions; the description says "an operator who configures OTLP could alert on session failures and latency; nothing measures queue depth, so backpressure is invisible"; confidence likely.

One finding per practice or posture, not per call site: "HTTP client retries are counted but not timed" beats forty citations of counter!. Cluster per subsystem when practices differ (the TUI logs differently than the server — that contrast is itself a finding). For a mid-size to large codebase expect roughly 10–16 findings; fewer for a small project, and prefer merging over splitting when in doubt — granularity consistency is what makes re-runs comparable.

Severity calibration

• high — telemetry that actively harms: secrets/credentials or clearly sensitive user content written to logs or shipped to a backend; error paths that crash the observer.
• medium — operationally blinding or privacy-material: a load-bearing subsystem whose failures are invisible (swallowed errors, no signal on the critical path); telemetry on by default without redaction of user content; unbounded metric cardinality on a hot path.
• On-by-default telemetry that ships only well-minimized data (sanitized low-cardinality metrics, no user content) is low at most — the finding then documents the minimization, and the severity reflects only the residual concern (default-on posture, embedded backend keys). Hardcoded telemetry credentials are worth noting here, but auditing them as a secret-exposure risk belongs to a security scan.
• low — friction and drift: inconsistent level usage that will mistrain operators, duplicated half-adopted telemetry stacks, dead instrumentation.
• info — the descriptive map: what exists, where data flows, what monitoring is possible. Well-instrumented areas deserve explicit info findings — the map of what works is half the audit's value.

Output

Follow the core workflow: write _sokrates/reports/ai-insights/observability-scan.json, validate until OK, render the explorer, report leading with the posture summary (what this system can see about itself, in two sentences) and any above-info findings.

Use these stats keys where they apply (canonical, so re-runs and dashboards can compare): log_call_sites, metric_call_sites, instrumented_fns, exporters (list), and telemetry_default — an object per signal when defaults differ, e.g. {"logs": "off", "traces": "off", "metrics": "statsig"}. Add scanner-specific extras freely alongside.