Observability Debugging

Narrow a symptom to a service and a cause by using each signal for what it is actually good at.

Purpose

Use this when:

• A service is slow or erroring and you have telemetry but no conclusion.
• You need to decide which of several services in a request path is at fault.
• Dashboards disagree with what users report.
• You have traces available and are not getting value from them.

Do not use this when:

• The telemetry itself is broken — use prometheus-troubleshooting.
• You are in an active outage and need to stabilise first — use incident-triage. Deep investigation comes after mitigation.
• You are defining what to measure — use slo-design.

What this skill assumes: metrics exist for the services in the path, logs are queryable, and traces may or may not exist. It works without traces, but says where they would have saved time.

Operating Procedure

Each signal answers a different question. Using the wrong one wastes the most valuable minutes.

The order that works: metrics to detect and scope → traces to localise → logs to explain. Starting with logs on a distributed system means reading a million lines to find a pattern that one metric query would have shown in seconds.

1. State the symptom with numbers. What, measured where, how much, since when.
2. Scope it with metrics. Which service, endpoint, region, version, customer segment?
3. Find the boundary. The first service in the path whose own latency or error rate moved.
4. Localise with traces. Compare a slow trace against a fast one for the same operation.
5. Explain with logs. Filter by the trace ID, or by the narrowest dimension you found.
6. Confirm causality. Correlation in time is not cause; check that the timeline actually works.
7. Hand off to the skill for whichever layer you landed in.

Label every finding: Observed from a query, Assumed, or Unverified. In observability work the most common error is asserting a cause that only correlates.

Investigation

Step 1 - scope with metrics

sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
  / sum by (service) (rate(http_requests_total[5m]))

histogram_quantile(0.99,
  sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))

Then split by the dimensions that separate a global problem from a local one:

sum by (version) (rate(http_requests_total{status=~"5..",service="<svc>"}[5m]))
sum by (zone) (rate(http_requests_total{status=~"5..",service="<svc>"}[5m]))
sum by (endpoint) (rate(http_requests_total{status=~"5..",service="<svc>"}[5m]))

This is the highest-value step in the whole skill. The answer usually falls out immediately:

Note the p99 caveat: histogram_quantile over a coarse bucket layout is an estimate, and if the highest bucket boundary is below your actual latency, the result is the +Inf bound and is meaningless. Check the bucket layout before trusting a p99.

Step 2 - find the boundary in the request path

histogram_quantile(0.99, sum by (service, le) (rate(http_server_duration_seconds_bucket[5m])))
histogram_quantile(0.99, sum by (service, peer_service, le) (rate(http_client_duration_seconds_bucket[5m])))

Server-side duration is time this service was responsible for. Client-side duration is time it spent waiting on someone else. The difference between them is where the service's own work happens.

Walk the path from the edge inwards until you find the first service whose server latency rose while its client latency did not. That service is the source; everything upstream is just reporting the consequence.

This distinction matters because a latency graph propagates upward through every caller. A dozen services all showing a p99 spike is one service with a problem and eleven victims.

Step 3 - localise with traces

Get a slow trace and a fast trace for the same operation, from the same period, and compare them span by span.

What to look for:

The gap case is the one worth learning to recognise. Unattributed time between spans is invisible in every per-span breakdown, and it is usually a queue: threads, connections, or the CPU run queue.

Without tracing, approximate it by comparing each service's own server-side latency histogram across the path. It is slower and cruder, and the gap case is effectively invisible — which is the concrete argument for adopting tracing.

Step 4 - explain with logs

{service="<svc>"} |= "<trace_id>"
{service="<svc>"} | json | status >= 500 | line_format "{{.msg}}"

Search by trace ID if you have one. Otherwise narrow by the dimension the metrics identified — endpoint, version, tenant — and by a tight time window around the change.

Look for: the first occurrence (when did it actually start, versus when did it get noticed), a change in log volume as well as content, and errors that stopped appearing. A dependency that went quiet is as significant as one that started erroring.

Step 5 - confirm the timeline

changes(kube_deployment_status_observed_generation[1h]) > 0

Build an explicit timeline: symptom start, deploy times, config changes, cron schedules, traffic changes, dependency incidents. Then check the direction of the relationship — a deploy that finished 20 minutes after the errors started did not cause them, however tempting it is.

Decision Tree

Symptom reported
├── Do metrics confirm it?
│   ├── No → the monitoring may be the problem → prometheus-troubleshooting
│   │        (or the symptom is real and unmeasured, which is itself the finding)
│   └── Yes → scope it
├── Concentrated in one version → deploy → roll back first, investigate after
├── Concentrated in one zone/node → infrastructure → linux-troubleshooting
├── Concentrated in one endpoint → that code path or its dependency
├── Spread evenly across everything
│   ├── Shared dependency: DB, cache, DNS, auth, external API
│   └── Check the dependency's own metrics before its clients'
├── Latency up, errors flat → saturation or a slow dependency
│   ├── Trace shows one slow span      → that operation
│   ├── Trace shows gaps between spans → queueing (threads, pool, CPU)
│   └── No traces → compare per-service server-side histograms
└── Errors up, latency flat → fast failures: validation, auth, circuit breaker open

Common Causes

Safety

Read-only

Everything in this skill. Querying metrics, reading logs, and viewing traces do not change the system under investigation.

Two genuine costs to keep in mind:

• An expensive query — an unbounded log search over weeks, or a high-cardinality PromQL aggregation — can degrade the observability stack itself. Losing your monitoring in the middle of an incident is a real and self-inflicted way to make things worse. Bound time ranges and label selectors.
• Raising log verbosity or trace sampling in production is a change: it increases CPU, network and storage cost, and on a saturated service can push it over. Treat it as a mutating action with a plan to revert, not as a free observation.

Potentially disruptive

• Increasing log level or sampling rate on a live service — usually requires a restart or a config push, and increases load. Revert it explicitly afterwards; temporary debug logging left on is a recurring cost and a common cause of later disk-full incidents.
• Adding an instrumentation library or a sidecar to diagnose an incident — that is a deploy during an incident, with all the risk that implies. Almost always the wrong trade.

Destructive

This skill proposes no destructive operations.

Verification

Verification here is about the conclusion, not about a fix — this skill ends when you have identified the layer.

Claim:  p99 latency rose from 200ms to 4.2s at 14:05 UTC because checkout-service's
        calls to the payment API began taking 3.8s.
Basis:  Observed - checkout http_client_duration p99 to peer=payment rose at 14:05;
        checkout's own server duration minus client duration is unchanged;
        payment's own server-side p99 rose at 14:04.

A conclusion is well-founded when:

1. The timing works: the cause preceded the effect.
2. The scope matches: everything affected shares the proposed cause, and unaffected things do not.
3. A prediction holds: if the cause is X, then Y should also be true — check Y.
4. An alternative was eliminated, not just ignored.

If the proposed fix is applied, verify with the same query that first showed the symptom, over a window long enough to cover the pattern's period.

Before: checkout p99 4.2s; payment client_duration p99 3.8s; error rate 6.1%
After:  checkout p99 210ms; payment client_duration p99 95ms; error rate 0.02% over 2h

Both numbers matter. If the downstream metric recovered and the symptom metric did not, the correlation you acted on was not the cause — which is a result worth recording rather than explaining away.

Anti-Patterns

Bad: Start by grepping logs. Better: In a distributed system, logs are the last step. Metrics tell you where to grep; without that, you are reading a million lines looking for a pattern you have not defined yet.

Bad: Every service is showing high latency, so it is the network. Better: Latency propagates upward through callers. Find the first service whose own server-side latency rose. Eleven of the twelve graphs are victims.

Bad: Errors started around the time of the deploy, so the deploy caused them. Better: Check the direction and the gap. "Around the time" has ended plenty of investigations at the wrong culprit while the real cause — a cron job on the hour — kept running.

Bad: p99 is fine, so users are fine. Better: p99 over all requests hides a fully broken endpoint that serves 0.5% of traffic, and hides one customer for whom everything fails. Break down before concluding.

Bad: Add more dashboards. Better: During an incident, more panels is more to scan. A handful of well-chosen queries beats a wall of graphs; the useful dashboard is the one built around the four signals for this service.

Bad: Turn on debug logging in production and leave it. Better: It changes the system's cost and performance. Set a reminder to turn it off, and expect it to be part of the next disk-full incident if you do not.

Composition

• prometheus-troubleshooting — the telemetry itself is missing or wrong.
• slo-design — deciding what should have been measured in the first place.
• production-debugging — once the failing layer is identified.
• incident-triage — if this is an active outage, mitigate before investigating.

References

• OpenTelemetry: semantic conventions — the attribute names used in this skill's queries.
• Google SRE Book: Monitoring Distributed Systems — the four golden signals and why symptom-based monitoring wins.