Production Debugging

Narrow a production problem to a cause you can test, without breaking the system further while doing it.

Purpose

Use this when:

• An incident is mitigated and you need the actual cause.
• A degradation is real but not urgent: intermittent errors, creeping latency, a daily spike.
• An investigation has stalled and needs restructuring.
• Multiple services are failing and you need to find which one started it.

Do not use this when:

• Users are currently affected and nothing has been mitigated — use incident-triage. Debugging during an unmitigated outage extends the outage.
• The failing layer is already known — go directly to that layer's skill.

What this skill assumes: the system is stable enough to investigate without time pressure, and that you can read telemetry and logs. If either is false, mitigate first.

Operating Procedure

1. Write the problem statement. What, measured where, how much, since when, for whom. If it cannot be written in one sentence with numbers, it is not yet understood well enough to debug.
2. Establish the timeline. First occurrence, not first report. Those differ, often by days.
3. Narrow the scope. Which service, version, zone, endpoint, tenant, time of day? Each dimension that concentrates the problem eliminates a class of causes.
4. Separate cause from symptom. In a distributed system most of what you see is propagation.
5. Form hypotheses, ranked. Write them down. Each must make a prediction that can be checked.
6. Test the cheapest discriminating hypothesis first — the one whose result eliminates the most alternatives, not the one you find most interesting.
7. Confirm with a mechanism, not with a correlation. You should be able to explain every step from cause to symptom.
8. Fix, verify, and record — hand off to postmortem-analysis.

Keep a running log with timestamps of what you ran and what you saw. Mark each finding Observed, Assumed, or Unverified. Investigations fail most often because an assumption from minute three is treated as fact at minute forty.

Investigation

Step 1 - when did it really start

<your sli or error metric>[7d]

Look at a week, not an hour. The reported start time is when someone noticed. A slow degradation that crossed a threshold today may have started on Tuesday, and the change you are looking for is on Tuesday.

The shape tells you a great deal:

A step change is a change you can find. A ramp is a resource you can measure. These lead to completely different investigations, and the shape distinguishes them in one glance.

Step 2 - narrow the scope

Split the failing metric by every dimension you have:

sum by (version)  (rate(errors_total[5m]))
sum by (zone)     (rate(errors_total[5m]))
sum by (endpoint) (rate(errors_total[5m]))
sum by (tenant)   (rate(errors_total[5m]))
sum by (instance) (rate(errors_total[5m]))

"Spread evenly across everything" is itself a strong signal, not a dead end: it means a shared dependency, and the list of those is short — database, cache, DNS, auth, config store, the network, an external API.

Step 3 - separate cause from symptom

In a distributed system, failure propagates. The service that is loudest is usually not the one that broke.

Rules that work:

• Latency propagates upward. If A calls B calls C and C is slow, all three show high latency. Find the deepest service whose own work slowed — server-side duration minus time spent calling others.
• Errors propagate upward too, but they change form: a timeout at C becomes a 500 at B and a 502 at A. Trace the error type back along the path.
• Saturation propagates downward. If A retries, B and C get more load than before, and the increase is worst at the deepest layer.
• The first thing to fail is not always the cause. The weakest component fails first under a pressure created elsewhere.

histogram_quantile(0.99, sum by (service, le) (rate(http_server_duration_seconds_bucket[5m])))
histogram_quantile(0.99, sum by (service, peer_service, le) (rate(http_client_duration_seconds_bucket[5m])))

Step 4 - recognise the standard failure modes

Metastable failure is the one worth recognising explicitly: when removing the original trigger does not fix things, adding capacity usually will not either. You have to shed load to break the cycle.

Step 5 - hypotheses that can be tested

Write each as a prediction:

H1: The 14:05 deploy of checkout-service v2.5.0 introduced a query without an index.
    Predicts: database CPU rose at 14:05; a new slow query appears in pg_stat_statements;
              the problem is absent on pods still running v2.4.1.
    Test:     query pg_stat_statements ordered by total_exec_time; compare error rate by version.
    Cost:     2 minutes, read-only.

H2: A payment-provider degradation is causing timeouts that exhaust the thread pool.
    Predicts: client-side duration to peer=payment rose before our errors;
              the provider's status page shows an event.
    Test:     client duration by peer; check the provider's status page.
    Cost:     2 minutes, read-only.

Test H2 first if it is cheaper and eliminates more. Ranking by likelihood alone is a trap — a slightly less likely hypothesis that takes 30 seconds to eliminate is the better first move.

If a test does not discriminate — both hypotheses predict the same outcome — it is not a test.

Step 6 - confirm the mechanism

Before declaring a cause, state the full chain:

A migration added a query filtering on orders.customer_id, which has no index.
  → sequential scans on a 40M-row table
  → each checkout request holds a database connection for 3.8s instead of 12ms
  → the 20-connection pool is exhausted at ~5 concurrent checkouts
  → further requests wait on the pool, exceed the 10s client timeout
  → the client returns 503

Every arrow must be something you observed or can explain. If a step is "and then somehow", the investigation is not finished — and the fix built on it will not hold.

Decision Tree

Production problem, system stable
├── Shape of the metric over 7 days
│   ├── Step      → find the discrete change at that timestamp
│   ├── Ramp      → find the growing resource
│   ├── Periodic  → find the schedule
│   └── Traffic-correlated → saturation
├── Scope
│   ├── Concentrated → the dimension names the cause
│   └── Everywhere   → shared dependency
├── Multiple services failing
│   ├── Find the deepest service whose own work slowed
│   └── Check whether load rose after the failure → retry storm
├── Does not recover when the trigger is removed → metastable → shed load
└── Cause identified?
    ├── Mechanism explains every step → fix and verify
    └── Only correlation → keep going; correlation has a high false-positive rate here

Common Causes

Safety

Read-only

Querying metrics, reading logs and traces, inspecting configuration, reading database statistics views (pg_stat_statements, SHOW ENGINE INNODB STATUS), and EXPLAIN without ANALYZE.

Watch the cost even here: an unbounded log query over weeks can degrade the logging stack, and EXPLAIN ANALYZE executes the query, which is not read-only for anything but a SELECT.

Potentially disruptive

• Taking a heap dump or CPU profile of a production process — pauses the process, sometimes for seconds. On a service with tight timeouts this causes a visible blip. Do it on one instance, out of the load balancer rotation if possible.
• Raising log verbosity — a config change with real cost. Revert it explicitly.
• Reproducing load against production — you are adding load to a system you already suspect of saturation. Use a non-production environment, or a single instance removed from rotation.
• Restarting an instance to test a leak hypothesis — destroys the evidence you were about to collect. Capture the heap or fd list first.
• Toggling a feature flag to test a hypothesis — a real production change affecting real users. Announce it; it is an experiment, not an observation.

Destructive

Debugging should not require destructive actions. If an investigation appears to need one, that is a signal to stop and reconsider. Specifically:

• Killing queries or connections (pg_terminate_backend, KILL) — aborts in-flight work and rolls back transactions. Legitimate to relieve a saturated database; capture the query text first, because it is usually the evidence you need.
• Clearing a cache to test a hypothesis — the resulting cold-cache load has taken databases down. Required first: confirm the backend can serve the miss traffic.
• Deleting or replaying queue messages — destroys or duplicates work. Sample and store before touching them.
• Restoring a backup to compare state — restore to a separate instance. Never restore over production to test a theory.

Verification

A cause is verified when it survives these four checks:

1. Mechanism — you can explain every step from cause to symptom.
2. Timing — the cause preceded the symptom, and the interval makes sense.
3. Scope — everything affected is exposed to the cause, and everything unaffected is not. The second half is the test people skip and the one that catches wrong answers.
4. Prediction — the hypothesis implies something else you have not checked. Check it.

Before: p99 checkout latency 12s; database CPU 95%; pool wait 3.8s
After:  index created; p99 240ms; database CPU 22%; pool wait 0ms
Prediction tested: pods still on v2.4.1 never showed the symptom - confirmed.

Then verify the fix over a full traffic cycle. A fix validated at low traffic that has not met the daily peak is unverified.

Anti-Patterns

Bad: Start from the service that is alerting loudest. Better: That service is usually a victim. Work down the dependency path to the deepest one whose own work degraded.

Bad: The errors started around the same time as the deploy, so it was the deploy. Better: Check the direction and the gap, and check the scope — if pods still running the old version show the same symptom, the deploy is exonerated in one query.

Bad: Test hypotheses in order of how interesting they are. Better: Test in order of cost-to-eliminate. A boring hypothesis you can rule out in 30 seconds comes first.

Bad: Add capacity because the system is overloaded. Better: If it is a retry storm or a metastable failure, more capacity absorbs briefly and then fails at a higher level. Break the feedback loop first.

Bad: Restart the instance to see if the problem comes back. Better: That destroys the only state that explains a leak. Capture the heap, fd count and goroutine dump first; the restart is still available afterwards.

Bad: Stop when you find something that correlates. Better: Distributed systems produce many coincident anomalies during an incident. Without a mechanism you have found an anomaly, not a cause, and the fix will not hold.

Bad: Debug from memory of how the system works. Better: Check the current configuration. The system is frequently not what the person who built it remembers.

Composition

• incident-triage — if users are affected right now, mitigate before continuing here.
• observability-debugging — the mechanics of querying metrics, logs and traces.
• postmortem-analysis — once the cause is known.
• kubernetes-troubleshooting, postgresql-troubleshooting, linux-performance, dns-troubleshooting — whichever layer this investigation lands in.

References

• Google SRE Book: Addressing Cascading Failures
• Google SRE Book: Effective Troubleshooting