Backend Debugging

Backend bugs are usually cheap to diagnose and expensive to find, because the evidence is scattered across a log stream, a database, a queue, a proxy, and a container runtime. The work is mostly locating the layer — after that, the code usually reads itself.

Isolate the layer before reading any code

A request crosses many boundaries. Find the first one where the answer is wrong, and stop searching the rest.

client → DNS/CDN → load balancer/proxy → app server → middleware chain → handler
       → service layer → ORM → database
                       → cache / queue / third-party API

Cheap, decisive probes:

• curl the endpoint directly, bypassing the client entirely: curl -i -sS -X POST … . Use -i to see status and headers, -w '\n%{http_code} %{time_total}s\n' for timing. If curl reproduces it, the frontend is innocent — stop looking there.
• Hit the app directly, bypassing the proxy (localhost/pod port). If the proxy is the difference, the bug is a header, a timeout, a body-size limit, or a rewrite rule — not your handler. Proxy-layer failures usually produce a 502/504 with an HTML body your handler never wrote.
• Run the query directly against the database with the exact parameters the ORM sent. Turn on query logging to get them; do not retype the query from the model code.
• Call the third-party API directly with the same payload. "Their API is down" and "we send a malformed body" look identical from inside the service.

Read the runtime, not just the code

• The logs, in full, around the failure — not just the error line. The lines immediately before it usually contain the cause; the lines after contain the consequences. Search by correlation/request ID if one exists; if none exists, adding one is often the fastest path to the fix.
• The exact status and body the server produced. A 500 with a stack trace, a 502 from the proxy, a 200 with {"error": …}, and a 401 from an expired token are four different investigations with the same user-visible symptom.
• Health of the process itself: memory (RSS vs the container limit), open file descriptors, thread/goroutine count, event-loop lag, GC time, and restart count. A container that keeps restarting (kubectl get pods restart count, docker ps uptime) is an OOM kill or a failing liveness probe, not an application logic bug.
• Database state: currently running queries and their wait state (pg_stat_activity / SHOW PROCESSLIST), locks and blocking chains, connection counts vs the pool limit and vs the server's max_connections, and table/index sizes.
• The queue: depth, oldest message age, in-flight count, dead-letter queue. A "job didn't run" bug is usually visible here in one glance.

The high-frequency bug classes

500 on a specific request

Get the real exception and full stack from the server logs. If it's caught and reformatted, find where and make it log the cause chain. Then determine what is special about this request: a null column on this record, a field the client omitted, a value that's a string when the code expects a number, an encoding or timezone edge, a size that exceeds a limit. Diff a working request against the failing one — the difference set is the suspect list.

Request hangs or times out

Something is waiting: a database query blocked on a lock, an outbound HTTP call with no timeout, an exhausted connection pool (every request waits for a connection that never frees), a deadlock between locks taken in different orders, or a blocking call on an event-loop thread. Distinguish them by looking at where the process is stuck — a thread/goroutine dump, event-loop lag, or the database's active-query list — rather than by reading the handler.

Wrong data written or read

Check the transaction boundary first: is the write inside a transaction, and does the transaction commit on every path? Then check isolation — a read-modify-write across two statements without a lock or an atomic update loses concurrent updates every time. Then check for a second writer (a job, a webhook, a retry, a trigger) that nobody was counting.

The write succeeded but the read doesn't see it

The POST returns 200 and the GET immediately after returns the old value. In rough order of frequency:

• Read-replica lag. The write went to the primary; the read was routed to a replica that is milliseconds — or, under load, seconds — behind. Measure the lag at the moment of failure (SELECT now() - pg_last_xact_replay_timestamp(), or Seconds_Behind_Source from SHOW REPLICA STATUS) and confirm which host the read connection actually used.
• A cache in front of the read that was never invalidated — or was invalidated before the commit rather than after, so the old value was immediately re-cached by a concurrent read.
• The read ran in a transaction that opened before the write committed. Snapshot isolation is then doing exactly what it promises; the bug is the transaction boundary.
• The write never actually committed — rolled back, swallowed, or issued inside a transaction that later aborted.

Distinguish them in one step: query the primary directly for the same key. Row present → it's routing or caching. Row absent → it's the write path, and the read is innocent.

The fix is routing (send read-your-writes traffic to the primary, or gate on a replication token) or correct invalidation. It is never a setTimeout, a sleep, or a retry loop — those appear to work at low lag and fail in exactly the conditions that produced the report.

Duplicate work, or a small failure that became an outage

• Inverted timeout hierarchy. The client gives up before the server finishes, retries, and the server now runs the work twice while the abandoned first attempt still holds a connection and its locks. Each layer's timeout must be shorter than the one in front of it. Write down the real numbers — client, proxy/LB, app server, pool checkout, statement timeout — and the inversion is usually obvious on sight.
• Retry storms. Retries without exponential backoff, jitter, and a budget convert a brief blip into sustained overload, and the struggling dependency never recovers because the load never drops. Watch for retries multiplying across layers: 3 in the client × 3 in the gateway × 3 in the service is 27 requests for one user action.
• A 502/504 with no matching entry in your application log is the proxy or load balancer, not your handler — nothing reached you. A recurring cause is an LB idle timeout longer than the upstream's keep-alive, so the LB sends a request down a socket the app has just closed.
• Anything retried without an idempotency key — payments, webhooks, job enqueues — produces duplicates rather than errors, so it is found by counting rows, not by reading logs.

Serverless and short-lived runtimes

The runtime's lifecycle breaks assumptions that hold in a long-running server:

• Connection pools multiply per instance. A pool of 10 is 1,000 connections at 100 concurrent instances, against a max_connections of 100. The symptom is connection errors that appear only under scale — use a proxy/pooler and a pool size of ~1.
• Work started after the response is frozen, not executed. Fire-and-forget logging, analytics, and cleanup either resume unpredictably on the next invocation or never run at all.
• Module-scope state survives on a warm instance, which makes it both an accidental cache and an accidental leak — and makes behavior depend on whether you got a warm or cold instance.
• Cold starts appear only in the tail. A p99 that looks like a code problem is often just initialization; compare cold and warm invocations separately before optimizing anything.

Slow endpoint

Measure the split before optimizing: time in the handler vs time in the database vs time in outbound calls. Then:

• N+1 queries — the query log shows the same statement hundreds of times with different parameters. The most common backend performance bug by a wide margin.
• A query with no usable index — EXPLAIN (ANALYZE, BUFFERS). Look for sequential scans on large tables, row-estimate errors of an order of magnitude, and sorts spilling to disk.
• Serial outbound calls that could be concurrent, or a call with no timeout.
• Work done per row that should be done per request — a config load, a permission lookup, a JSON parse inside a loop.

Memory growth / OOM

Distinguish a leak (grows without bound and never returns after load stops) from high watermark (large but stable) from fragmentation. Find it by heap snapshot comparison between two points under the same workload, or the language's profiler (pprof, tracemalloc, --inspect heap snapshots, JFR). Recurring causes: an unbounded in-process cache or Map with no eviction, listeners/subscribers added per request and never removed, closures capturing large objects, accumulating buffers on a stream that isn't consumed, and holding whole result sets in memory instead of streaming.

Race conditions and concurrency

Reproduce with concurrency, not with a single request — fire N simultaneous requests and check the invariant. Recurring causes: check-then-act (if not exists: create) without a unique constraint, read-modify-write without an atomic operation or a row lock, missing idempotency on a retried webhook or payment, non-atomic increment, and shared mutable state on a module-level variable that's fine single-threaded and wrong under a worker pool. The durable fix is usually a database constraint or an atomic operation, not a lock in application code.

Background jobs and queues

Check, in order: was it enqueued (log the enqueue), was it dequeued (log the dequeue with the same ID), did it throw (dead-letter queue / retry count), did it commit. A job that runs before its transaction commits will look up a row that doesn't exist yet — an extremely common and confusing failure. Also check for a worker that died and wasn't restarted, and a schedule that stopped firing.

Only broken during a deploy, restart, or scale event

Shutdown is where deploy-only bugs hide, and their signature — errors that spike at deploy time and then clear on their own — is exactly the signature that gets them dismissed as noise.

On SIGTERM: in-flight requests get clipped mid-response, keep-alive sockets linger, WebSocket clients are dropped without a close frame, and a worker that exits mid-job either loses the work or, once the queue's visibility timeout expires, lets a second worker run it again. A process that ignores SIGTERM altogether is SIGKILLed when the grace period ends, and no cleanup survives that.

Check, in order: does the process handle SIGTERM at all; does it stop accepting new work before draining; does it deregister from the load balancer before it stops listening (the reverse order produces a burst of 502s on every single deploy); is the termination grace period longer than the longest request and the longest job; and do the readiness and liveness probes mean different things (a readiness probe that fails during a long GC pause will have the orchestrator kill a perfectly healthy process).

Only broken in staging/production

This is an environment diff, not a code bug, until proven otherwise. Compare: environment variables and secrets, dependency versions actually installed (lockfile vs image), database version and extensions, migration state, data shape and scale, the proxy in front, TLS and certificates, timezone and locale, filesystem permissions and read-only mounts, resource limits, and the number of processes/replicas (code that is correct with one worker and wrong with four). Get both sides of the diff explicitly rather than assuming which one differs.

Two that are missed often enough to name: clock skew between hosts, which makes token expiry, signature validation, and TTL checks fail on one instance and pass on the next; and character set and collation (utf8 vs utf8mb4, case- and accent-insensitive collations), which changes what counts as a duplicate, what a unique index rejects, and how sorting orders — silently, and only for the rows that contain the affected characters.

Reproducing safely

• Prefer a local or staging reproduction. Never test a theory with a destructive command against production data. Never UPDATE/DELETE without a transaction you can roll back.
• Reproduce with the real data shape. Restore an anonymized copy or construct the exact record; most "unreproducible" bugs are data-shaped, not code-shaped.
• Replay the exact request captured from logs (curl with the same headers and body) rather than a request you compose from the API docs.
• Reproduce with concurrency and with load when the report says "sometimes" — for i in $(seq 20); do curl … & done, or a short k6/ab/hey run. Timing bugs are deterministic once you apply the right pressure.
• Write it as a failing test as soon as it reproduces. That is the artifact that makes the fix verifiable and keeps the bug from returning.
• Redact before pasting. Credentials, tokens, customer PII, and full production payloads do not belong in a transcript — refer to them by location and shape.

Instrumenting a live system

When you cannot reproduce locally and must observe the real thing, prefer the least invasive tool that answers the question:

1. Existing structured logs and traces, queried by correlation ID.
2. Metrics and dashboards — rate, error, duration, saturation. The shape of the graph at the moment of failure narrows the cause fast (a step change means a deploy or a config change; a ramp means a leak or growth; a sawtooth means restarts).
3. A temporary, targeted log line behind a flag or sampled by request ID — deployed, read, and then removed.
4. A profiler or dump captured on demand (pprof endpoint, heap dump, thread dump), taken during the bad state and compared against one taken during the good state.
5. System-level tools as a last resort — strace/dtruss, lsof, ss/netstat, tcpdump — for the cases where the process claims to be doing something the OS disagrees with.

Always compare against a healthy baseline. A number with nothing to compare it to explains nothing.