log-driven-diagnosis

Purpose

When you cannot step through code, logs are your only visibility. Be methodical about extracting signal from noise — never dump whole log files into context.

When to Use

• Runtime failures where a debugger cannot be attached
• Distributed systems where the failure spans services
• Issues reproducible only from log evidence

Inputs

• Log files or a log explorer covering the incident window
• The incident timestamp and, if available, a request/trace ID

Workflow

1. Time-bound the search. Never dump the whole log file — always grep for timestamps around the reported incident, or use tail:

   tail -n 200 app.log                          # most recent context
   grep -n "2026-06-06T14:0" app.log            # window around the incident
   awk '/14:02:00/,/14:05:00/' app.log          # bounded slice between timestamps

2. Identify the request ID. If the system uses distributed tracing or request IDs, find the ID associated with the error, then search the log corpus for only that ID to trace the complete lifecycle of the failed request:

   grep -n "ERROR" app.log | head -5            # find the failing entry and its ID
   grep -n "<request-id>" app.log               # full lifecycle of that request
   # structured (JSON) logs:
   jq -c 'select(.request_id == "<request-id>")' app.log.json

3. Look for preceding warnings. The ERROR log is usually just the final crash. The actual root cause is often a WARNING or unexpected INFO log that occurred milliseconds earlier (e.g., a connection retry failing, or an empty array being returned):

   grep -n -B 20 "<error-text>" app.log | grep -inE "warn|retry|timeout|empty"

4. Add missing logs. If the logs do not provide enough visibility, your first action must be to add temporary logging to the application, reproduce the bug, and gather the new signals. Do not guess blindly if the logs are insufficient.

Output

• The traced lifecycle of the failing request
• The suspected root-cause log line(s), including any preceding warnings
• Any temporary logging added (flagged for later removal)

Verification

• Every search was time-bounded or ID-bounded — no full-file dumps in context
• Full request lifecycle traced when an ID exists
• Lines preceding the error inspected, not just the error line itself
• Temporary logging added (and flagged for removal) where visibility was missing

Failure Modes

• Error-line tunnel vision — reading only the final ERROR while the cause sits in an earlier WARNING.
• Context flooding — dumping megabytes of log into the conversation instead of bounded slices.
• Blind guessing — iterating on fixes when the honest move is adding logging and reproducing once more.