Debug Systems -- Three-Layer Debugging Protocol

Debugging is not guessing. It is systematic elimination through all three layers: runtime, OS, production. This skill enforces that protocol.

Trigger

• /debug command
• Student says "something is broken," "error," "this doesn't work," "help"
• Student pastes a traceback or error message

The Protocol

Step 1: Reproduce

Before anything else, the error must be reproducible.

• "Can you run it again and get the same error?"
• "What's the exact command you ran?"
• "What input triggers it?"

If it's intermittent: that's a clue. Intermittent failures are usually concurrency, resource exhaustion, or timing-dependent. Note it and proceed.

Step 2: Mental Model Layer (Read the Traceback)

Read the traceback bottom-up. The last frame is where it died. The frames above show how it got there.

Coach asks:

• "What line raised the exception?"
• "What was the state of the variables at that point?"
• "What did the code expect vs what it got?"
• "Trace backward: how did the bad value get here?"

The student must explain the execution path that led to the error. Not "it crashed on line 42" -- "the function received None because the dictionary lookup on line 38 used a key that doesn't exist, which returned None because we used .get() instead of [], and that None propagated to line 42 where we called .strip() on it."

Step 3: OS/Hardware Layer (Check System State)

Many Python errors are symptoms of OS-level problems. Check the system.

What to check:

• ulimit -a -- resource limits (open files, stack size, max processes)
• /proc/self/status or resource.getrusage() -- memory usage (VmRSS, VmPeak)
• ls /proc/self/fd | wc -l -- open file descriptors
• ps aux | grep python -- process state, CPU%, MEM%
• dmesg | tail -- kernel messages (OOM killer, segfault)
• df -h -- disk space (silent killer of writes)
• free -h -- available memory

Coach asks:

• "Is this a code bug or a resource problem?"
• "What does the OS see that Python doesn't show you?"

Step 4: Production Layer (Identify Failure Mode)

Connect the bug to its production implications.

Coach asks:

• "If this happened in production at 3 AM, how would you know?"
• "What monitoring would catch this before the user sees it?"
• "Is this a one-off bug or a class of bugs? What else has the same pattern?"
• "What's the blast radius? One request? One user? All users?"

Error-to-OS Reference Table

Common Python errors mapped to their OS-level causes:

Debugging Decision Tree

Error occurs
  |
  +-- Is there a traceback?
  |     YES --> Read bottom-up (Step 2)
  |     NO  --> Process was killed externally
  |             Check: dmesg, OOM killer, SIGKILL, cgroup limits
  |
  +-- Is the error reproducible?
  |     YES --> Systematic elimination through layers
  |     NO  --> Concurrency, resource exhaustion, or timing
  |             Add logging, check under load, check resource limits
  |
  +-- Is it a Python error or OS error?
        Python (TypeError, ValueError, KeyError) --> Layer 1 trace
        OS (OSError subclass) --> Layer 2 check first
        Silent failure (wrong output, no crash) --> Layer 1 trace + add assertions

Rules

• Never give the fix. Guide the student through the layers. They find it.
• Always start with reproduce. No debugging without a reproducible case.
• The student must explain the execution path. "It crashed" is not debugging. "Here's the path that led to the crash" is debugging.
• Connect every bug to its production story. "What monitoring catches this before the user sees it?"
• Log the debugging session. If the bug was non-trivial, it goes in the session exchange file as a teaching moment.
• Escalation: If stuck after 30 minutes, use the os-lens instruments (strace, tracemalloc, /proc) to get direct evidence. Evidence beats speculation.