Systematic Debugging

Overview

Random fixes waste time, create new bugs. Quick patches mask real issue.

Core principle: ALWAYS find root cause before fixes. Symptom fixes = failure.

Break letter of process = break spirit of debugging.

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

Phase 1 not done? Cannot propose fixes.

When to Use

ANY technical issue: test failures, production bugs, unexpected behavior, performance problems, build failures, integration issues.

ESPECIALLY when:

• Under time pressure (emergencies make guessing tempting)
• "Just one quick fix" seems obvious
• Already tried multiple fixes
• Previous fix didn't work
• Don't fully understand the issue

Don't skip when:

• Issue seems simple (simple bugs have root causes too)
• In a hurry (rushing guarantees rework)
• Manager wants it NOW (systematic beats thrashing)

The Four Phases

You MUST complete each phase before the next.

Phase 1: Root Cause Investigation

BEFORE attempting ANY fix:

1. Read Error Messages Carefully
   • Don't skip errors or warnings — often contain the exact solution
   • Read stack traces completely; note line numbers, file paths, error codes
2. Reproduce Consistently
   • Trigger it reliably? Exact steps?
   • Not reproducible → gather more data, don't guess
3. Check Recent Changes
   • What changed that could cause this? (git diff, recent commits, new deps, config)

4. Gather Evidence in Multi-Component Systems

   WHEN system has multiple components (CI → build → signing, API → service → database), BEFORE proposing fixes, add diagnostic instrumentation at each boundary:

   For EACH component boundary:
     - Log what data enters the component
     - Log what data exits the component
     - Verify environment/config propagation
   Run once to gather evidence showing WHERE it breaks
   THEN investigate that specific component

   Example (multi-layer system):

   echo "=== Secrets available in workflow ==="
   if [ -n "${IDENTITY:-}" ]; then echo "IDENTITY: SET"; else echo "IDENTITY: UNSET"; fi
   echo "=== Env vars in build script ==="
   printenv IDENTITY >/dev/null 2>&1 && echo "IDENTITY in environment" || echo "IDENTITY not in environment"
   echo "=== Keychain state ==="
   security find-identity -v

   This reveals which layer fails (secrets → workflow ✓, workflow → build ✗).

5. Trace Data Flow

   WHEN error deep in call stack: see Root Cause Tracing reference (auto-included below) for complete backward-tracing technique.

   Quick version: Where does bad value originate? What called this with bad value? Trace up until you find source. Fix at source, not symptom.

Phase 2: Pattern Analysis

1. Find Working Examples — locate similar working code in same codebase
2. Compare Against References — implementing a pattern? Read reference implementation COMPLETELY (every line, no skimming)
3. Identify Differences — list every difference between working and broken, however small. Don't assume "that can't matter"
4. Understand Dependencies — what components, settings, config, environment, assumptions does this need?

Phase 3: Hypothesis and Testing

1. Form Single Hypothesis — "I think X is root cause because Y." Write it down. Be specific.
2. Test Minimally — SMALLEST possible change to test it. One variable at a time.
3. Verify Before Continuing — Worked? → Phase 4. Didn't? → form NEW hypothesis. DON'T pile fixes on top.
4. When You Don't Know — say "I don't understand X." Don't pretend. Ask for help. Research more.

Phase 4: Implementation

Fix root cause, not symptom:

1. Create Failing Test Case — simplest reproduction, automated if possible. MUST have before fixing. Use superpowers-test-driven-development skill.
2. Implement Single Fix — address root cause. ONE change. No "while I'm here" improvements, no bundled refactoring.
3. Verify Fix — test passes now? No other tests broken? Issue actually resolved? (Use superpowers-verification-before-completion skill.)
4. If Fix Doesn't Work — STOP. Count fixes tried. < 3: return to Phase 1 with new info. ≥ 3: STOP and question architecture (step 5). DON'T attempt fix #4 without architectural discussion.
5. If 3+ Fixes Failed: Question Architecture
   • Pattern: each fix reveals new coupling elsewhere; fixes require "massive refactoring"; each fix creates new symptoms.
   • STOP. Question fundamentals: Is this pattern sound? Continuing through inertia? Refactor vs. keep patching symptoms?
   • Discuss with your human partner before more fixes. NOT a failed hypothesis — a wrong architecture.

Red Flags - STOP and Follow Process

Catch yourself thinking:

• "Quick fix for now, investigate later"
• "Just try changing X and see if it works"
• "Add multiple changes, run tests"
• "Skip the test, I'll manually verify"
• "It's probably X, let me fix that"
• "I don't fully understand but this might work"
• "Here are the main problems: [lists fixes without investigation]"
• Proposing solutions before tracing data flow
• "One more fix attempt" (when already tried 2+)
• Each fix reveals a new problem in a different place

ALL of these mean: STOP. Return to Phase 1. 3+ fixes failed: question architecture (Phase 4.5).

Signals You're Doing It Wrong

Watch for these redirections from your human partner:

• "Is that not happening?" — you assumed without verifying
• "Will it show us...?" — you should have added evidence gathering
• "Stop guessing" — you're proposing fixes without understanding
• "We're stuck?" (frustrated) — your approach isn't working

See these: STOP. Return to Phase 1.

Common Rationalizations

Quick Reference

When Process Reveals "No Root Cause"

Systematic investigation shows issue is truly environmental, timing-dependent, or external? Process complete. Document what you investigated, implement appropriate handling (retry, timeout, error message), add monitoring/logging.

But: 95% of "no root cause" cases = incomplete investigation.

Supporting Techniques (auto-included references)

• Root Cause Tracing — trace bugs backward through call stack to original trigger
• Defense in Depth — add validation at multiple layers after finding root cause
• Condition-Based Waiting — replace arbitrary timeouts with condition polling

Related skills: superpowers-test-driven-development (Phase 4 failing test) · superpowers-verification-before-completion (confirm the fix).