Systematic Debugging

Do root-cause investigation before proposing fixes.

Iron Law

No fix without evidence for root cause.

This is non-negotiable. Every fix must trace back to a proven root cause. A fix applied without understanding why it works is not a fix — it is a mask.

Four Phases

Phase 0: Check Known Issues

• If known-issues.md exists at the project root, search it for the error message, error code, or failing test name.
• If a match is found, try the documented solution first. If it resolves the issue, stop — no further investigation needed.
• If no match or the documented solution doesn't work, proceed to Phase 1.

Phase 1: Investigate

• Read the full error output — not just the last line.

• Reproduce the bug reliably. If you cannot reproduce, you cannot fix. For tests that fail only in certain orderings (test pollution), run bash find-polluter.sh <marker> <test-pattern> from this skill's directory to identify which test is corrupting shared state. Read its exit code — they are not interchangeable:

  Exit	Meaning
  0	Every matched test ran and none created the marker. A real clean result.
  1	Polluter found; it is named in the output.
  2	The pattern matched no test files. Nothing ran, so nothing was proven — fix the pattern and re-run. Never report this as clean.
  3	The marker already existed before the run. Reset it first; no test can be blamed until you do.

• Check recent changes — what changed since it last worked?
  • If context-snapshot.json exists at the project root: read it. The changed_files and recent_commits fields answer this immediately without additional git commands.
  • Otherwise: run git log --oneline -10 and git diff HEAD~1..HEAD --name-only.
• Add instrumentation (logging, breakpoints) at component boundaries.
• Multi-component systems: When the system has multiple components (CI → build → signing, API → service → database), add diagnostic logging at EACH component boundary before proposing fixes:

  For EACH component boundary:
    - Log what data enters the component
    - Log what data exits the component
    - Verify environment/config propagation
    - Check state at each layer
  
  Run once to gather evidence showing WHERE it breaks
  THEN analyze evidence to identify the failing component
  THEN investigate that specific component

  Example (multi-layer system):

  # Layer 1: Workflow
  echo "=== Secrets available in workflow: ==="
  echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"
  
  # Layer 2: Build script
  echo "=== Env vars in build script: ==="
  env | grep IDENTITY || echo "IDENTITY not in environment"
  
  # Layer 3: Signing script
  echo "=== Keychain state: ==="
  security list-keychains
  security find-identity -v
  
  # Layer 4: Actual signing
  codesign --sign "$IDENTITY" --verbose=4 "$APP"

  This reveals which layer fails (e.g., secrets → workflow OK, workflow → build FAILS).
• Trace data/control flow backward from the error to its source.

Phase 2: Compare Patterns

• Find a known-good implementation in the repo that does something similar.
• Compare: behavior, config, assumptions, environment.
• List concrete differences — these are your investigation leads.

Phase 3: Hypothesize and Test

Self-Consistency Gate — Before committing to a hypothesis, apply multi-path reasoning (see self-consistency-reasoner):

1. Make 3-5 attempts at the root cause, each from a different starting point: trace forward from inputs, backward from the error, from recent changes, from similar past bugs. Each attempt names the one cause it arrives at — you are not brainstorming alternatives, you are checking whether different routes converge.
2. Group attempts that name the same cause, then take the majority. Report agreement as a count (e.g. 4/5):
   • Unanimous: test the majority hypothesis.
   • Majority (more than half, not all): test the majority hypothesis; name the minority one and test it next if the majority fails.
   • No majority (half or fewer): STOP. Do not pick a hypothesis. The bug is ambiguous or multi-causal. Gather more evidence (add logging, reproduce under different conditions) before choosing a direction.
3. These attempts share your context and your blind spots, so agreement is weak evidence and disagreement is strong evidence. The smallest test below — not the vote — is what proves the hypothesis.

Then for the selected hypothesis:

• State it clearly before testing it.
• Run the smallest possible test/change to validate or disprove it.
• If disproven, discard it entirely and form a new hypothesis.
• Never stack guesses. One hypothesis at a time, tested in isolation.

Phase 4: Fix and Verify

• Add a failing regression test that reproduces the bug first.
• Apply one focused fix — never bundle multiple changes.
• Re-run the tests that cover the fix and confirm the issue is resolved. Scope per verification-before-completion — a root cause that spanned subsystems earns the full suite; a one-module fix earns that module's suite.

Rationalization Table — Do Not Skip Investigation

Escalation Rule

If 3 fix attempts fail, stop. Do not attempt a fourth. Discuss architecture with the user before continuing. The bug may indicate a design flaw, not a code defect.

Red Flags

If you catch yourself doing any of these, restart from Phase 1:

• "Quick fix now, investigate later"
• Changing multiple things in one attempt
• Proposing a fix without reproduction evidence
• Claiming success without re-running verification
• "It should work now" without running the test
• Copy-pasting a fix from elsewhere without understanding why it works

Supporting References

• root-cause-tracing.md — trace bugs backward through the call stack
• defense-in-depth.md — validation at every data layer
• condition-based-waiting.md — replace timeouts with condition polling

Post-Fix: Update Knowledge Base

After resolving a bug:

1. Recurring error? (environment-dependent, configuration, platform-specific, external-state) → offer to add the error→solution mapping to known-issues.md using the format defined in error-recovery.
2. Permanent constraint discovered? (API limitation, platform behavior, library quirk that will never change) → offer to add it to project-map.md Critical Constraints section via context-management. These are facts every future session needs, not just error→fix mappings.

Related Skills

• self-consistency-reasoner — multi-path reasoning for root cause diagnosis
• test-driven-development — write the regression test before the fix
• verification-before-completion — prove the fix with fresh evidence
• error-recovery — consult and update project-specific known issues