Debugging

Constraint

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION

If Phase 1 is incomplete, no fix proposals allowed. Period.

When to Activate

Any technical malfunction: test failures, runtime errors, unexpected behavior, performance regressions, build breaks, integration mismatches.

Activate especially when:

• Time pressure makes guessing tempting
• An "obvious quick fix" presents itself
• Previous attempts have not resolved the issue
• The failure mechanism is not fully understood

Investigation Protocol

Complete each phase sequentially. Do not skip ahead.

Phase 1: Root Cause Investigation

1. Read error output thoroughly

• Full stack traces, not just the top frame
• Note line numbers, file paths, error codes
• Warnings preceding the error often contain the answer

2. Reproduce consistently

• Determine exact trigger steps
• Confirm it repeats reliably
• If non-reproducible: gather more data — do not guess

3. Check recent changes

• git diff, git log --oneline -10
• New dependencies, config modifications, environment drift

4. Instrument multi-component boundaries

When the system spans multiple layers (CI pipeline, API chain, build tooling):

For EACH component boundary:
  - Log inputs entering the component
  - Log outputs leaving the component
  - Verify config/env propagation across the boundary
  - Run once, collect evidence, identify the failing seam

5. Trace data flow backward

See references/root-cause-tracing.md for the full technique.

Short version: start at the crash site, ask "where did this bad value come from?", trace upstream until you reach the originating mutation. Fix there.

Phase 2: Pattern Comparison

1. Locate analogous working code in the same codebase
2. If implementing a known pattern, read the reference implementation completely — no skimming
3. Diff working vs. broken — list every difference, however trivial
4. Map dependencies: config, environment, shared state, ordering assumptions

Phase 3: Hypothesis Testing

1. State one hypothesis explicitly: "X causes the failure because Y"
2. Make the smallest possible change to test it — one variable only
3. Evaluate:
   • Confirmed → proceed to Phase 4
   • Refuted → form a new hypothesis from the new evidence
   • Do not layer fixes on top of each other

Phase 4: Fix and Verify

1. Write a failing test that captures the bug (use arsyn:tdd if available)
2. Apply a single fix targeting the root cause — no bundled refactoring
3. Run the full relevant test suite — not just the new test
4. If the fix fails:
   • Count total attempts
   • Under 3: return to Phase 1 with new data
   • 3 or more: stop fixing. The problem is likely architectural.

On 3+ failed attempts: Indicators of structural mismatch:

• Each fix surfaces new coupling in a different location
• Fixes demand large-scale refactoring to implement
• Each patch spawns new symptoms elsewhere

Stop and reconsider fundamentals: Is the current approach viable, or is inertia the only reason to continue? Escalate for discussion before further changes.

File-System-as-State Pattern

For investigations spanning many files or long debugging sessions, offload findings to disk instead of relying on context memory:

# Initialize a debug log at the start of investigation
echo "# Debug Log — $(date -Iminutes)" > debug-log.md

# Append findings as you go
echo "## Phase 1 — Error Analysis" >> debug-log.md
echo "- Stack trace points to auth.ts:142" >> debug-log.md
echo "- Token is null at middleware boundary" >> debug-log.md

# Append hypotheses and results
echo "## Hypothesis 1: Token not forwarded by proxy" >> debug-log.md
echo "- Result: REFUTED — proxy logs show token present" >> debug-log.md

# Later, grep across your own findings instead of scrolling context
grep "REFUTED\|CONFIRMED" debug-log.md

Benefits:

• Survives context window limits in long sessions
• Grep-searchable — faster than re-reading conversation history
• Shareable artifact for post-incident review
• Prevents re-testing already-refuted hypotheses

Use this when: the investigation touches 5+ files, spans multiple tool calls, or you catch yourself re-reading earlier findings.

Stop Signals

If any of these thoughts arise, return to Phase 1 immediately:

• "Quick fix now, investigate later"
• "Let me try changing X and see"
• "Multiple changes at once, then run tests"
• "Skip the test, I'll verify manually"
• "It's probably X" (without evidence)
• "I don't fully understand but this might work"
• Proposing solutions before tracing data flow
• "One more attempt" after 2+ failures

Common Rationalizations

Quick Reference

Supporting References

• references/root-cause-tracing.md — backward tracing through call stacks
• references/defense-in-depth.md — multi-layer validation after root cause is found
• references/condition-based-waiting.md — replace arbitrary sleeps with condition polling

Gotchas

• Fix-first impulse — Modifying code before understanding the root cause. The constraint is absolute: investigate first.
• Assuming the obvious — "It's probably X" leads to symptom patches. Follow evidence chains, not intuition.
• Symptom suppression — Adding null guards around crashes instead of determining why the value is null.
• Skipping reproduction — Attempting fixes without a reliable trigger. No reproduction means no verification.
• Confirmation bias — Seeking only evidence that supports the first theory. Actively attempt to disprove each hypothesis.
• Scope drift — "While I'm here, let me also..." Stay on the reported defect. File separate issues for other findings.
• Context amnesia — In long sessions, forgetting earlier findings and re-testing refuted hypotheses. Use the file-system-as-state pattern to persist investigation state.
• Layered patches — Applying fix #2 on top of fix #1 without reverting. Each hypothesis test should start from a clean baseline.