Debugging

Overview

Find the root cause. Prove it. Then fix it.

Core principle: If you can't explain why the bug happens, you don't know if your fix is correct.

Violating the letter of the rules is violating the spirit of the rules.

Pre-flight: Model Recommendation

Your current effort level is {{effort_level}}.

Skip this step silently if effort is high, xhigh, or max (the scale is low < medium < high < xhigh < max, so xhigh and max are already above high) AND you are Opus (1M context). If effort is low or medium (i.e. below high), you MUST show the recommendation prompt — regardless of model. If you are not Opus (1M context), you MUST show the recommendation prompt - regardless of effort level.

Otherwise → use AskUserQuestion:

{
  "questions": [{
    "question": "Do you want to switch? Falsifiable hypothesis formation and resisting the temptation to jump to fixes benefit from deeper reasoning.\n\nTo switch: cancel, run `/model opus[1m]` and `/effort high`, then re-invoke this skill.",
    "header": "Recommended: Opus (1M context) at high effort",
    "options": [
      { "label": "Continue" },
      { "label": "Cancel — I'll switch first" }
    ],
    "multiSelect": false
  }]
}

If the user selects "Cancel — I'll switch first": output the switching commands above and stop. Do not proceed with the skill.

When to Use

Always:

• Test failures
• Unexpected behavior
• Error messages
• Regressions
• "It works on my machine" situations

Exceptions (ask your human partner):

• Typos and obvious one-character fixes
• Configuration value changes with known correct values

Thinking "I already know the fix"? Stop. That's guessing, not debugging.

The Iron Law

NO CODE CHANGES UNTIL YOU CAN EXPLAIN THE ROOT CAUSE

Think you know the fix? Prove the root cause first. Then fix.

No exceptions:

• Don't "try a quick fix to see if it works"
• Don't change multiple things hoping one helps
• Don't fix symptoms instead of causes
• Don't skip reproduction because "it's obvious"

Understand first. Fix second. Period.

Five-Phase Workflow

digraph debugging {
    rankdir=LR;
    understand [label="UNDERSTAND\nRead the system", shape=box, style=filled, fillcolor="#cce5ff"];
    reproduce [label="REPRODUCE\nMake it fail", shape=box, style=filled, fillcolor="#ffcccc"];
    isolate [label="ISOLATE\nDivide and\nconquer", shape=box, style=filled, fillcolor="#fff3cc"];
    fix [label="FIX\nMinimal change\n+ failing test", shape=box, style=filled, fillcolor="#ccffcc"];
    verify [label="VERIFY\nProve it's fixed", shape=box, style=filled, fillcolor="#ccccff"];
    escalate [label="ESCALATE\nReport to human", shape=box, style=filled, fillcolor="#ffccee"];
    journal [label="DEBUG JOURNAL\n{{debug_dir}}/{slug}-xxxxxx/journal.md", shape=note, style=filled, fillcolor="#f0f0f0"];

    understand -> reproduce;
    reproduce -> isolate;
    isolate -> fix;
    fix -> verify;
    verify -> fix [label="fix failed\n(max 3)"];
    verify -> escalate [label="3 fixes\nfailed"];

    understand -> journal [style=dashed];
    reproduce -> journal [style=dashed];
    isolate -> journal [style=dashed];
    fix -> journal [style=dashed];
    verify -> journal [style=dashed];
}

Phase 1: UNDERSTAND

Read the system. Check your assumptions. Don't touch anything yet.

Before investigating the bug, understand the code it lives in. Your mental model is a hypothesis — treat it as one.

Required steps:

1. Read the relevant source files end-to-end (not just the failing line)
2. Trace the code path from input to failure
3. Identify what the code should do vs. what it does
4. Check the obvious: dependencies installed? correct branch? config valid? environment correct?
5. Read error messages completely — stack traces, line numbers, all of it
6. Use git to fetch recent changes
7. For multi-component systems (API → service → database, CI → build → deploy), instrument at each boundary before forming hypotheses. Run once, collect evidence from every layer, then analyze where the chain breaks.

Example — multi-layer instrumentation:

System: Frontend → API Gateway → Auth Service → Database
Step 1: Add logging at each boundary (request in, response out)
Step 2: Trigger the failure once
Step 3: Read the logs layer by layer:
  - Frontend sent correct request ✓
  - API Gateway forwarded correctly ✓
  - Auth Service received request ✓
  - Auth Service → Database query returned empty result ✗
  - Auth Service returned 500 to Gateway
Conclusion: Problem is between Auth Service and Database.
Now investigate that specific boundary.

Don't guess which layer is broken. Let the evidence from all layers tell you.

Check the plug:

• Is the service running?
• Is the database connected?
• Are environment variables set?
• Is the right version deployed?
• Did a dependency update break something?

Don't assume the obvious is fine. Verify it.

Phase 2: REPRODUCE

Make it fail. Consistently. On demand.

If you can't reproduce it, you can't prove you fixed it. Observation before theory — watch what actually happens, don't theorize about what might happen.

Required steps:

1. Create a minimal reproduction (smallest input that triggers the bug)
2. Run it. Watch it fail. Note the exact output.
3. Run it again. Confirm it fails the same way.
4. Add instrumentation (logging, breakpoints) if the failure path is unclear
5. Document the reproduction steps in the debug journal

Reproduction checklist:

• Can trigger the bug on demand
• Same failure every time (or documented intermittent pattern)
• Minimal reproduction — removed everything not needed to trigger it
• Exact error output recorded

Phase 2 Gate — MANDATORY

Before proceeding to Phase 3, your debug journal MUST contain:

• The exact command/steps you executed (not theorized about)
• The exact error output you observed (not inferred from code)
• Confirmation this matches the user-reported symptoms

Static code reading is NOT reproduction. If you have only read source files and formed theories about what might happen, you are still in Phase 1.

Simulation is NOT reproduction. Writing scripts that mimic what the code would do (e.g., Node.js scripts simulating CoffeeScript logic, Python scripts simulating import resolution) is theorizing, not reproducing. Run the actual code path instead.

Can't reproduce?

• Check environment differences
• Check timing/race conditions
• Check data-dependent paths
• Add logging and wait for it to recur
• Don't guess. If you can't reproduce it, you can't fix it.

Can't execute directly? (GUI app, external service, no local environment)

1. Add temporary logging/instrumentation to the actual code FIRST
2. Ask the user to trigger the bug with your instrumentation in place
3. Read the instrumented output — this is your reproduction evidence
4. Do NOT proceed to Phase 3 based on code reading alone
5. If instrumentation isn't possible either, document WHY in the journal and escalate to your human partner for reproduction assistance

Phase 3: ISOLATE

Narrow the search space. One variable at a time.

Form hypotheses. Make them falsifiable. Test them. Eliminate possibilities systematically — don't jump to conclusions.

Required steps:

1. Form a specific, falsifiable hypothesis: "The bug occurs because X. If I do Y, I should see Z."
2. Test the hypothesis. One change at a time.
3. Record the result: confirmed or eliminated.
4. Repeat until the root cause is isolated.

Investigation Techniques

Spawning Investigation Subagents

When dispatching subagents for parallel investigation:

1. Pass context forward — Include files already read and findings so far in the subagent prompt, not just "investigate X"
2. Specify concrete actions — "Run pip install --target $(mktemp -d) and return the error output" not "explore the dependency setup"
3. Require reproduction — Subagents must execute the failing operation, not just read code
4. Reference the debug journal — Pass the literal resolved journal path (not the {slug} pattern) and tell the subagent to read it first so it doesn't repeat prior work
5. Validate findings — If a subagent returns without direct evidence (only file reads, no command output), run the reproduction yourself before accepting its conclusions

One thing at a time:

• Change one variable per test
• Revert changes that didn't help
• Don't combine hypotheses ("maybe it's A and B together")
• If you changed two things and it worked, you don't know which one fixed it

Phase 4: FIX

State the root cause. Write a failing test. Make the minimal fix.

You've isolated the root cause. Now prove it with a test, then fix it.

Required steps:

1. State the root cause — Write it in the debug journal with a verification level (see Verification Levels below)
2. Write a failing test — Invoke Skill(skill="groundwork:test-driven-development") and follow the TDD cycle. The test must reproduce the exact bug.
3. Minimal fix only — Change the least amount of code to fix the root cause. Don't refactor. Don't "improve" nearby code. Don't fix other bugs you noticed.
4. One change at a time — If the fix requires multiple changes, make them incrementally.

Fix: Parse ID to integer at the controller layer before passing to the service.

Root cause stated, explains the symptom chain, minimal targeted fix
</Good>

<Bad>

Fixed the timeout by:

• Increasing pool size to 200
• Adding parseInt to 5 different endpoints
• Refactoring the connection manager
• Adding retry logic to all database calls

Shotgun fix, no root cause stated, changes way beyond the bug
</Bad>

## Phase 5: VERIFY

Prove the fix works. Prove nothing else broke. Get a fresh perspective.

**Required steps:**
1. Run the failing test — it must pass now
2. Run the full test suite — nothing else broke
3. Re-run the original reproduction — the bug is gone
4. Review your fix with fresh eyes — does it actually address the root cause, or mask it?
5. Check for related code paths — could the same bug exist elsewhere?

**Verification checklist:**
- [ ] Failing test now passes
- [ ] Full test suite passes
- [ ] Original reproduction no longer triggers the bug
- [ ] Fix addresses root cause (not symptoms)
- [ ] No unrelated changes included
- [ ] Related code paths checked for same bug

**Fix didn't work?** Go back to Phase 3. Your root cause analysis was wrong. Don't patch the patch — re-investigate.

## Escalation Rules

**3-fix limit.** If your third fix attempt fails, stop and escalate.

Three failed fixes means your root cause analysis is wrong. More attempts won't help — you need a different perspective.

**Question the architecture.** Three failed fixes often signal something deeper than an incomplete investigation. The bug may exist because the design is fundamentally unsound — a race condition baked into the concurrency model, a data integrity problem caused by a missing constraint, a failure mode that the architecture doesn't account for. When you escalate, shift the question from "help me fix this bug" to "is this design sound?"

**Escalation output format:**

```markdown
## Escalation: [Bug Description]

### What I tried

| # | Hypothesis | Fix Attempted | Result |
|---|-----------|---------------|--------|
| 1 | [hypothesis] | [what you changed] | [what happened] |
| 2 | [hypothesis] | [what you changed] | [what happened] |
| 3 | [hypothesis] | [what you changed] | [what happened] |

### Eliminated hypotheses
- [hypothesis]: eliminated because [evidence]

### What I know
- [confirmed fact with evidence]

### Could this be architectural?
[If 3 fixes failed, consider whether the bug reflects a design problem, not just a code problem. What assumption does the current architecture make that might be wrong?]

### Recommended next step
[specific suggestion for the human partner]

Revert all failed fix attempts. Leave the codebase clean.

Debug Journal

Maintain a persistent debug journal at an invocation-unique path — the slug plus this run's identity — never at a fixed slug-only name. Concurrent debugging of the same bug in the same project would silently overwrite a fixed journal; a unique path gives every invocation its own intact record, and prior work is found by slug prefix when resuming.

{{debug_dir}} resolves inside the selected project's root (mirroring {{plans_dir}}), so the same bug slug in two monorepo projects produces two distinct journal locations instead of colliding at the repository root.

Derive the slug once, at the start of Phase 1: kebab-case of the bug's one-line description, truncated to 40 characters (e.g. "login timeout on token refresh" → login-timeout-on-token-refresh). Then allocate the journal's run directory once — an atomically unique mktemp directory keyed on the slug:

mkdir -p {{debug_dir}}
grep -qxF '.debug/' .gitignore 2>/dev/null || printf '.debug/\n' >> .gitignore
RUN_DIR="$(mktemp -d "{{debug_dir}}/${SLUG}-XXXXXX")"
JOURNAL_PATH="$RUN_DIR/journal.md"

The .gitignore pattern is unanchored, so it covers debug directories at any project depth; the append is idempotent.

Use that literal resolved path everywhere for the rest of the session — including in every subagent prompt — so the path can never drift between derivations, and return it in your final output and any handoff so the next session can adopt it directly.

Resuming prior work on the same slug: find earlier run directories by slug prefix — ls -dt {{debug_dir}}/${SLUG}-*/ — and read the journal inside. If more than one match exists, do not silently pick one: use the literal journal path from your handoff, or ask the user which run to continue. Then continue writing to your own invocation-unique journal (never a shared current pointer, never the older file).

One-time migration: if a legacy journal for this slug exists at the unscoped repository-root location .debug/{slug}.md while {{debug_dir}} points elsewhere, move it into a fresh run directory with two visible commands and say so — do not keep reading the legacy location:

RUN_DIR="$(mktemp -d "{{debug_dir}}/{slug}-migrated-XXXXXX")"
mv .debug/{slug}.md "$RUN_DIR/journal.md" # legacy location migration

# Debug: {slug}

## Status
[Active | Root Cause Found | Fixed | Escalated]

## Symptoms
- [What the user or test reported]
- [Exact error messages]
- [When it started / what changed]

## Reproduction
### Command Executed
[The exact command you ran — NOT theoretical steps, NOT code you read]

### Actual Output
[Paste the exact error/output you observed when you ran it]

### Expected Output
[What should have happened instead]

### Matches User Report?
[Yes/No — does this reproduce the same symptoms the user described?]

## Hypotheses

### Active
- [ ] [Hypothesis]: [How to test it]

### Eliminated
- [x] [Hypothesis]: Eliminated because [evidence]

## Evidence Log
| # | Action | Observation | Conclusion |
|---|--------|-------------|------------|
| 1 | [what you did] | [what happened] | [what it means] |

## Root Cause
**Verification level:** [Verified | Corroborated | Unverified]

[Root cause statement]

[Evidence that confirms it]

## Fix
[What was changed and why]

## Resolution
[Final status, test references, PR link]

Update the journal as you work. Every hypothesis, every test, every result goes in the journal. When context compacts, the journal remains.

Verification Levels

Root cause claims require a confidence level. This prevents false confidence and forces honest assessment.

Rule: Never fix based on an unverified root cause. If your confidence is "unverified," you're guessing. Go back to Phase 3.

Common Rationalizations

Signals From Your Human Partner

During interactive debugging, your human partner's reactions are a signal channel. Frustration and redirection mean your process is off — not that the human is being difficult.

Rule: Treat pushback as data, not criticism. If your human partner redirects you, the most productive response is to follow the redirect — not to defend your hypothesis.

Red Flags — STOP and Go Back to Phase 1

• Changing code without stating the root cause
• Multiple changes at once ("let me try a few things")
• Fix attempt #4+
• "I think it might be..." without a falsifiable test
• Increasing timeouts, pool sizes, or retry counts as a "fix"
• Adding error suppression (empty catch blocks, silent fallbacks)
• Can't explain why the fix works
• Fix is larger than the bug warrants
• Fixing a different bug than the one you started with
• No reproduction case
• No failing test
• Skipping the debug journal
• Forming hypotheses based only on code reading without running anything
• Writing simulation scripts instead of running the actual failing code
• Spawning investigation subagents without specifying what to execute

All of these mean: Stop. Go back to Phase 1. Understand the system.

When Stuck

When Investigation Finds No Root Cause

Sometimes systematic investigation legitimately concludes without a clear root cause — environmental issues, timing-dependent failures, external service flakiness, or bugs that disappear under observation. This is rare, but it happens.

When you genuinely cannot identify the root cause:

1. Document what was investigated and eliminated. Every hypothesis tested, every result observed. This prevents the next investigator from repeating your work.
2. Implement appropriate handling. Retry logic, timeouts, clearer error messages, or monitoring — not as a "fix," but as resilience for a known-unknown failure mode. Be explicit that this is mitigation, not a root cause fix.
3. Add instrumentation for recurrence. Logging, metrics, or alerts that will capture more data when the failure happens again. Design the instrumentation to answer the specific questions you couldn't answer this time.
4. Record in the debug journal with status "Inconclusive." Include what you know, what you don't, and what data would resolve it.
5. Communicate honestly. Tell your human partner what you investigated, what you eliminated, and that you couldn't pin it down — not a vague "it should be fine now."

Caveat: 95% of "no root cause found" conclusions are actually incomplete investigations. Before declaring inconclusive, honestly assess:

• Did you complete all five phases?
• Did you actually reproduce the failure, or just read the code?
• Did you test falsifiable hypotheses, or just form opinions?
• Is the journal thorough, or are there gaps?

If there are gaps, go back and fill them. "Inconclusive" is a legitimate outcome — but only after exhausting the process, not instead of it.

Example: Full Walkthrough

Bug report: "API returns 500 on user profile page for some users"

Phase 1: UNDERSTAND

Read the profile endpoint handler, the user service, and the database query. The endpoint calls getUserProfile(id) which joins users and preferences tables.

Check the plug: database connected, service running, other endpoints working fine.

Phase 2: REPRODUCE

curl -s localhost:3000/api/profile/42   # 200 OK
curl -s localhost:3000/api/profile/107  # 500 Internal Server Error

Consistent. User 42 works, user 107 fails. Minimal reproduction found.

Error log: TypeError: Cannot read property 'theme' of null

Phase 3: ISOLATE

Hypothesis 1: User 107 has a null preferences record. Test: Query the database directly.

SELECT * FROM preferences WHERE user_id = 107;
-- Empty result set

Result: Confirmed — user 107 has no preferences row.

Hypothesis 2: The JOIN returns null for preferences, and the code doesn't handle it. Test: Read getUserProfile — it does user.preferences.theme with no null check. Result: Confirmed. The LEFT JOIN returns null preferences, and the code assumes it exists.

Root cause: getUserProfile accesses preferences.theme without null-checking the join result. Users without a preferences row trigger a TypeError.

Verification level: Verified (direct evidence: null row + no null check + exact error match).

Phase 4: FIX

Failing test:

test('returns default theme when user has no preferences', async () => {
  const user = await createUser({ id: 999 }); // no preferences row
  const profile = await getUserProfile(999);
  expect(profile.theme).toBe('default');
});

Test fails with TypeError: Cannot read property 'theme' of null. Correct failure.

Fix:

const theme = user.preferences?.theme ?? 'default';

Test passes.

Phase 5: VERIFY

• Failing test passes: yes
• Full test suite: all green
• curl localhost:3000/api/profile/107: 200 OK, theme is "default"
• Fix addresses root cause: null preferences handled
• Checked other code paths: preferences.language has same bug — filed separate issue

Done.

Final Rule

Root cause identified → Fix with failing test → Verify fix
Otherwise → not debugging, just guessing

No exceptions without your human partner's permission.