Systematic Debugging

Contract

• Input: an observed failure, expected behavior, current state, and the evidence available from the repository or runtime.
• Output: a reproducible diagnosis, one evidence-backed root-cause hypothesis, the smallest safe correction, and verification evidence—or a bounded blocked report when the cause cannot be established.
• Owner: investigation and repair of an observed failure.
• Non-goals: general code review, speculative redesign, production intervention without authority, or stacking unrelated cleanup into a fix.
• Side effects: read-only investigation first; edits, commands with external effects, and commits follow the user's authority and repository rules.

Workflow

1. Confirm that an observed failure exists. If the request is a general review, planned feature, or quality pass without failing behavior, route it to code-review, quality-hardening, or the ordinary implementation workflow instead of inventing a reproduction or root cause.
2. Preserve the failure. Read the complete error, stack trace, warning, or failing assertion. Record the exact command, inputs, environment, and expected versus observed result.

3. Reproduce it with the smallest reliable case. If it is intermittent, narrow the conditions and gather more evidence before guessing. For timing or concurrency failures, wait on an observable condition with a bounded timeout instead of adding an arbitrary sleep or retry-until-green loop.

   For regressions and intermittent failures:

   • Regression bisection: For regressions after recent changes, inspect git commit history and diffs across the failing subsystem. Reproduce the test on the last known-working commit to confirm that the failure is a genuine regression rather than an environment issue.
   • git bisect (with a run script) is the preferred deterministic tool for isolating the regressing commit; use it instead of manual history scanning for non-trivial histories.
   • Intermittent / CI-only failures: Compare environment differences (OS, architecture, lockfiles, timezone, concurrency). Execute the reproduction under a stress loop (20–50 iterations) to establish an empirical baseline failure rate before testing fixes.

Repository-state safety during history debugging

Treat history traversal as temporary diagnostic state, not as the new working state.

Before git bisect, checking out an older revision, or otherwise moving HEAD:

1. Record the starting branch or commit and git status --short.
2. Do not begin history traversal in a dirty working tree if the operation could mix, hide, or overwrite those changes. Prefer an isolated worktree for non-trivial investigations.
3. Record any environment, fixture, dependency, or generated-state change needed to reproduce older revisions; a checkout alone does not prove the historical environment is equivalent.
4. Always leave the repository in the starting state when the diagnostic experiment ends. For git bisect, run git bisect reset even after an error, timeout, or successful identification of the culprit.
5. Before applying a fix, verify that HEAD, branch, and working-tree state are the intended implementation state rather than an intermediate diagnostic revision.

A correctly identified regressing commit does not justify leaving the caller's repository in detached HEAD, bisect mode, or a modified diagnostic state.

Historical-execution safety

Treat historical revisions as untrusted executable code. Before an automated bisect, inspect the fixed driver and the historical build or test entrypoint it invokes. Prefer a disposable worktree, container, or VM with no ambient credentials, a minimal environment, constrained write access, and network disabled when feasible. Keep the bisect driver outside the changing checkout. If historical revisions cannot be executed safely, use read-only history inspection or report the automated bisect as BLOCKED; worktree isolation alone is not process isolation.

4. Establish the change and data path. Inspect the relevant diff, recent changes, configuration, dependencies, and a known-working neighboring path. Trace the bad value or state backward to its first incorrect origin.
5. State one falsifiable hypothesis: “X is the root cause because Y.” Run the smallest diagnostic or test that can distinguish it from the alternatives. For performance regressions, keep commit, input data, environment, dependency state, and load comparable; include appropriate warm-up and repeated measurements. Choose a metric and distribution appropriate to the system and use the project's supported profiler when attribution is needed. A single timing sample is not sufficient evidence. Record measurement commands and relevant tool versions, distinguish signal from variance, and do not generate production load without explicit authority. Change one variable at a time and do not bundle multiple fixes into the experiment.
6. Once the cause is confirmed, add or update the smallest regression test or repeatable reproduction at the failing seam. Apply the minimal root-cause correction, then run the focused test, the original reproduction, and the repository's relevant complete gate.
7. If the hypothesis fails, record what the evidence ruled out and return to investigation. Continue only while the next experiment can add discriminating evidence. If repeated attempts leave the same uncertainty, stop and discuss whether the architecture, available evidence, or problem framing is wrong.

Boundaries and gotchas

• Prohibit the "null-check bandage": Never fix a crash by merely suppressing the symptom at the crash site (e.g., adding if obj is None: return, default fallbacks, or try/except: pass) unless the component contract explicitly dictates that null/empty input is valid at that layer. Always trace upstream to find why the invariant was violated at the data origin.
• Maintain diagnostic environment hygiene:
  • Run reproductions in an isolated scratch workspace or with clean test fixtures.
  • Cleanly remove all temporary debug prints, logging probes, and diagnostic mocks before finalizing the root-cause fix.
  • Verify that local caches, build artifacts (.pyc, build outputs), and environment variables are reset between reproduction attempts to prevent Heisenbugs.
• A passing health check, linter, retry, or timeout does not establish root cause; verify the failing behavior itself.
• Do not hide an unknown behind a catch-all fallback, retry loop, delay, or broad refactor.
• Do not print credentials, tokens, private data, or sensitive production payloads while gathering evidence. Redact examples and minimize scope.
• If the failure depends on an unavailable external system, document the attempted reproduction, the missing evidence, and the safest local check; do not claim the fix is verified.

Report and stop condition

Report:

1. observed behavior and exact reproduction;
2. evidence collected and the confirmed root cause;
3. correction and regression protection;
4. focused and complete verification results;
5. residual uncertainty, environment limits, or follow-up work.

Stop before changing code when the failure is not reproducible, the evidence supports multiple unresolved causes, the next action has external side effects without authority, or further attempts cannot add meaningful evidence.