sleuth: find the cause, not a place to type

The failure mode this skill exists to kill: change something, run, works, move on: bug resurfaces in a month. You ship a cause chain (symptom → mechanism → root cause) with evidence at every link, then the smallest fix that breaks the chain.

Purpose, authority, and safety

This is a scientific debugging skill: it converts a report into falsifiable claims and observations. It is not permission to experiment recklessly in production. Preserve evidence before changing state, distinguish observation from intervention, and state the confidence of every conclusion.

• In an active incident, stabilize users and data first. Coordinate with the incident owner; do not run an invasive diagnosis that worsens impact.
• Ask before changing production configuration, replaying customer traffic, enabling expensive logging, querying sensitive records, triggering jobs, or rolling back a release.
• Redact secrets and personal data from logs, commands, reproduction fixtures, commits, and reports. Use smallest safe samples.
• Freeze only the test inputs and environment needed for a reproduction. Do not alter unrelated working-tree changes or assume they caused the failure.
• A symptom with a strong correlation is not a root cause. Record the competing explanation or the observation that excluded it.

Prerequisites

Access to the code plus one runnable path to the symptom (command, endpoint, test, or user steps) and the runtime's evidence sources (logs, store, traces). No runnable path → say what reproduction needs (step 1 covers the honest exit); debugging a description without the system is theorizing, not sleuthing.

Tool selection/fallback

• Live process + debugger/logs → reproduce and observe directly; logs at boundaries, actual store state, git bisect for regressions.
• No debugger but runnable → prints at boundaries + deterministic replay (freeze time, seed RNG, pin data); printf beats guessing.
• Nothing runnable → static hypotheses only, each marked unconfirmed; no fix ships: report routes to reproduction steps or a provisional label per below.
• Prefer existing tests, logs, traces, metrics, and history before adding instrumentation.
• git bisect only with a deterministic, inexpensive, side-effect-safe signal; honor the skip contract and stop if the range cannot be classified.
• Replay with masked fixtures where inputs are deterministic; if a third party cannot be replayed, record its response contract and substitute with lower confidence.
• Profiling answers resource questions only -- never infer a functional cause from a hot frame; route performance diagnosis to hotpath.

When to use

• "this bug keeps coming back, figure out the real cause"
• "it works locally but breaks in production, why"
• "find what's actually causing this before you change anything"

When NOT to use

• The behavior is correct but slow → hotpath.
• The fix is known and documented → scout confirms, then just fix it.
• The user wants tests around known-good behavior → proof.

Operating modes

Do not call a static investigation confirmed. Do not call a production mitigation a root-cause fix unless it has broken the evidenced causal chain.

Investigation notebook

Create a small notebook in chat, an issue, or a local artifact before the first intervention. This prevents repeated dead ends and makes a handoff useful.

Keep facts, inferences, and unknowns visibly distinct. A useful negative result rules out something specific; it is not an empty line in the notebook.

Repository and system inspection strategy

1. Read repository instructions, incident/runbook context, recent change log, dependency/configuration manifests, and relevant test conventions.
2. Find the symptom boundary: user action, public API, queue, cron job, CLI, data import, or background worker. Capture expected and actual artifacts.
3. Map one request or job across code, configuration, state, and dependencies. Inspect interfaces and ownership boundaries before instrumenting internals.
4. Locate existing logs, traces, metrics, feature flags, test fixtures, and previous failures. Verify the time range, environment, release version, and correlation identifier match the reported event.
5. Diff known pass/fail cases: input, identity, data shape, locale/time, deployment, flag, dependency response, runtime version, and ordering.
6. Read references/techniques.md only for the situation encountered. It is a decision aid, not a list to perform blindly.

If evidence is missing, specify the smallest safe instrument at the closest boundary that separates the leading hypotheses. Logging every internal value is noise, cost, and sometimes a privacy incident.

Workflow

1. Reproduce

Reliably, from the reported symptom. Intermittent: extract the pattern (specific input? timing? order? environment?) before theorizing. Cannot reproduce at all → say so early; a fix without reproduction is a guess in a costume (see Provisional fixes).

2. Minimize

Strip the reproduction to the smallest input that still fails (smaller payload, fewer steps, isolated module, one commit's diff). Minimizing is diagnosis: the step that stops being required usually points at the mechanism.

3. Establish a failing signal

A test, a command, an assertion: something that fails now and will flip green when truly fixed. This is the arbiter for everything after; without it you cannot tell fix from coincidence. (proof formalizes it at the end.)

4. Gather evidence

Read errors/stacks fully (check exception chaining and the actual frame order for this runtime; no universal top/bottom rule locates the cause), logs around the failure, the diff between working and broken (git bisect when "used to work" is known), actual data in the actual store. The full checklist of techniques lives in references/techniques.md.

5. Hypothesize: competing explanations when plausible

Track them in a working table, not in your head:

"X is null" and "X is fine but the caller passes the wrong id": if both explain the evidence, keep both. Do not invent a second explanation when a deterministic contract violation already establishes the mechanism. The table forces the question the gut skips: what experiment separates these two? Run the cheapest one first.

6. Eliminate

Run the cheapest discriminating observation per hypothesis (log at the boundary, reproduce in isolation, binary-search the diff). Update or kill hypotheses on evidence: never bend evidence to fit the favorite.

Choose experiments by information gained per unit of risk and cost. A useful experiment says in advance: "if A is true, I expect X; if B is true, I expect Y; this observation would leave both unresolved." Prefer passive observation, then isolated replay, then reversible intervention. Record confounders such as cache warmth, changed timing from logging, different identity, data drift, or an upstream retry. Repeating the same observation without a changed condition is not new evidence.

7. Identify the root cause

Name all five parts of the causal chain, because fixes land on the wrong one otherwise: symptom (what the user sees) → mechanism (what actually happens inside) → root cause (the defect the mechanism flows from) → trigger (what activated it now) → contributing conditions (what made it possible). Keep asking "why" until the answer is a thing you can fix without the bug growing back elsewhere: not "the JSON is malformed" but "the uploader never escaped newlines, added in commit X". The Five Whys, but with evidence at each link, not vibes.

8. Smallest correct fix

Fixes the identified cause, nothing else. Unrelated bugs spotted en route → noted in the report, not fixed silently. If the honest fix needs a decision (schema change, API break) → stop and surface it.

9. Regression test

Turn the failing signal into a permanent test (proof boundary rules). No regression test = the bug is rented, not killed.

10. Verify

Signal green, related tests green, original reproduction clean. State what was verified and how.

Intermittent, concurrent, and distributed failures

Read references/techniques.md ("Intermittent, concurrent, and distributed failures" and "Fix-selection and verification decision framework") when the failure depends on timing, order, load, environment, or distribution. Intermittence is evidence about hidden inputs: capture the rate, compare pass/fail runs across dimensions, and select the narrowest cause-eliminating fix -- mitigations ship labeled.

Failure handling and edge cases

• Cannot reproduce: report commands, environments, inputs, and time range attempted; preserve evidence and ask for the smallest missing differentiator.
• Production impact: stabilize and preserve evidence first; keep a separate mitigation and diagnosis timeline.
• Noisy telemetry: validate time, cardinality, sampling, and release before using a graph. Add one narrow correlation field instead of broad debug logs.
• Flaky test: determine whether the test or product is nondeterministic. Do not quarantine it as a resolution without a linked cause and owner.
• Multiple causes: report separate causal chains and test each fix. Do not force a single root cause just to simplify the report.
• Unsafe experiment: stop and request authority or use a lower-risk observation. The most discriminating experiment is not always safe to run.

The Banned List (Anti-Patterns)

• The "Guess-and-Check" Loop: making random code edits and re-running the test hoping it passes, without updating the hypothesis table. If you don't know why it should work, don't run it.
• Repeating failed interventions: do not rerun an unchanged fix hoping for green. A failed fix can disprove the implementation, experiment setup, or causal hypothesis; record which prediction failed before rejecting the hypothesis or selecting a materially different experiment.
• Hallucinated Fixes: suggesting a patch before reading the runtime state (logs, stack traces, DB rows). A fix proposed without a confirmed mechanism is a hallucination.

Provisional fixes (the honest exception)

A workaround is not a root-cause fix: it suppresses the symptom and rents the bug. When the user explicitly accepts a hotfix without confirmed root cause (production is down, diagnosis needs days): ship it labeled PROVISIONAL: cause unconfirmed in the report, with the working hypothesis and the plan to confirm. Still ship the failing signal, and open the follow-up that converts provisional into root-cause. Provisional fixes without labels are how bugs get tenure.

Quality gates

• Cause chain complete: symptom → mechanism → root cause, each link with evidence (log line, stack frame, commit, observation).
• Competing hypotheses were considered whenever the evidence allowed more than one plausible cause; otherwise explain what forced the single hypothesis.
• The failing signal existed before the fix and flips after it.
• Diff contains the fix + the regression test, nothing else.

Stop conditions

• Verified fix + regression test → cause-chain report, stop.
• Cannot reproduce → report what was tried, the exact information needed (environment, input, timing), and instrument if possible. Do not ship a guess.
• Root cause sits above the code (product decision, upstream service, data migration) → report the evidence and route the decision.

Output contract

symptom:   checkout 500s intermittently (~1 in 20 submits)
cause:     Stripe webhook handler ignores `retry` events → duplicate
           order rows → unique-constraint violation surfaces as 500
evidence:  network.log shows double delivery; orders table has dup rows
           (ids 4812/4813); constraint error in app logs at 14:02:11
fix:       handler now dedupes on event id (upsert): 12 lines
verified:  regression test fails pre-fix, passes post-fix; 50/50
           webhook replay simulation clean

The chain is the deliverable. A fix without a chain is provisional, label it or don't ship it.

Research basis

Read references/research.md when a decision depends on an external standard, a numerical claim, or a fast-moving practice. The ledger records what the source supports, what it does not support, and when to reverify.