agentleFS
Sign inStart free
reddygopalkrishnareddygopalkrishna/developer-agent-skills/skills/root-cause-debugging/SKILL.md

root-cause-debugging

Skill8 starsChanged 8 days agoMITView on GitHub

Finds the real cause of a bug before changing code, using a minimal reproduction, ranked hypotheses, one-variable experiments and bisection (git bisect, input bisection). Use when something is broken, failing, crashing, flaky, intermittent, behaving differently between environments, a test fails for unclear reasons, or an earlier fix did not work. Not for performance tuning without a defect or for designing new features.

9 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use ui-ux-quality.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • AL-JANEFSkill

    ui-ux-quality

    Make product flows clear and recoverable: primary action and next step obvious, forms with clear labels, validation timing, actionable errors, preserved input on failure, consistent navigation and terminology, and no hidden or surprising destructive actions. Use for user flows, forms, navigation, onboarding, dialogs, and interaction design.

    Also in 3 projects that use this

The file

We split this skill into the rules it gives an agent

10 rules. 0 are common, 9 are distinctive, and 1 is very specific to this project.

  • 0

    Common

    Most of the 8 edited versions we read keep these.

  • 9

    Distinctive

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Never change code to fix a bug you cannot explain.

    line 9
  2. Distinctive:

    Not for: production outages where mitigation must come first (stabilize, then come back here), or performance work where nothing is …

    line 18
  3. Distinctive:

    If you cannot reproduce it, do not guess.

    line 30
  4. Distinctive:

    Narrow by bisection: when the search space is large:

    line 34
  5. Distinctive:

    History: if it worked at some earlier commit, use git bisect (automate it with git bisect run).

    line 35
  6. Very specific:

    Instrument where you are blind: Add logs at boundaries with a unique prefix (e.g.

    line 39
  7. Distinctive:

    Clean up and report: Remove temporary instrumentation.

    line 43
  8. Distinctive:

    If three fixes in a row have not worked, stop editing code.

    line 60
  9. Distinctive:

    Write down every assumption you are making and verify each one.

    line 64
  10. Distinctive:

    Ask whether the design itself is the problem (shared mutable state, wrong abstraction, hidden coupling).

    line 65

“Common” is measured: kept by at least 6 in 10 of the 8 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Root-Cause Debugging

Never change code to fix a bug you cannot explain. Reproduce it, find the cause with evidence, and prove the fix with a test that failed before the fix and passes after it.

When to use

  • An error, exception, crash, wrong output, failing test or regression.
  • "Works on my machine", or works in one environment but not another.
  • Intermittent or flaky failures.
  • A fix was already tried and the bug came back or moved.

Not for: production outages where mitigation must come first (stabilize, then come back here), or performance work where nothing is actually wrong.

Workflow

  1. Capture the exact failure. Copy the full error message, stack trace, exit code, relevant log lines, and the exact command, request or steps that produced them. Write down expected vs actual behavior.
  2. Read the whole stack trace. Include every Caused by: / inner exception / wrapped error. The top frame is often library code; the first frame in the project's own code is usually where to start. The line where an error surfaces is where a bad value was used, not necessarily where it was produced.

  3. Reproduce it. Get a command that fails every time. Then shrink it: smallest input, fewest steps, a single test, for example:

    • pytest tests/test_orders.py::test_refund -x
    • npx vitest run src/cart.test.ts -t "applies discount"
    • go test -run '^TestRefund$' ./orders
    • dotnet test --filter "FullyQualifiedName~RefundTests"

    If you cannot reproduce it, do not guess. Collect more data (step 7) or ask the user for environment details, logs and exact steps.

  4. Check the boring causes first (table below). A large share of "impossible" bugs are stale builds, wrong branches, wrong config or a different dependency version.
  5. Write ranked hypotheses. List 2-5 candidate causes. For each: evidence for, evidence against, and the cheapest experiment that would confirm or rule it out. Start with the one that is most likely and cheapest to test.
  6. Experiment one variable at a time. Predict the result before running. If the result surprises you, your model of the system is wrong: update it before continuing. Revert experiments that did not help.
  7. Narrow by bisection when the search space is large:
    • History: if it worked at some earlier commit, use git bisect (automate it with git bisect run). See references/git-bisect.md.
    • Input: halve the input file, payload, dataset or list of tests until you find the minimal failing case.
    • Code path: disable half of the suspect code (early return, feature flag, stub) and see which half matters.
    • Environment: diff a working and a failing environment (versions, env vars, config, OS, data) and apply half of the differences at a time.
  8. Instrument where you are blind. Add logs at boundaries with a unique prefix (e.g. DBG-1234), log types and lengths as well as values, use breakpoints and conditional breakpoints, or attach a tracer. See references/debugging-tools.md for per-language tools. Never log secrets or personal data.
  9. State the root cause as a mechanism. "The cache key omits tenant_id, so tenant B reads tenant A's entry" — not "bug in cache.py". Keep asking "why?" until you reach something you can fix that prevents this class of bug, not just this instance.
  10. Write a failing regression test at the lowest level that still shows the bug. Run it and confirm it fails for the right reason (the assertion you expect, not an import or setup error).
  11. Fix the cause, then run the new test and the surrounding suite. Search the codebase for the same pattern (same call, same misuse) and fix or report the other places.
  12. Clean up and report. Remove temporary instrumentation. Tell the user: symptom, root cause, evidence, fix, regression test, other places affected, and anything you could not verify.

Check assumptions first

AssumptionHow to verify
You are running the code you think you areAdd a temporary log line and confirm it prints. Check git status, git branch --show-current, git log -1. Rebuild. Check which binary or interpreter runs: which/command -v (POSIX), where (Windows), python -c "import sys; print(sys.executable)".
Dependency versions matchRead the lockfile. npm ls <pkg>, pip show <pkg>, go list -m all, mvn dependency:tree, ./gradlew dependencies, dotnet list package, cargo tree.
Config and env vars are what you expectPrint the effective config at startup (redacted). Check precedence: defaults, files, env vars, CLI flags, remote config.
Nothing is cached or staleClean build (delete build/output dirs), rebuild containers with --no-cache, clear package caches, browser cache, CDN, service worker, memoization layers.
The data is what you thinkQuery the actual record or message. Do not reason from what the data "should" be.
Time, timezone and localeUTC vs local, DST transitions, month-end, leap days, locale-specific number and date parsing.
Network pathDNS resolution, proxies, TLS, which host and port is actually hit (curl -v).
Permissions and identityFile permissions, OS user, service account, IAM role, token scopes.

The three-strike rule

If three fixes in a row have not worked, stop editing code. Each failed fix is evidence that your hypothesis is wrong.

  1. Revert the failed attempts.
  2. Re-read the original evidence and the full stack trace.
  3. Write down every assumption you are making and verify each one.
  4. Ask whether the design itself is the problem (shared mutable state, wrong abstraction, hidden coupling).
  5. Tell the user what you have ruled out and what you plan to try next.

Intermittent bugs

SymptomLikely causesTechnique
Fails only under load or in parallelRace condition, shared mutable state, pool exhaustion, missing lock or transactionRepeat with parallelism: go test -race -count=100 ./..., pytest -n 8 (pytest-xdist) with --count=50 (pytest-repeat). Use race detectors: Go -race, ThreadSanitizer (-fsanitize=thread) for C/C++.
Depends on test orderState leaking between tests (globals, DB rows, env vars, singletons, mocks not reset)Run in random order: pytest-randomly, jest --randomize, vitest --sequence.shuffle, go test -shuffle=on. Run the failing test alone, then with its neighbors.
Fails after hours or daysMemory, file handle, socket or connection leak; counter overflow; token or certificate expiryTrack resource counts over time; heap snapshots; check what expires.
Fails only in CIDifferent OS, CPU count, timezone, locale, missing secrets, slower machine hitting timeouts, parallel jobs sharing resourcesDiff the environments; run the CI container image locally; print env (redacted) in the CI job.
Disappears with logging or a debuggerTiming-sensitive raceUse low-overhead instrumentation (in-memory ring buffer dumped on failure), stress runs, race detectors.
Fails at certain timesMidnight, month or year boundaries, DST, leap years, cron overlapInject a clock and test with fixed times around the boundary.

For an intermittent bug, "fixed" means the reproduction passes many repetitions (e.g. 100+ runs or a stress run) that previously failed at a known rate.

Anti-patterns

  • Shotgun debugging — changing several things at once. When it starts working you do not know why, and you may have added new bugs.
  • Fixing the symptom — swallowing the exception, adding a null check without knowing why the value is null, adding a retry or sleep to hide a race, raising a timeout. These move the bug, they do not remove it.
  • Trusting the error location — the crash site is where the bad value was used. Trace the value backwards to where it was produced.
  • Reasoning instead of running — reading code for an hour when a 30-second experiment would answer the question.
  • Blaming the platform first — compiler, library and OS bugs exist but are rare. Rule out your own code and config first, and get a minimal reproduction before claiming one.
  • Declaring victory early — the symptom disappeared once, so it must be fixed. Re-run the reproduction, and for flaky bugs, re-run it many times.
  • Leaving debris — debug logs, commented-out code, disabled tests, temporary flags.

Done when

  • You can state the root cause as a mechanism and point to the evidence.
  • A regression test failed before the fix and passes after it.
  • For intermittent bugs, the reproduction passes many repeated runs that failed before.
  • The same pattern was searched for elsewhere and handled or reported.
  • Temporary instrumentation is removed and the relevant test suite passes.

References

  • references/git-bisect.md — read when the bug is a regression (it worked at some earlier commit or version).
  • references/debugging-tools.md — read when you need a debugger, thread or heap dump, tracer, sanitizer or network tool for a specific language or runtime.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free