Flaky test hunt

"Flaky" describes a symptom, not a cause. A test that fails some of the time is reporting a real nondeterminism: in the test, in its environment, or in the product. This skill finds which one, fixes it, and proves the fix with numbers rather than with one green run.

Context from the user: $ARGUMENTS

Reference files, read when you reach their phase:

• ${CLAUDE_SKILL_DIR}/COMMANDS.md: per-ecosystem commands to isolate, repeat, shuffle and parallelize
• ${CLAUDE_SKILL_DIR}/TAXONOMY.md: the causes, each with symptoms, how to confirm, right and wrong fix
• ${CLAUDE_SKILL_DIR}/repeat.mjs: runs any command N times and reports the failure rate, a 95% interval and the distinct failure signatures

Rules that apply throughout

• Do not "fix" by retrying, waiting longer, or hiding the test. Adding retries (jest.retryTimes, pytest --reruns, Surefire rerunFailingTestsCount, a CI retry step), raising a timeout, adding or lengthening a sleep, skipping, or quarantining are not fixes. Each one makes the signal quieter while the cause stays, and the cause is sometimes a real product bug that users will hit. You may propose one of these as a temporary measure, clearly labelled as such, only with the user's explicit approval, and only alongside a ticket for the real fix.
• Do not call anything "flaky" in the report without a category from TAXONOMY.md and the evidence for it. If you could not find the cause, say "not reproduced" or "cause unknown" and list what you tried and the counts.
• Do not change the assertion to make it pass (loosening equality, widening a tolerance, deleting the check) unless the analysis shows the assertion itself was wrong, and say so explicitly.
• Numbers, not impressions. "Ran it a few times and it passed" is not evidence. Every claim about frequency comes from a counted run: failures / runs, with the mode it ran in.
• Change one thing at a time between measured batches, so a change in the rate can be attributed.

Phase 1. Capture the evidence

Before running anything, collect what already exists. Read it from the repository and CI; ask the user only what those cannot answer. For a CI failure, do "Is it a flake at all?" below first.

• The exact test id (file, class, name, parameters) and the full failure output of at least one failing run: assertion message, stack trace, and the lines before it. Save it; later phases compare against it.
• How often: count failures of this test or job over recent CI history (for GitHub Actions, gh run list --workflow <file> --limit 100 --json conclusion,headSha,createdAt and gh run view <id> --log-failed). A rough historical rate tells you how many local runs you will need to see it.
• Where: which runners, OS images, architectures, language versions, shard or worker numbers, and time of day. A failure that only appears on one shard, after midnight UTC, or on the macOS runner is already a strong clue.
• What changed: when it started failing, and the commits, dependency bumps, CI image changes and test additions around that date. git log --since on the test file, the code it covers and the test config.
• Whether the failure output is the same every time. Two different messages are often two different problems.
• The test runner and how CI invokes it (the actual command in the workflow, with its flags, env and parallelism).

Is it a flake at all?

When the failure is in CI, and above all when the question is "can we just retry it" or "it passes locally", classify it before treating it as flaky, in this order:

1. Is the failing code in the diff? Run git fetch first, so the commands below see the current base. If git rev-parse --is-shallow-repository prints true, deepen first (git fetch --deepen=<n> or --unshallow): a shallow clone gives no merge base, a wrong ancestry answer, and lists the boundary commit as touching every file. Then compare the failing test and the code it covers with the branch's changes (git diff --name-only origin/<base>...HEAD). If they do not overlap, suspect a stale base, even when the failure reproduces locally: a branch that lacks the base's fix fails the same way everywhere.
   • For a GitHub pull_request run (and any CI that tests a merged result), CI did not test the branch. It tested a merge commit whose message names the base (Merge <head> into <base sha>, in the checkout step's log), so check for the base's fix against that base SHA, not against origin/<base>. The merge commit is not in an ordinary clone: git fetch origin pull/<N>/merge (or git fetch origin <sha>); exit 128 means the server no longer has it, so read the base SHA from the log instead. Also check whether the base branch itself is red at that SHA; if it is, the failure is not this branch's.
   • Otherwise run git merge-base --is-ancestor origin/<base> HEAD: exit 0 means the branch contains the base tip, exit 1 means it is behind, so look at what the base has for the failing code (git log HEAD..origin/<base> -- <files>). A fix on the base means the answer is to update the branch, not to retry and not to patch outside the diff what the base already fixed. Offer to bring the branch up to date the way the project does (merge or rebase), so CI runs once on the result.
2. One re-run at most, as a diagnostic. A single re-run shows whether the failure repeats. It is never the fix (see the rules above). Running the same test locally is free and counts. A CI re-run costs runner minutes, so ask before starting it unless the user asked for a re-run; a question such as "can we just retry it?" is not that request. Do not re-run CI a second time.
3. Two identical failures are not proof of a consistent failure. A test that fails one run in three fails twice in a row one time in nine. Read the test's CI history first. End the hunt as a consistent failure only when history shows no pass for this test at this commit and a few local runs (for example 3 of 3) fail the same way. Then say what failed and stop, and offer to run bug-fix if it is installed; with or without it, ask before debugging.
4. Everything else (a pass on a re-run of the same commit, a message that changes between runs, a history of occasional failures) goes on to Phase 2.

Done when you can state the failing test, one or more saved failure outputs, the historical frequency (or "unknown"), the environments it fails in, and the CI command, and, for a CI failure, which of the cases above it is.

Phase 2. Reproduce by repetition, with a measured rate

Detect the project's own runner first (lockfiles, package.json scripts, pyproject.toml, go.mod, Cargo.toml, build files, the CI workflow) and use the commands in COMMANDS.md for it. Then run the ladder below, recording failures / runs for each rung. Use repeat.mjs to do the counting, so the numbers and the failure signatures are not tallied by hand:

node "${CLAUDE_SKILL_DIR}/repeat.mjs" --runs 50 --timeout 120 --json flake-isolated.json -- pytest "tests/test_x.py::test_y" -p no:randomly

It sets REPEAT_RUN in each run's environment, groups failures by a normalized signature of their last lines, and exits 1 if any run failed. Add --parallel K to run copies at once, --until-fail to stop at the first failure, and --shell for npm/npx on Windows, pipes or &&.

Pick N from the historical rate: to have a good chance (about 95%) of seeing a failure that happens with probability p, you need about 3/p runs. A 10% flake needs about 30 runs; a 1% flake needs about 300, which usually means running the isolated test in a loop rather than the suite.

If it does not reproduce locally, make the local run look like CI before raising N: same command and flags, same worker count, CI=true, same timezone and locale (TZ=UTC, LANG=C.UTF-8), fewer CPUs (taskset -c 0 on Linux, docker run --cpus=1), background load, a clean checkout and cache. If it still does not reproduce, run the counting in CI itself (a manual workflow that runs repeat.mjs on the same runner image).

Done when you have a table of rungs with failures / runs each, at least one rung where it fails, and the failing seed if order matters. If no rung fails after the CI-like attempts, stop and report "not reproduced" with the table.

Phase 3. Classify the cause

Open TAXONOMY.md and match the evidence against the categories: order dependence and shared state, time and date boundaries, unseeded randomness, concurrency and races, async waits and sleeps, network and external services, resource exhaustion (ports, files, memory, handles), pollution of globals and environment variables, floating point, unordered collections, environment differences, and real product bugs that only appear under certain timing.

The rung where it first failed narrows the list. The failure signatures from repeat.mjs tell you whether there is one cause or several; treat each distinct signature as its own hunt.

Then confirm the category with the experiment TAXONOMY.md gives for it. The strongest confirmation turns the intermittent failure into a deterministic one: force the bad order, freeze the clock at the boundary, fix the seed that fails, inject a delay at the suspected race point, block the network. A test that now fails 20/20 under the forced condition and 0/20 without it has a confirmed cause. Keep that forcing setup; it often becomes the regression test.

Done when one category is named, with the confirming experiment and its counts. "Probably a race" without a forced reproduction is a hypothesis, not a classification; say so if that is where you stopped.

Phase 4. Fix the root cause and prove it

Apply the right fix for the category from TAXONOMY.md, where the cause is, even in code the branch never touched. If the cause is in the product (a real race, a missing await in application code, a date bug), fix the product and tell the user plainly that the test was right. If the fix chooses a product behaviour (a tie-break order, a rounding rule, which of two results wins), describe the choice and get the user's yes before the edit.

Then prove it, in this order:

1. The forced reproduction from Phase 3 now passes, or is kept as a regression test that fails without the fix (revert the fix once to watch it fail, then restore it).
2. Re-run the rung that reproduced it, in the same mode, until the clean count is large enough to mean something.
3. Run the full suite once to check the fix broke nothing else.

How many clean runs are enough: if the failure rate before the fix was p, then 0 failures in n runs makes a remaining rate of p or more unlikely at about 95% confidence when n is about 3/p. This is the rule of three: with zero events in n trials, the upper 95% bound on the rate is roughly 3/n. Examples:

If the "before" rate came from only a few failures, its interval is wide; use its lower end (printed by repeat.mjs) to pick n, which asks for more runs and is the honest choice. Twenty clean runs after a 5% flake proves almost nothing: a 5% flake passes 20 times in a row about a third of the time.

Done when the report can say "before: X/N in mode M; after: 0/N' in the same mode", with N' at least about 3/p, and the full suite green.

Phase 5. Report

Produce this, filled in:

## Flaky test hunt: <test id>

Symptom:        <failure message, one line> (saved output: <path or excerpt>)
History:        <failures / runs in CI over <period>, or "unknown">; environments: <where it failed>
Reproduction:
  | Rung | Command (short) | Failures / runs | 95% interval |
  | ...  | ...             | ...             | ...          |
  Failing seed / condition: <seed, order, TZ, worker count, or "none">
Category:       <TAXONOMY.md category>
Evidence:       <the confirming experiment and its counts, e.g. "forced order A then B: 20/20 fail; B alone: 0/20">
Root cause:     <file:line and one or two sentences on the mechanism>
Fix:            <what changed and why it removes the cause; product or test>
Proof:          before <X/N> -> after <0/N'> in <mode>; regression test <name> fails without the fix: yes/no
Full suite:     <pass/fail, command>
Not done:       <other signatures seen, related tests with the same pattern, anything left for the user>

If the cause was not found, the report says "Cause: not found" and keeps the reproduction table, the categories ruled out with the experiment that ruled each out, and what to try next. It does not say "flaky" as a conclusion.

When to stop and ask

• The only fix you can find is a retry, a longer timeout, a skip or a quarantine: present the evidence and ask.
• The cause is in a third-party service or in CI infrastructure the user controls but you cannot change.
• Reproducing needs more runs than are reasonable locally (a 0.1% flake needs thousands): propose running the counter in CI and estimate its cost first.