Flaky Test Fixer

When this fires

Flaky tests get "fixed" by re-running until green, adding a retry wrapper, or bumping a timeout — none of which address why the test is nondeterministic in the first place. The flake keeps eating CI time and occasionally masks a real bug. This skill forces a classify-prove-fix sequence instead of a band-aid.

Protocol

1. Classify the likely nondeterminism source. Common categories:

   • Time/clock — wall-clock assertions, timing-sensitive waits
   • Test ordering — state leaking between tests, order-dependent fixtures
   • Concurrency/races — unsynchronized shared state across threads/async
   • Shared mutable state — module/class-level globals mutated by multiple tests
   • Network/IO — real network calls, filesystem races, external services
   • Random seeds — unseeded randomness in test data or logic

   Run scripts/detect_nondeterminism.py <path> to get a ranked list of suspects across these categories as a starting point — it's a grep-based heuristic, not a diagnosis by itself.

2. Prove it. Reproduce the nondeterminism directly:
   • Re-run the test N times in isolation (e.g. 20–50x) and record the pass/fail rate.
   • Re-run the full suite with test order randomized/reversed to check for ordering dependence.
   • If a category from step 1 is suspected, target evidence for it specifically (e.g. run with a fixed seed to confirm randomness is the driver, or run single-threaded to rule out races).
3. Quarantine — if the test needs to stop blocking CI while under investigation, mark/skip it explicitly with a comment referencing a tracking issue and an owner. Quarantining is a holding pattern, not a fix, and must never be silent (no unexplained skips).
4. Fix at the source. Address the actual mechanism found in step 2 — seed the randomness, isolate the shared state, mock the clock/network, fix the race. Do not fix flakiness by adding blind retries or increasing timeouts as the primary remedy; that hides the mechanism instead of removing it.
5. Re-run N times to confirm stability before removing any quarantine. The same N (or greater) used to prove the flake in step 2 must now pass consistently.

Hard rules

• Never treat "add a retry" or "increase the timeout" as the fix — those are acceptable only as a temporary quarantine measure, explicitly labeled as such, never presented as root-caused.
• Never permanently delete or silently skip a flaky test without stating the suspected or confirmed cause.
• A cause is not confirmed until reproduced — "probably a race condition" without a repeated-run or targeted test is a hypothesis, not a finding.
• Quarantine must be trackable: a skip/xfail needs a comment with a reason and a reference (ticket, TODO with owner) — never a bare skip.

Output

Classification: <category from step 1, with reasoning>
Proof: <N runs, X passed / Y failed — and/or the targeted evidence gathered>
Quarantine: <applied / not needed — reference if applied>
Fix: <what was changed and why it addresses the mechanism, not the symptom>
Stability re-run: <N/N passed after fix>