Flaky Test Fixer

Overview

A flaky test has a cause that can be found: timing, shared state, order, environment or randomness. Reproduce the flake reliably, identify which of those it is, and fix the cause. Retries and longer timeouts hide the problem; they do not fix it.

Method

1. Reproduce the flake. Run the test many times, alone and with the full suite:
   • Jest/Vitest: run in a loop, or --runInBand vs parallel.
   • pytest: pytest -x --count=50 (pytest-repeat) and random order with pytest-randomly.
   • Playwright: npx playwright test <file> --repeat-each=20 --workers=4.
   • JUnit: @RepeatedTest(50) on the test, or run the suite in a shell loop. Record the failure rate and the exact error.
2. Classify using the table below. Compare passing and failing runs: logs, timing, order, environment.
3. Confirm the cause with a deliberate change that makes it fail every time (shuffle order, add a delay, slow the network, change the time zone).
4. Fix the cause, then rerun the loop until there are zero failures across many runs.
5. Remove any retries, sleeps or skips added as workarounds.

Causes and fixes

Output

Test: <name and file>
Failure rate: <before> → <after> over <N> runs
Cause: <category and the specific line or condition>
Fix: <change made>

Common mistakes

• Adding retries or increasing timeouts and calling it fixed.
• Running the test once after the fix.
• Fixing the test when the flake is a real race condition in product code.
• Quarantining tests with no owner or follow-up date.

Made by Wageesha Tripathi · LinkedIn · Email · More skills