fix-flaky-test
Skill2 starsChanged 26 days agoMITView on GitHub
Use when a Playwright test is flaky, intermittent, or "passes locally but fails in CI", or the user asks to stabilise, de-flake or investigate a test that sometimes fails. Reproduces the flake, finds the root cause from traces, fixes it without retries or sleeps, and proves the fix over repeated runs.
6 other projects use a version of this.
Adopted by
Most recently taken up by
- yaohuangguan/remote-arctheir own version, first seen Nov 1, 2026
- Remote-Arc/remote-arctheir own version, first seen Nov 1, 2026
- yaohuangguan/agent-playbooktheir own version, first seen Nov 1, 2026
- dotnet/aspiretheir own version, first seen Oct 8, 2026
- qazando/skills-qastheir own version, first seen Jul 29, 2026
- microsoft/aspiretheir own version, first seen May 27, 2026
Used with
Projects with it also use audit-test-suite, change-data-safely and 10 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
yaohuangguanSkillaudit-test-suite
Find where an existing test suite is weak and decide what to test next, ranked by risk. Use this when asked to add missing tests, find gaps, raise coverage, assess test quality or see which areas are thin, even if the request only mentions a coverage percentage.
Also in 3 projects that use this
yaohuangguanSkillchange-data-safely
Change a database schema or stored data without breaking running code or losing data. Use this for any migration, new or changed column, backfill, bulk update, delete, index, or change to a stored format, including in small projects. Data mistakes are often impossible to undo.
Also in 3 projects that use this
SauravSrivastavSkilldebug-failure
Diagnose a failing test, command, or runtime behavior using evidence and produce a minimal verified fix.
Also in 3 projects that use this
dabstractorSkillwrite-pull-request
Write a pull request to an open source repository. Lead with the user-facing problem in a personal voice, then a scannable technical fix. Use when opening or drafting a PR or contribution.
Also in 3 projects that use this
Codagent-AISkillimplement-change
Implements an approved change in one continuous session, with tiered test runs. Use when asked to implement a change from its approved definition, or to invoke codagent:implement-change.
Also in 3 projects that use this
srclightSkillrefactor-safely
This skill should be used when the user asks to "refactor", "rename", "move", "change a function signature", "extract a method", "delete unused code", "clean up", or any modification that could break callers or dependents. Provides a pre-change safety checklist using srclight's graph tools to assess blast radius before touching code.
Also in 3 projects that use this
- +6 more from yaohuangguan
Other versions6 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 6
- dotnet/aspire★ 6.4ksame name
- microsoft/aspire★ 6.4ksame name
- yaohuangguan/remote-arc★ 5same name
- qazando/skills-qas★ 4same name
- Remote-Arc/remote-arc★ 1same name
- yaohuangguan/agent-playbook★ 1same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository6 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- implement-test-caseSkill
- playwright-cliSkill
- playwright-traceSkill
- skill-creatorSkill
The file
We split this skill into the rules it gives an agent
9 rules. 0 are common, 7 are distinctive, and 2 are very specific to this project.
0
Common
Most of the 5 edited versions we read keep these.
7
Distinctive
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 13
Never trust a single run. Run the test repeatedly from tests/ and record the pass count:
- Very specific: line 19
If it never fails with --repeat-each, the flake may depend on which tests run together.
- Distinctive: line 26
Write the numbers down. They are the baseline the fix has to beat.
- Distinctive: line 60
Check the project's AGENTS.md and fixtures.
- Distinctive: line 69
Not a fix, never propose one as the answer:
- Distinctive: line 79
If the root cause is in the app (a real race a user could hit), fix the test anyway and report the app issue separately.
- Distinctive: line 84
Run the fixed test with the same method that reproduced the flake, at least 20 times, plus the project's typecheck.
- Distinctive: line 87
before: N/M passed; after: M/M passed.
- Distinctive: line 91
If you didn't see a green repeated run, say so.
“Common” is measured: kept by at least 6 in 10 of the 5 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Fix a flaky test
A flaky test is a test with a hidden assumption about timing or shared state. The job is to find that assumption, not to make the symptom rarer.
1. Reproduce and measure
Never trust a single run. Run the test repeatedly from tests/ and record the pass count:
cd tests && npx playwright test <file> --repeat-each=20If it never fails with --repeat-each, the flake may depend on which tests run
together. Run the whole file (or suite) several times without it, and compare
with --workers=1 and with the test run alone (-g "<title>"):
- fails only alongside other tests → shared state (step 3, category E)
- fails alone too → timing (categories A–D)
Write the numbers down. They are the baseline the fix has to beat.
2. Collect evidence
Failed runs keep a trace (trace: 'retain-on-failure'). Passing runs don't —
add --trace on to the reproduce command to keep both. Read them from the CLI
with the playwright-trace skill, not the GUI viewer:
npx playwright trace open test-results/<failing-run>/trace.zip
npx playwright trace actions --errors-only # which step failed
npx playwright trace requests --grep api # what was in flight, in what order
npx playwright trace snapshot <action-id> # the DOM at that stepThen open a passing trace and compare:
- requests: which were in flight at each step, and in what order they finished;
- snapshot at the failing assertion: what was actually on screen;
- actions: which step returned before the app had finished reacting.
State the root cause as one sentence: "The test does X assuming Y, but Y is only true when Z."
3. Classify
| Category | Typical smell | Typical fix | |
| A | Snapshot read | expect(await locator.textContent()), .count(), .isVisible(), allTextContents() | Web-first assertion: await expect(locator).toHaveText/... |
| B | Waiting for the wrong signal | waitForResponse with a loose predicate, networkidle, waiting for an element that renders before the data | Assert on the outcome the user sees; if a request must finish, match it exactly and register the wait before the action |
| C | Optimistic UI / in-flight writes | UI changes before the server confirms, then a reload or API check | Wait for the write's response, or poll the API until it agrees |
| D | Real time | waitForTimeout, setTimeout, sleeps, "X seconds plus a margin" | page.clock for browser timers; expect.poll / toPass for server-side processing |
| E | Shared data | Hard-coded ids, global resets, tests that pass alone but fail together | Give each test or worker its own data (fixtures); scope cleanup to the test's own data |
Check the project's AGENTS.md and fixtures. Flaky tests often break a rule the
project already has.
4. Fix
Allowed: web-first assertions, expect.poll, expect(...).toPass(), precise
waitForResponse set up before the action, page.clock, fixtures that isolate
data, documented timeouts for documented waits.
Not a fix, never propose one as the answer:
retries,test.slow(), or raising a timeout "to be safe";page.waitForTimeout, sleeps, or polling loops with a fixed delay;workers: 1,test.describe.serial, or--workers=1;- skipping,
test.fixme, or deleting the assertion that fails.
These make a flake rarer, slower, or invisible. If the user asks for one, say what it hides before doing it.
If the root cause is in the app (a real race a user could hit), fix the test anyway and report the app issue separately. Don't modify the app.
5. Prove it
Run the fixed test with the same method that reproduced the flake, at least 20 times, plus the project's typecheck. Report:
- before: N/M passed; after: M/M passed;
- the one-sentence root cause;
- why the fix removes the cause rather than reducing its probability.
If you didn't see a green repeated run, say so. Don't claim it.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

