agentleFS
nora-weissernora-weisser/ai-assisted-testing-demo/.claude/skills/fix-flaky-test/SKILL.md

fix-flaky-test

Skill2 starsChanged 26 days agoMITView on GitHub

Use when a Playwright test is flaky, intermittent, or "passes locally but fails in CI", or the user asks to stabilise, de-flake or investigate a test that sometimes fails. Reproduces the flake, finds the root cause from traces, fixes it without retries or sleeps, and proves the fix over repeated runs.

6 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use audit-test-suite, change-data-safely and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • yaohuangguanSkill

    audit-test-suite

    Find where an existing test suite is weak and decide what to test next, ranked by risk. Use this when asked to add missing tests, find gaps, raise coverage, assess test quality or see which areas are thin, even if the request only mentions a coverage percentage.

    Also in 3 projects that use this

  • yaohuangguanSkill

    change-data-safely

    Change a database schema or stored data without breaking running code or losing data. Use this for any migration, new or changed column, backfill, bulk update, delete, index, or change to a stored format, including in small projects. Data mistakes are often impossible to undo.

    Also in 3 projects that use this

  • SauravSrivastavSkill

    debug-failure

    Diagnose a failing test, command, or runtime behavior using evidence and produce a minimal verified fix.

    Also in 3 projects that use this

  • dabstractorSkill

    write-pull-request

    Write a pull request to an open source repository. Lead with the user-facing problem in a personal voice, then a scannable technical fix. Use when opening or drafting a PR or contribution.

    Also in 3 projects that use this

  • Codagent-AISkill

    implement-change

    Implements an approved change in one continuous session, with tiered test runs. Use when asked to implement a change from its approved definition, or to invoke codagent:implement-change.

    Also in 3 projects that use this

  • srclightSkill

    refactor-safely

    This skill should be used when the user asks to "refactor", "rename", "move", "change a function signature", "extract a method", "delete unused code", "clean up", or any modification that could break callers or dependents. Provides a pre-change safety checklist using srclight's graph tools to assess blast radius before touching code.

    Also in 3 projects that use this

  • +6 more from yaohuangguan

The file

We split this skill into the rules it gives an agent

9 rules. 0 are common, 7 are distinctive, and 2 are very specific to this project.

  • 0

    Common

    Most of the 5 edited versions we read keep these.

  • 7

    Distinctive

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Never trust a single run. Run the test repeatedly from tests/ and record the pass count:

    line 13
  2. Very specific:

    If it never fails with --repeat-each, the flake may depend on which tests run together.

    line 19
  3. Distinctive:

    Write the numbers down. They are the baseline the fix has to beat.

    line 26
  4. Distinctive:

    Check the project's AGENTS.md and fixtures.

    line 60
  5. Distinctive:

    Not a fix, never propose one as the answer:

    line 69
  6. Distinctive:

    If the root cause is in the app (a real race a user could hit), fix the test anyway and report the app issue separately.

    line 79
  7. Distinctive:

    Run the fixed test with the same method that reproduced the flake, at least 20 times, plus the project's typecheck.

    line 84
  8. Distinctive:

    before: N/M passed; after: M/M passed.

    line 87
  9. Distinctive:

    If you didn't see a green repeated run, say so.

    line 91

“Common” is measured: kept by at least 6 in 10 of the 5 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Fix a flaky test

A flaky test is a test with a hidden assumption about timing or shared state. The job is to find that assumption, not to make the symptom rarer.

1. Reproduce and measure

Never trust a single run. Run the test repeatedly from tests/ and record the pass count:

cd tests && npx playwright test <file> --repeat-each=20

If it never fails with --repeat-each, the flake may depend on which tests run together. Run the whole file (or suite) several times without it, and compare with --workers=1 and with the test run alone (-g "<title>"):

  • fails only alongside other tests → shared state (step 3, category E)
  • fails alone too → timing (categories A–D)

Write the numbers down. They are the baseline the fix has to beat.

2. Collect evidence

Failed runs keep a trace (trace: 'retain-on-failure'). Passing runs don't — add --trace on to the reproduce command to keep both. Read them from the CLI with the playwright-trace skill, not the GUI viewer:

npx playwright trace open test-results/<failing-run>/trace.zip
npx playwright trace actions --errors-only    # which step failed
npx playwright trace requests --grep api      # what was in flight, in what order
npx playwright trace snapshot <action-id>     # the DOM at that step

Then open a passing trace and compare:

  • requests: which were in flight at each step, and in what order they finished;
  • snapshot at the failing assertion: what was actually on screen;
  • actions: which step returned before the app had finished reacting.

State the root cause as one sentence: "The test does X assuming Y, but Y is only true when Z."

3. Classify

CategoryTypical smellTypical fix
ASnapshot readexpect(await locator.textContent()), .count(), .isVisible(), allTextContents()Web-first assertion: await expect(locator).toHaveText/...
BWaiting for the wrong signalwaitForResponse with a loose predicate, networkidle, waiting for an element that renders before the dataAssert on the outcome the user sees; if a request must finish, match it exactly and register the wait before the action
COptimistic UI / in-flight writesUI changes before the server confirms, then a reload or API checkWait for the write's response, or poll the API until it agrees
DReal timewaitForTimeout, setTimeout, sleeps, "X seconds plus a margin"page.clock for browser timers; expect.poll / toPass for server-side processing
EShared dataHard-coded ids, global resets, tests that pass alone but fail togetherGive each test or worker its own data (fixtures); scope cleanup to the test's own data

Check the project's AGENTS.md and fixtures. Flaky tests often break a rule the project already has.

4. Fix

Allowed: web-first assertions, expect.poll, expect(...).toPass(), precise waitForResponse set up before the action, page.clock, fixtures that isolate data, documented timeouts for documented waits.

Not a fix, never propose one as the answer:

  • retries, test.slow(), or raising a timeout "to be safe";
  • page.waitForTimeout, sleeps, or polling loops with a fixed delay;
  • workers: 1, test.describe.serial, or --workers=1;
  • skipping, test.fixme, or deleting the assertion that fails.

These make a flake rarer, slower, or invisible. If the user asks for one, say what it hides before doing it.

If the root cause is in the app (a real race a user could hit), fix the test anyway and report the app issue separately. Don't modify the app.

5. Prove it

Run the fixed test with the same method that reproduced the flake, at least 20 times, plus the project's typecheck. Report:

  • before: N/M passed; after: M/M passed;
  • the one-sentence root cause;
  • why the fix removes the cause rather than reducing its probability.

If you didn't see a green repeated run, say so. Don't claim it.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.