ci-quarantine
Skill11 starsChanged 51 days agoMITView on GitHub
Put a known-flaky test into, or take it out of, CI Steward's quarantine lane: it keeps running and reporting but cannot fail the merge queue. Use when a test that passes on retry keeps ejecting green PRs, when an entry is about to expire, or when a quarantined test is fixed and should block again.
In this repository44 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .claude/CLAUDE.mdCLAUDE.md
- auditing-uiSkill
- browser-testingSkill
- capturing-product-mediaSkill
- ci-local-exportSkill
- ci-pulseSkill
- ci-statusSkill
- creating-pull-requestsSkill
- debugging-systematicallySkill
- debugging-test-failuresSkill
- debugging-typescript-errorsSkill
- deps-weeklySkill
- designing-frontendSkill
- opensrcSkill
- organizing-fsd-architectureSkill
- runtime-watchSkill
- stewarding-ci-pipelineSkill
- syncing-agent-skillsSkill
- verification-before-completionSkill
- vision-drift-checkSkill
- visual-companionSkill
- writing-adrsSkill
- writing-app-copySkill
- writing-blog-postsSkill
- writing-changelogsSkill
- writing-developer-guidesSkill
- writing-for-humansSkill
- writing-to-usersSkill
- adding-config-fieldsSkill
- clarifying-requirementsSkill
- maintaining-dependenciesSkill
- maintaining-dev-playgroundSkill
- managing-specsSkill
- marketplace-devSkill
- orchestrating-parallel-workSkill
- reading-session-transcriptsSkill
- receiving-code-reviewSkill
- requesting-code-reviewSkill
- styling-with-tailwind-shadcnSkill
- test-driven-developmentSkill
- upgrading-runtime-dependenciesSkill
- working-in-worktreesSkill
- dorkbot-triageSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 3 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 13
It is debt, not a fix: Every entry expires after 7 days, at most 10 can exist at once, and adding one writes a proposed ledger entry that …
- Distinctive: line 72
Browser: if a quarantined test does not run — missing from the reports, or collected and then skipped — the build fails.
- Distinctive: line 74
If the list is missing, unreadable, invalid, over the cap, expired, dated in the future, or holds an entry that would outlive the ceiling …
- Distinctive: line 78
Do not quarantine a test that fails deterministically.
- Very specific: line 79
Do not edit quarantine.json by hand or push to ci-steward-data.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
/ci-quarantine
One in four merge-queue builds fails, and most of those failures are not bugs. A build that dies on a flake costs about thirty minutes of queue time for every PR behind it. The quarantine lane takes one named test out of the blocking path while it is being fixed.
A quarantined test still runs, still retries and is still reported. It only loses the power to fail the queue. Nothing is skipped, deleted or filtered out, so the day it stops flaking is visible in the same reports.
It is debt, not a fix. Every entry expires after 7 days, at most 10 can exist at once, and adding one writes a proposed ledger entry that says to fix or delete the test.
Look before you add
git fetch --quiet origin ci-steward-data || true
pnpm ci:flakyIt lists every test that failed and then passed on the same merge-group tree, with counts and dates, and says which qualify. Read the status column before anything else:
- QUALIFIES — flaked on at least two different trees, recently enough that nothing suggests it was fixed. This is the only thing that may be added.
- COOLING — it flaked often and has now been quiet for longer than its own gap between flakes. That is the shape of a test somebody fixed; quarantining it spends a slot on nothing. The add path refuses it.
- BELOW-THRESHOLD — one bad runner. Not a classification.
Right now, --fetch is the only path that returns anything. pnpm ci:flaky reads per-test flake names out of the daily collector's snapshots, and the collector only starts recording them with this change — so until it has run for about a week, the plain command correctly reports nothing. --fetch reads the Actions artifacts directly instead (a few hundred API requests, several minutes); artifacts expire after seven days, so that window is what it can see.
--json saves the classification, and quarantine add --evidence <that file> reuses it instead of downloading everything twice.
Copy the file and title, do not retype them. They are matched byte for byte, so a title that differs by one character — or by Unicode normal form — silently matches nothing. That fails safe (the entry excuses nothing, and the fan-in names it as a quarantined test that did not run), but it wastes a slot.
Add one
pnpm ci:quarantine add \
--runner playwright \
--file 'rooms/canvas/room-follow.spec.ts' \
--title 'moves this window onto the document the person you follow is on' \
--by 'your name' \
--reason 'what races, and what the fix would be'Copy --file and --title exactly as pnpm ci:flaky prints them; they are the identity the queue matches on. Nothing is published without --publish — a run without it prints the list that would be written and changes nothing.
It refuses a test with no flaky evidence, and it is right to. A test that fails every time is a real bug. Fix it, revert what broke it, or delete the test; the lane must never be the reason a break reached main.
With --publish it writes quarantine.json on the ci-steward-data branch and the lane is live on the next queue build. No PR, no merge, no review — which is why every guard is enforced when the list is read, and why the reason field is worth writing properly.
It also writes ci/ledger/<id>-quarantine-*.md, the entry that says to fix or delete the test — after the list goes out, so a failed publish never leaves debt recorded against a quarantine that does not exist. Commit it with your next PR.
Pass --ledger <id> to reuse an entry that already exists instead of scaffolding another: re-quarantining a test after a fix attempt belongs on the same debt item, and so does a real --publish after a dry run that already wrote one.
Take one out
pnpm ci:quarantine list
pnpm ci:quarantine remove --runner playwright --file '<file>' --title '<title>' --publishIf the list is refused — one entry over the cap, one bad timestamp — you are not locked out of it. remove still works, because a refused list still parses, and that is the repair. If it does not parse at all, pnpm ci:quarantine reset --publish replaces it with an empty one. Never hand-edit the data branch.
Release a test as soon as it is fixed. pnpm ci:status shows the lane and flags anything about to expire; a quarantined test that passed in a queue build is named in that build's job summary as a candidate to release.
What the queue does with it
- Each shard runs the suite, captures its exit code, and a gate asks whether every failure in the report is quarantined. One that is not fails the shard exactly as before.
- The gate reads three numbers out of the same reports: the runner's own failed-test count, its own per-test walk, and the failures that belong to no test — a file that throws on import never becomes a failed test, so it is counted separately and can never be excused. A non-zero exit is excused only when the count is exactly the set the lane absorbed, so a crash, a dead server or a timeout does not ride out on a quarantined flake from the same run.
- The one hole, named: a package whose process dies before writing any report at all is invisible to all three, and nothing downstream catches it. It is written up in
gateSuite's header and incontributing/ci.md. - The whole lane is printed in the job summary of every queue build that read it.
- Browser: if a quarantined test does not run — missing from the reports, or collected and then skipped — the build fails. The lane must never be the reason a test silently stopped running.
- Vitest: the same check is file-level. If a quarantined test's file contributes nothing to any shard, the build fails; a quarantined vitest test that stops running while its file still collects other tests is not caught. Prefer a browser quarantine when you have the choice.
- If the list is missing, unreadable, invalid, over the cap, expired, dated in the future, or holds an entry that would outlive the ceiling measured from now, it is ignored entirely and every test blocks as normal.
Do not
- Do not quarantine a test that fails deterministically. That is a bug report, not a lane entry.
- Do not edit
quarantine.jsonby hand or push toci-steward-data; use this command. - Do not widen the cap or the expiry to fit more in. Both are thresholds in
ci/config.yamlwith a ratchet behind them, and a wider lane is a bigger blind spot. An entry that gives itself more than the ceiling does not get more time — every reader refuses the whole list, and nothing is quarantined at all.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

