flaky-triage
Skill0 starsChanged 5 months agoMITView on GitHub
When a test goes red then green with no intervening code change, confirm it's flaky, record it in the registry, and optionally quarantine it so it stops blocking PRs but stays visible — instead of silently blocking the queue or being silently ignored
1 other project uses a version of this.
Adopted by
Most recently taken up by
- AlpayC/claude-skill-settheir own version, first seen Sep 14, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- AlpayC/claude-skill-set★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository29 other files it gives its agents.
- adopt-existing-projectSkill
- adrSkill
- branch-conflict-checkSkill
- ci-watchSkill
- compound-learningsSkill
- config-auditSkill
- cost-checkSkill
- coverage-ratchetSkill
- doc-syncSkill
- eval-harnessSkill
- followup-trackingSkill
- incident-captureSkill
- kit-doctorSkill
- kit-updateSkill
- logging-initSkill
- migration-checkSkill
- new-project-bootstrapSkill
- nightly-auditSkill
- observability-checkSkill
- parallel-workSkill
- perf-budgetSkill
- pr-babysitterSkill
- pre-prSkill
- prompt-regressionSkill
- releaseSkill
- session-handoffSkill
- sync-healthSkill
- using-the-kitSkill
- writing-kit-skillsSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 1 is distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 43
Run the suspect node flaky.rerun_count times (default 5).
- Very specific: line 72
Use the followup-tracking skill to append an entry to docs/followups.md (resolved from capabilities.docs.dir in kit.yaml, default: docs).
- Distinctive: line 94
If quarantine is allowed and the verdict is Flaky:
- Very specific: line 104
Check that toolchain.test in kit.yaml includes -m 'not <marker>'.
- Very specific: line 117
If the file does not exist yet, create it from docs/flaky-tests.template.md.
- Very specific: line 125
Quarantine is a loan, not forgiveness: Every quarantined test must have an owning follow-up ID in the registry.
- Very specific: line 130
Never quarantine without recording: If quarantine_allowed is true but the registry update fails for any reason, revert the marker change.
- Very specific: line 135
Borderline tests are recorded, not quarantined: If a borderline test (pass rate ≥ flaky_threshold) surfaces again on a subsequent run, the …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Flaky Test Triage
Profile-driven. This skill reads flaky.* keys from .claude/kit.yaml (listed in Profile keys below) and toolchain.test for the base runner command.
A test that passes on re-run with no code change is not noise — it is a latent reliability bug. This skill confirms the flakiness with a systematic re-run, files a durable follow-up so the finding is not lost, and optionally quarantines the test so it stops burning CI time while remaining visible to nightly-audit. Triggered by ci-watch (red→green, no change) or by a human who notices the same pattern.
Profile keys
| Key | Default | Meaning |
flaky.rerun_count | 5 | Number of times to re-run the suspect test node |
flaky.flaky_threshold | 0.8 | Pass-rate ceiling for a flaky verdict (exclusive); range 0.0–1.0 |
flaky.marker | "flaky" | Pytest marker name applied when quarantining |
flaky.quarantine_allowed | false | When false, skip quarantine and only record |
flaky.registry | "docs/flaky-tests.md" | Registry file path relative to the repo root |
toolchain.test | "uv run pytest tests/unit/ …" | Base runner; substitute <nodeid> for the path glob when re-running |
Workflow
1. Identify the suspect node
The caller (ci-watch or a human) provides the test node ID — a string like
tests/unit/test_scheduler.py::TestRetry::test_timeout. If it is not known, grep
the CI failure log for the FAILED line:
# From `gh run view <run_id> --log-failed` output:
grep -E "^FAILED " ci-failed.log2. Re-run and record the pass rate
Run the suspect node flaky.rerun_count times (default 5). Substitute the node ID
into the runner from the profile:
# kit.yaml → toolchain.test (replace the path glob with <nodeid>)
PASSES=0
RUNS=5 # flaky.rerun_count
for i in $(seq 1 $RUNS); do
uv run pytest <nodeid> -q --no-header 2>/dev/null && PASSES=$((PASSES + 1))
done
echo "Pass rate: $PASSES / $RUNS"Compute pass rate = passes ÷ runs.
3. Render a verdict
| Pass rate | Verdict | Next step |
0 (never passes) | Real failure — not flaky | Stop. Report to the caller as a real failure; do not file a flaky entry. |
> 0 and < flaky_threshold | Flaky | Proceed to step 4. |
≥ flaky_threshold | Mostly-passes — borderline | Proceed to step 4 with a borderline note; record but do not quarantine. |
flaky_threshold defaults to 0.8. A test that passes 4 of 5 runs (0.8) is
borderline — record it and watch it; do not quarantine yet. A test that passes 2 of
5 runs (0.4) is clearly intermittent.
4. File a follow-up
Use the followup-tracking skill to append an entry to docs/followups.md
(resolved from capabilities.docs.dir in kit.yaml, default: docs). Assign a
stable ID with the prefix FLAKY- and a short slug derived from the node id
(e.g. FLAKY-scheduler-timeout):
### FLAKY-<slug> — <test short name> is intermittent
- **Spec:** `flaky.registry` registry entry; owner: <assignee or unassigned>
- **Status:** open
- **Gap:** `<nodeid>` passed <N>/<M> re-runs (pass rate <P%>). No code change
between the red and green runs. Needs root-cause investigation and a fix or
explicit deletion. Quarantine status: <quarantined | not quarantined>.Note the follow-up ID — you will write it into the registry in step 6.
5. Quarantine (optional)
Skip this step entirely if flaky.quarantine_allowed is false (default), or if
the verdict from step 3 is Mostly-passes (borderline).
If quarantine is allowed and the verdict is Flaky:
a. Add the marker to the test source.
Open the test file and add @pytest.mark.<marker> (default: @pytest.mark.flaky)
directly above the test function or class. If the marker is not registered in
conftest.py or pyproject.toml, add the registration now.
b. Verify the marker is excluded from the gating run.
Check that toolchain.test in kit.yaml includes -m 'not <marker>'. If it does
not, add the exclusion now and note the change in your output.
c. Confirm the test still runs somewhere.
The marker must be included in at least one non-gating run — for example, a nightly
suite or a manual pytest -m <marker> invocation. If no such run exists, add a
follow-up item noting this gap: a quarantined test that never runs is a deleted test
with extra steps.
6. Update the registry
Append a row to the registry file (flaky.registry, default docs/flaky-tests.md).
If the file does not exist yet, create it from docs/flaky-tests.template.md.
| Test node id | Pass rate | First seen | Quarantined? | Follow-up ID | Owner |
<nodeid> | <N>/<M> (<P%>) | <YYYY-MM-DD> | Yes / No | FLAKY-<slug> | <assignee or —> |
Guardrails
- Quarantine is a loan, not forgiveness. Every quarantined test must have an
owning follow-up ID in the registry.
nightly-auditsurfaces the total quarantined count and the age of each entry so the list cannot grow unnoticed. A quarantined test that still has an open follow-up after 30 days should be escalated bynightly-audit, not quietly carried forever. - Never quarantine without recording. If
quarantine_allowedis true but the registry update fails for any reason, revert the marker change. An unrecorded quarantine is worse than no quarantine — it is a silently deleted test. - Do not quarantine a real failure — pass rate 0 means broken, not flaky (see step 3).
- Borderline tests are recorded, not quarantined. If a borderline test (pass
rate ≥
flaky_threshold) surfaces again on a subsequent run, the accumulated evidence justifies reopening triage and potentially quarantining then. - Never widen
flaky_thresholdretroactively to reclassify a failing test as flaky. The threshold is a project-wide policy inkit.yaml, not a per-test escape hatch.
Output
## Flaky Triage
- Test node: <nodeid>
- Reruns: <N> (flaky.rerun_count)
- Pass rate: <passes>/<N> (<P%>)
- Verdict: Flaky | Mostly-passes (borderline) | Real failure (not flaky)
- Follow-up filed: <FLAKY-slug> | none (real failure)
- Quarantine: applied (<marker> marker added) | skipped (quarantine_allowed=false) | skipped (borderline) | n/a (real failure)
- Registry updated: yes (<flaky.registry>) | noDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

