agentleFS
opsmillopsmill/infrahub/.agents/skills/analyzing-ci-flakiness/SKILL.md

analyzing-ci-flakiness

Skill548 starsChanged 11 days agoApache-2.0View on GitHub

Analyzes recent CI failures on pull requests to identify flaky tests, using retry outcomes (failed attempt → green re-run) and cross-PR recurrence as evidence, and maintains a local longitudinal ledger so flakiness can be tracked over time. TRIGGER when: the user wants to find flaky tests, correlate recent CI failures, check which tests fail across PRs or recover on retry, or refresh the flakiness trend report. DO NOT TRIGGER when: babysitting a single PR's CI until green → monitoring-pull-requests; diagnosing or fixing one specific failing test → the bug-analysis skills.

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run the bundled collector (repo-root relative):

    line 43
  2. Very specific:

    Write ANALYSIS.md into the window directory, then give the user a summary.

    line 117
  3. Distinctive:

    Do not propose fixes unless asked.

    line 129

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

CI Flakiness Analyzer

Introduction

A test is flaky when its failure does not reproduce on the same code: the run was retried and went green, or the same test fails on unrelated PRs. This skill mines both signals from GitHub Actions history, downloads the failed job logs once into a local cache, and appends every observation to a ledger (~/ci-cache/<owner>-<name>/ledger.jsonl) so repeated invocations — weekly, or ad hoc — accumulate trend data instead of starting from scratch.

The mechanical part (fetching, caching, test-name extraction, known-signature classification) is done by the bundled script. Your job is the judgment part: separating flakes from real regressions, spotting new systemic signatures, and writing the report.

Step 1 — Parse arguments

  • Base-branch filter: any arguments that look like branch names or globs (release-1.11, release-*, stable). Default: no filter (all PR bases), which is usually what "how flaky is CI" means. Filter when the user names a branch.
  • Window: a bare integer is a number of days (default 7). An ISO date means "since that date".

Step 2 — Collect

Run the bundled collector (repo-root relative):

python3 .agents/skills/analyzing-ci-flakiness/scripts/collect.py \
  [--base <glob> ...] [--days N | --since YYYY-MM-DD] [--repo owner/name]

It prints a JSON report to stdout and writes everything under ~/ci-cache/<owner>-<name>/windows/<since>_<until>/:

  • runs.jsonl — every pull_request workflow run created in the window
  • failed_jobs_with_tests.json — failed jobs of the interesting run-attempts, with extracted failing tests, systemic-bucket tags, and a recovered_same_run flag
  • report-data.json — headline numbers, ranked per-test table, per-bucket incident counts (bucket_incidents: distinct jobs/runs/PRs per systemic bucket), and the ledger's weekly history
  • joblogs/<job_id>.log — raw logs (ANSI intact; strip with sed 's/\x1b\[[0-9;]*m//g')

Notes the script already accounts for — don't re-derive them:

  • The runs API's pull_requests field is empty for many runs; the script joins runs to PRs through every PR head commit SHA as well. Don't trust the field alone.
  • "Interesting attempts" = every earlier attempt of a retried run (that's what the retry fixed) plus final attempts that failed. Runs cancelled on attempt 1 are concurrency noise and skipped.
  • Logs already on disk are never re-downloaded; the ledger is deduplicated by (job, test). Old logs expire on GitHub's side (~90 days) — an empty joblogs/*.log means expired, not passing.

Step 3 — Investigate what the script could not name

For failed jobs with an empty tests list and no bucket tag, read the log yourself (grep for ##[error], FAILED, Error:, Timeout). Two outcomes:

  • It matches a new systemic signature (infra failure that cascades over many tests). Add a regex for it to BUCKETS in collect.py and to the table below, so future runs classify it.
  • It's a genuine test failure the extraction regexes missed — note the test manually and consider extending extract_tests.

Known systemic signatures (as of 2026-08 — keep in sync with BUCKETS in collect.py)

BucketSignatureMeaning
stack-readinessServerNotResponsiveError … /api/schema/loadSeeded testcontainers stack not ready; the whole pytest-playwright shard errors. One incident, not N flaky tests.
vitest-mock-corruptionTypeError: vi.mocked(...).mockX is not a functionvitest browser-mode module-mocking race; hits a different test file each time.
prefect-setup-triggers-timeoutSetup triggers task ReadTimeoutPrefect hang at session setup; downstream tests hit their own timeouts.
neo4j-deadlockNeo.TransientError.Transaction.DeadlockDetectedConcurrent-write deadlock, usually integration suites under xdist.
compose-boot-failuredocker compose … up --wait non-zero exitStack never booted; job-level infra failure.
sqlite-locked(sqlite3.OperationalError) database is locked (also matches the raw sqlite3.OperationalError: form)Prefect's sqlite under contention.
runner-oomProcess completed with exit code 137Runner OOM/SIGKILL; the mass test failures in the same job are casualties, not flakes.
docker-network-pool-exhaustedall predefined address pools have been fully subnettedLeaked compose networks exhausted the docker address pools on a self-hosted runner.
actions-download-429Failed to download action … 429GitHub rate-limited its own action download; pure platform flake.
prefect-task-manager-wedgedRuntimeError: Prefect task manager setup already failed for http…The memoized task-manager setup (backend/tests/helpers/task_manager.py) timed out once against a Prefect test server; every later class fail-fasts on the remembered failure. One incident, hundreds of cascaded ERRORs.
pytest-green-exit-1green pytest summary (no failed, no errors) directly followed by exit 1Session-teardown/plugin abort after all tests passed (e.g. testcontainers result reporting).

Step 4 — Judge: flake vs regression

For each test in the ranked table, classify:

  • Flaky (strong) — fails on ≥2 unrelated PRs, or recovered_on_retry > 0. The more distinct PRs, the stronger.
  • Flaky (weak) — single occurrence with an infra-flavored error (locator timeout, transient branch not found) and the PR later went green. List, but rank low.
  • Suspect regression, not a flake — the same test fails on every attempt of the same commit and the PR is still red, or the failures started only after a specific merge. Say so explicitly; do not bury it in the flake list. Cross-check: does the test fail on any PR that does not contain the suspect change?
  • Systemic bucket — tests whose only failures carry a bucket tag are casualties, not causes. Report the bucket (with the incident count from bucket_incidents in report-data.json), not the individual tests.

Different tests failing on successive attempts of the same run = two independent flakes, not a regression.

Step 5 — Report

Write ANALYSIS.md into the window directory, then give the user a summary. Lead with the ranked flake candidates. Include:

  1. Headline numbers: PRs in scope, runs matched, retried runs, retried-and-recovered runs (pure-flake evidence), hard failures.
  2. Ranked flake candidates — test id, distinct PRs/runs, recovered-on-retry count, one-line error cause. Group systemic buckets as single entries.
  3. Suspected real regressions, clearly separated.
  4. Trend — from weekly_history in report-data.json: which offenders are new this window, which recur week over week, which disappeared (likely fixed). This section is the reason the ledger exists; don't skip it once ≥2 windows of data exist.

Do not propose fixes unless asked; the deliverable is the evidence-ranked candidate list.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.