agentleFS
ClickHouseClickHouse/loom/.claude/skills/loom-investigate-ci/SKILL.md

loom-investigate-ci

Skill87 starsChanged 5 days agoApache-2.0View on GitHub

Takes a failing Loom CI check to its cause and fix — the failed log, whether main fails too, a local reproduction. Use when a PR check is red, a PR shows no checks, a job looks flaky, or when asked why CI fails. Not for bugs found outside CI (loom-debug).

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    No checks at all: the PR is stacked on a branch other than main or a release/X.Y line (retarget it), has a merge conflict, or the push was …

    line 18

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Investigating Loom CI

1. Which job, which line

gh pr view <n> --json statusCheckRollup --jq '.statusCheckRollup[] | select((.conclusion // .state) | IN("SUCCESS", "SKIPPED", "NEUTRAL") | not) | "\((.conclusion // "") as $c | if $c == "" then (.state // .status) else $c end)\t\(.name // .context)\t\(.detailsUrl // .targetUrl)"'
gh run view <run-id> --log-failed | tail -200

No checks at all: the PR is stacked on a branch other than main or a release/X.Y line (retarget it), has a merge conflict, or the push was a no-op.

2. Is it yours?

gh run list --workflow ci.yml --branch main -L 5 --json conclusion,headSha,createdAt
  • main red on the same job: the cause is on main. Run the failing test on a clean origin/main worktree before touching your branch. A test with a literal date in its fixture and a wall-clock staleness check fails the first day the date ages out; the fix is the clock seam, not a new date.
  • main green, your branch red: your change, or your change combined with something that merged since your base — rebase on origin/main and re-run locally.

3. By job

JobUsual causeReproduce / fix
the ClickHouse release this run tests (ci.yml, docker-smoke.yml)the release lookup in dev/clickhouse_release.sh or the registry it readsrerun once; if it fails again, run the script locally and read its error
Quickstart (<arch>) (docker-smoke.yml, on PRs and pushes to main or a release/X.Y line touching its paths:, weekly and on dispatch)the image or its bundle, the quickstart stack, then the one-line installer in clean containers (scripts/gates/installer_rehearsal.py; its FAIL lines name the distribution they ran on); on a weekly or dispatched run, a scan step: a critical CVE with a fixed release in the imagethe dashboard test dependencies CI installs first (npm --prefix frontend ci && npx --prefix frontend playwright install chromium, after loom-dev-setup §6's link check), docker build -t "loom:smoke-$(basename "$PWD")" ., then python3 scripts/gates/docker_quickstart_smoke.py --image "loom:smoke-$(basename "$PWD")" --clickhouse-image <image> (a tag per worktree: two sessions share the image store; docker rmi it afterwards), with the image from bash dev/clickhouse_release.sh. The rehearsal, with the tags fetched and Docker on /var/run/docker.sock: python3 scripts/release/build_stack_bundle.py build --version <server __version__> --clickhouse-image <image> --output-dir <dir>, then python3 scripts/gates/installer_rehearsal.py --release-dir <dir> --upgrade --image <server image> --extractor-image <extractor image>. A scan: build with docker build --no-cache (a cached apt layer keeps a fixed package), then trivy image --severity CRITICAL --ignore-unfixed on it names the package; move the pin that brings it in (a Dockerfile ARG, the base-image digest for a package the base image carries, or a lock file)
Repository-job sandbox under the shipped profiles (docker-smoke.yml)the AppArmor or seccomp profile, or a sandbox change the profile blocksthe profiles row of loom-pre-push §3
ruff + mypy-gate (lint)a generator --check is stale (the plugin sync also compares executable bits); ruff; function-length or mypy ratchetmake lint, make typecheck; run the named generator without --check and commit its output; split the long function
pytest unit part korder dependence; two tests sharing a file or port while they run side by side; a fake that drifted; a snapshot; a migration number main took since your base (renumber per the migrations rule)the reproduction in tests/README.md, with the part's LOOM_TEST_SHARD=k/6 on the commands that name tests/unit; update a snapshot only for an intended change
pytest unit coverage, floors and benchthe combined coverage under 40% or a per-module coverage floor; a bench harness; integrations/generic/test.shthe coverage-floor command in loom-pre-push §3; pytest bench -q; PYTHON=.venv/bin/python bash integrations/generic/test.sh
repository-job sandbox (real bubblewrap)a change to the repository-job sandbox or the files it runsthe sandbox-suite row of loom-pre-push §3
schema.sql in sync with migrationsschemas/schema.sql dumped from another ClickHouse release, or not regeneratedstart a private ClickHouse on the release the run names (job "the ClickHouse release this run tests"): LOOM_CLICKHOUSE_IMAGE=clickhouse/clickhouse-server:<ver> plus the name and ports of loom-dev-setup §3, bash dev/ci_start_clickhouse.sh; migrate; make schema-dump
builds N-1 and N on one ClickHouse (mixed-version)N writes something N-1 cannot read, or a lease or bundle rule brokenmake test-mixed-version REQUIRE=1 (docker); rolling upgrades names the rule
formatan unformatted file; a dashboard accessibility rule (step "biome lint (dashboard accessibility)")make format from the repository root (the pinned Biome reads biome.json there); make lint for the accessibility rules, fixed in the TSX (dashboard design)
helm chart (lint, kubeconform)a template or value that does not render, or a manifest that fails the schemathe deploy/helm/ row of loom-pre-push §3
coding-agent pluginsa hook, grep-shim or Cursor test; a syntax error in a shipped .mjs; a non-executable integrations/grep-shim/bin/ entry pointthe commands of the integrations/ row in loom-pre-push §3, which mirror the job's steps
tsc (frontend)types, bundle budgetnpm --prefix frontend run build && npm --prefix frontend run budget
frontend vitest part ka vitest or axe failurenpm --prefix frontend run test -- --shard=k/n, n being the parts in the job's matrix in ci.yml, or npm --prefix frontend run test for all of it
frontend playwright part kan e2e spec failurenpx --prefix frontend playwright install chromium once (its system libraries, playwright install-deps, only on a machine you own), then E2E_PORT=<free port> npm --prefix frontend run e2e -- --shard=k/n, n as above; the part's traces are the frontend-e2e-k artifact
typescript sdk (...)generated types stale, lint, a vitest failure, the ESM/CJS dist build, publint or attw.venv/bin/python -m scripts.codegen.sdk_ts_gen, then npm --prefix sdk/typescript run check && npm --prefix sdk/typescript run check:dist
python sdk wheel (...)an undeclared dependency or a 3.12-only API (3.11 legs), a pin floor lower than the code needs (lowest-deps legs), a new release of a dependency (the other legs)the sdk/python/ row of loom-pre-push §3, on the leg's Python and with --resolution highest unless the job name says lowest deps
pytest integration part k of 4, pytest integration isolation gatesbehaviour change; a test that needs an env CI lacks (self-signup)the failing file, or the part (LOOM_TEST_SHARD=k/4), against a private server (loom-dev-setup); the isolation gates part runs the provider staging tests (LOOM_PROVIDER_STAGING_LIVE_TEST=1, on a freshly migrated database), the auth-on gates (reproduce against a server with LOOM_AUTH_REQUIRED=true) and the analyzer corpus; the failure report lists the last requests with X-Request-Id to find the server log line
pytest integration (Postgres n memory authority)a read or write path that skips the Postgres authority; a head and its ClickHouse copy out of stepLOOM_PG_IMAGE=postgres:n bash dev/ci_start_postgres.sh, then the failing file against a private server with LOOM_STORAGE_AUTHORITY=postgres and LOOM_PG_DSN set as the job sets them

4. Flaky or real

A failure is flaky only when the same commit passes on re-run and you can name the race (timing, order, a shared resource). Re-run once with gh run rerun <run-id> --failed; a second failure is real. Name the cause in the PR even when it is flaky, and fix the test rather than adding retries.

5. Report

The failing job and line, the root cause, whether main is affected, the fix (or the commit that fixes it), and the local command that now passes.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.