CI failure triage for splice

Input: one or more (GitHub run URL or id, job name, cn-test-failures ref) tuples from the user. A ref is an issue number in the failure tracker DACH-NY/cn-test-failures, readable with a GitHub login that has DACH-NY access. Output: one evidence packet per ref in ci-triage/, a row in ci-triage/README.md, a duplicate verdict, and optionally a fix branch. Everything you write must be reproducible by someone else: every claim is a command plus its verbatim output. Read references/conventions.md once before starting; it overrides defaults.

The skill lives on main; the triage data does not. Packets, ci-triage/README.md, ci-triage/HANDOVER.md and the flake family catalogue ci-triage/known-families.md live on the shared triage branch, which is never merged. If it is not checked out, ask the user which triage branch is current before writing anything.

Procedure

1. Record the tuple verbatim. Never infer which ref belongs to which job by elimination. If a run has more failed jobs than refs, list all failed jobs (gh run view <run> --json jobs) and leave the mapping open until the user states it. If you cannot read the tracker, ask for the run URL rather than guess.
2. Fetch the job log by job id (references/recipes.md section 1). Classify in one pass:
   • Tests: succeeded N, failed 0 + contains problems = checkErrors failure: the only evidence is the flagged log lines. The console masks {} as *** and prints every ignored line with the suffix (ignore this line in check-sbt-output.sh); the real problems are the @timestamp lines WITHOUT it.
   • *** FAILED *** = assertion or command failure: take the failing clue and stack head.
   • no report, job cancelled or exit without summary = evidence loss (JVM exit, timeout); say so.
3. Duplicate check BEFORE deep analysis: compare the flagged line and the shard's suite list against ci-triage/known-families.md on the triage branch. A match still needs the confirming grep listed there (for example the multi-host clue 1-15 s before an ACS mismatch period). Say "duplicate of X" only after that grep.
4. Establish what runs: canton pin from nix/canton-sources.json at the run's sha, never from canton/. For Canton behaviour cite the jar (references/recipes.md section 6), and state the version next to every source citation.
5. Download the artifact gzipped to log/<ref>/<artifact>/ (gitignored), stream with zcat. Build the suite timeline from canton_network_test.clog.gz, then correlate flagged timestamps to it and to the canton log. Chase "what is it waiting on, and why" until the cause is a concrete event with a timestamp; do not stop at the timeout symptom.
6. Decide flake versus real, and where the fix belongs (test, splice app, Canton, infra). For a candidate fix that already exists on main, check git merge-base --is-ancestor <sha> origin/<branch>: a missing backport is the most common resolution for release-line failures.
7. Write the packet from references/packet-template.md: command, then verbatim output, per finding. Add the README row (mapping table, overview table, cross-cutting notes when a pattern spans refs). Add new families or occurrences to ci-triage/known-families.md.
8. Fix, only when it is test-side and small enough to review in one screen: one branch per PR off origin/main (or the release line for a backport, cherry-pick -x -s), in its own worktree so the triage branch stays checked out (references/recipes.md section 8), named <user>/fix-<ref>-<slug> (backports: <user>/backport-<ref>-<pr>-<release-line>), where <user> is your own short git prefix, so the branch carries the cn-test-failures ref; when one branch fixes several refs, use the parent ref and list the others in the README row. Single-subject commit with a CI tag and DCO sign-off, no code comments, no AI attribution. State plainly what was and was not verified. Do not compile or start Canton unless asked; cheap checks only (scalafmtCheck, prettier --check, config greps).
9. Commit the packet on the triage branch with [skip ci] and -s. Report: one line per ref with duplicate verdict and fix state, then the branch tips.

Anti-patterns

• Proposing an ignore pattern for a WARN whose cause is not understood. Ignore patterns hide the only signal on shards where the underlying bug does not wedge (see 10010 in the catalogue).
• Citing canton/ source line numbers: it is a stale rsync copy.
• Treating the suite list of a checkErrors shard as the culprit: the WARN's suite is the one running at the flagged timestamp, and commitment or ack WARNs often land minutes after the causing step.
• Pasting only conclusions. A packet without the commands that produced them is not accepted.