agentleFS
khivikhivi/cockpit/.claude/skills/spec-audit/SKILL.md

spec-audit

Skill0 starsChanged 42 days agoMITView on GitHub

Judge whether each claiming test actually asserts its specs/ bullet — the exists-vs-strong gap the coverage gate cannot see. TRIGGER on "audit the spec", "spec audit", "verify covers markers", or re-verifying after a ~rev bump.

7 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

4 rules. 0 are common, 2 are distinctive, and 2 are very specific to this project.

  • 0

    Common

    Most of the 7 edited versions we read keep these.

  • 2

    Distinctive

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Not for this: finding unclaimed bullets or dead markers (the gate fails those), untested source lines (/coverage-audit), or writing the …

    line 17
  2. Distinctive:

    Read the bullet first, and before opening a test, write one line naming what a test would have to construct for the claim to be falsifiable.

    line 87
  3. Distinctive:

    Read the whole test function under each marker — the fixtures and helpers it leans on too, when the assertion's strength lives there.

    line 92
  4. Very specific:

    Run scope.py --style with the same scope to lint each in-scope bullet against the ASD-STE100 subset docs/specs.md adopts: one fact per …

    line 123

“Common” is measured: kept by at least 6 in 10 of the 7 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

spec-audit

Answers does this test prove its bullet. tests/test_invariant_coverage.py proves every unwaived bullet is claimed; it cannot see whether the claiming test would fail if the claim were false — the same author writes the bullet and the marker, so the two errors correlate rather than cancel. This skill is the third party.

Not for this: finding unclaimed bullets or dead markers (the gate fails those), untested source lines (/coverage-audit), or writing the spec (it is hand-owned).

Arguments

You typeScopeShape of the run
/spec-auditthis worktree's branch vs origin/maininline, usually a handful of ids
/spec-audit allevery unwaived bullet in the ledgerfan out — see below
/spec-audit <PR#>that PR's diffinline
/spec-audit <id>~1one bullet, however scopedinline

Anything else in the argument is an instruction about how to run, not what to audit (all - use subagents). A bare /spec-audit on a branch that touched no spec or test file is a legitimate empty run — say so rather than silently widening to all, which is a hundred-and-twenty-seven-bullet read nobody asked for.

Scope

scope.py resolves it and sizes the ledger in one call. Start every run here:

python3 .claude/skills/spec-audit/scope.py          # branch, vs origin/main
python3 .claude/skills/spec-audit/scope.py all      # every unwaived bullet
python3 .claude/skills/spec-audit/scope.py --pr 539  # a PR's diff
python3 .claude/skills/spec-audit/scope.py --ids-only # just the ids, to iterate

It prints per-file bullet / waived / auditable counts and a total, then the in-scope ids with how many tests claim each — so a bullet claimed by three tests is flagged as a union to judge before you open a single file. Report the totals you audited against; they are the denominator a reader needs to know whether "all entailed" covered nine bullets or a hundred and twenty-seven.

Do not count bullets with rg -c. A bullet's text wraps, so its id and its (untested: …) waiver sit on different lines: rg -c '\(untested:' counts LINES and silently reports a different number than scope.py for the same ledger. That is what the script exists to stop.

A changed spec file scopes to the ids of its edited bullets; a changed test file scopes to the ids its markers claim. Both are resolved from ADDED lines only, never a hunk's declared range — gh pr diff carries three lines of context, so a range would pull in whatever marker happens to sit beside an edit. Skip (untested: …) bullets either way; nothing runnable to judge.

The script sizes the run; it does not perform it. It cannot judge a pair — that is reading a bullet against every test claiming it and deciding whether the test would fail if the claim were false, which is the one thing here that is not mechanical. scope.py exists so the denominator stops being guessed, not to shrink the reading.

all is ~130 bullets — fan it out. One pass cannot hold that many pairs honestly; it degrades into skimming, which reports "all entailed" for a ledger nobody read. Launch one subagent per spec file (group the small ones to even out the load), give each the rubric and the read-only constraint verbatim, and judge their findings yourself before relaying — a subagent returns a summary, not a fact. A branch or --pr scope is usually small enough to do inline.

Pair each id

scope.py already told you how many tests claim each id. The mapping itself is two greps (the same pair AGENTS.md documents):

rg -n '<id>~' specs/    # the bullet
rg -n '<id>~' tests/    # every claiming marker

Read the bullet first, and before opening a test, write one line naming what a test would have to construct for the claim to be falsifiable. Judge against that line. Opening the test first anchors the reading on whatever it happens to assert, and the bullet then reads as satisfied whatever it says.

Read the whole test function under each marker — the fixtures and helpers it leans on too, when the assertion's strength lives there. A bullet claimed by several tests is judged against their union: the tests together may entail what no single one does.

The rubric

One question per pair: would this test fail if the bullet's claim were false? Concretely:

  • The bullet's given/when scenario is the one the test constructs — not a narrower cousin that happens to share the assertion.
  • The assertion checks the claimed behavior, not just the shape around it (a call was made ≠ the call did what the bullet says).
  • It cannot pass vacuously: the test proves its own setup took effect the way the restamp test's "this isn't vacuously true" asserts do.
  • For a ban bullet, the needle/AST check actually catches the violation shape the bullet describes, not only the spelling the author imagined.
  • Length and scaffolding are not strength. A long test with elaborate fixtures and a confident name is the adversarial case, not the safe one — judge the assertion, not the setup around it.

Verdicts: entailed, partial (name exactly what goes unasserted), mismatch (the test asserts something else — the marker is vouching for the wrong bullet), or unassertable (the bullet is the defect: it claims something no test could falsify as worded, or two things at once). Report an unassertable bullet and stop — do not reword it and do not propose a waiver, both being the author's call, and an under-asserted bullet is not this verdict.

Style pass (--style)

Run scope.py --style with the same scope to lint each in-scope bullet against the ASD-STE100 subset docs/specs.md adopts: one fact per sentence, at most 25 words, no rationale inside the promise. CI already fails a bullet the script would flag, so on a green branch this pass is for a draft not yet committed. The script flags; it cannot rewrite. For each flagged bullet, propose a rewrite that keeps every clause of the claim and drops only the reason (point at where the reason belongs, usually AGENTS.md). A rewrite that keeps the meaning keeps its ~rev. If the bullet cannot be split without changing what it claims, report that and stop.

Style is a separate section of the report and never a pair verdict. A well-written bullet can still be unassertable, and a 40-word one can still be entailed.

Never weaken

Propose, never edit. Specifically banned, because "make the audit clean" is the adversarial reading of this skill:

  • Do not reword a bullet down to what the test happens to assert — the spec demands tests, never the reverse.
  • Do not bump a ~rev or touch a marker; both are the author's re-verify act, not the auditor's.
  • Do not propose a waiver for a bullet that is merely under-asserted.
  • Do not replace the reading with a confidence score over the pairs, fanned out or otherwise. A cascade needs a threshold validated on labelled pairs of the kind it will meet; there are none here, and self-reported confidence from the judging model is not the third party this skill exists to be.

Report

Mismatches first, then unassertable bullets, then partials with the missing assertion sketched as a snippet, entailed as a bare count. Open with scope.py's denominator — 130 auditable, 5 in scope — since "all entailed" means nothing without it. Cap findings at five and give the remainder as a number. When every pair holds, say "all entailed" — promoting a nit to fill the section is how the audit stops carrying signal. Style findings come last, under their own count, capped at five like the rest.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.