spec-audit
Skill0 starsChanged 42 days agoMITView on GitHub
Judge whether each claiming test actually asserts its specs/ bullet — the exists-vs-strong gap the coverage gate cannot see. TRIGGER on "audit the spec", "spec audit", "verify covers markers", or re-verifying after a ~rev bump.
7 other projects use a version of this.
Adopted by
Most recently taken up by
- veraetogrex-portfolio/solo-sddtheir own version, first seen Nov 15, 2026
- vlad-ryzhkov/ai-context-engineering-for-frontendtheir own version, first seen Nov 15, 2026
- grooownow/qa-pilottheir own version, first seen Nov 1, 2026
- Fadi88/one-man-startuptheir own version, first seen Oct 15, 2026
- pierre-lecocq/claude-skillstheir own version, first seen Oct 11, 2026
- jsulmont/mcp-lisptheir own version, first seen Jun 27, 2026
Other versions7 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 7
- Fadi88/one-man-startup★ 16same name
- jsulmont/mcp-lisp★ 9same name
- grooownow/qa-pilot★ 6same name
- vlad-ryzhkov/ai-context-engineering-for-frontend★ 2same name
- caseyng/Claude-skills★ 1same name
- pierre-lecocq/claude-skills★ 0same name
- veraetogrex-portfolio/solo-sdd★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository4 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- cockpit-devSkill
- coverage-auditSkill
The file
We split this skill into the rules it gives an agent
4 rules. 0 are common, 2 are distinctive, and 2 are very specific to this project.
0
Common
Most of the 7 edited versions we read keep these.
2
Distinctive
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 17
Not for this: finding unclaimed bullets or dead markers (the gate fails those), untested source lines (/coverage-audit), or writing the …
- Distinctive: line 87
Read the bullet first, and before opening a test, write one line naming what a test would have to construct for the claim to be falsifiable.
- Distinctive: line 92
Read the whole test function under each marker — the fixtures and helpers it leans on too, when the assertion's strength lives there.
- Very specific: line 123
Run scope.py --style with the same scope to lint each in-scope bullet against the ASD-STE100 subset docs/specs.md adopts: one fact per …
“Common” is measured: kept by at least 6 in 10 of the 7 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
spec-audit
Answers does this test prove its bullet. tests/test_invariant_coverage.py
proves every unwaived bullet is claimed; it cannot see whether the claiming
test would fail if the claim were false — the same author writes the bullet and
the marker, so the two errors correlate rather than cancel. This skill is the
third party.
Not for this: finding unclaimed bullets or dead markers (the gate fails those),
untested source lines (/coverage-audit), or writing the spec (it is
hand-owned).
Arguments
| You type | Scope | Shape of the run |
/spec-audit | this worktree's branch vs origin/main | inline, usually a handful of ids |
/spec-audit all | every unwaived bullet in the ledger | fan out — see below |
/spec-audit <PR#> | that PR's diff | inline |
/spec-audit <id>~1 | one bullet, however scoped | inline |
Anything else in the argument is an instruction about how to run, not what to
audit (all - use subagents). A bare /spec-audit on a branch that touched no
spec or test file is a legitimate empty run — say so rather than silently
widening to all, which is a hundred-and-twenty-seven-bullet read nobody asked
for.
Scope
scope.py resolves it and sizes the ledger in one call. Start every run here:
python3 .claude/skills/spec-audit/scope.py # branch, vs origin/main
python3 .claude/skills/spec-audit/scope.py all # every unwaived bullet
python3 .claude/skills/spec-audit/scope.py --pr 539 # a PR's diff
python3 .claude/skills/spec-audit/scope.py --ids-only # just the ids, to iterateIt prints per-file bullet / waived / auditable counts and a total, then the in-scope ids with how many tests claim each — so a bullet claimed by three tests is flagged as a union to judge before you open a single file. Report the totals you audited against; they are the denominator a reader needs to know whether "all entailed" covered nine bullets or a hundred and twenty-seven.
Do not count bullets with rg -c. A bullet's text wraps, so its id and its
(untested: …) waiver sit on different lines: rg -c '\(untested:' counts
LINES and silently reports a different number than scope.py for the same
ledger. That is what the script exists to stop.
A changed spec file scopes to the ids of its edited bullets; a changed test
file scopes to the ids its markers claim. Both are resolved from ADDED lines
only, never a hunk's declared range — gh pr diff carries three lines of
context, so a range would pull in whatever marker happens to sit beside an
edit. Skip (untested: …) bullets either way; nothing runnable to judge.
The script sizes the run; it does not perform it. It cannot judge a pair —
that is reading a bullet against every test claiming it and deciding whether
the test would fail if the claim were false, which is the one thing here that
is not mechanical. scope.py exists so the denominator stops being guessed,
not to shrink the reading.
all is ~130 bullets — fan it out. One pass cannot hold that many pairs
honestly; it degrades into skimming, which reports "all entailed" for a ledger
nobody read. Launch one subagent per spec file (group the small ones to even
out the load), give each the rubric and the read-only constraint verbatim, and
judge their findings yourself before relaying — a subagent returns a summary,
not a fact. A branch or --pr scope is usually small enough to do inline.
Pair each id
scope.py already told you how many tests claim each id. The mapping itself is
two greps (the same pair AGENTS.md documents):
rg -n '<id>~' specs/ # the bullet
rg -n '<id>~' tests/ # every claiming markerRead the bullet first, and before opening a test, write one line naming what a test would have to construct for the claim to be falsifiable. Judge against that line. Opening the test first anchors the reading on whatever it happens to assert, and the bullet then reads as satisfied whatever it says.
Read the whole test function under each marker — the fixtures and helpers it leans on too, when the assertion's strength lives there. A bullet claimed by several tests is judged against their union: the tests together may entail what no single one does.
The rubric
One question per pair: would this test fail if the bullet's claim were false? Concretely:
- The bullet's given/when scenario is the one the test constructs — not a narrower cousin that happens to share the assertion.
- The assertion checks the claimed behavior, not just the shape around it (a call was made ≠ the call did what the bullet says).
- It cannot pass vacuously: the test proves its own setup took effect the way the restamp test's "this isn't vacuously true" asserts do.
- For a ban bullet, the needle/AST check actually catches the violation shape the bullet describes, not only the spelling the author imagined.
- Length and scaffolding are not strength. A long test with elaborate fixtures and a confident name is the adversarial case, not the safe one — judge the assertion, not the setup around it.
Verdicts: entailed, partial (name exactly what goes unasserted), mismatch (the test asserts something else — the marker is vouching for the wrong bullet), or unassertable (the bullet is the defect: it claims something no test could falsify as worded, or two things at once). Report an unassertable bullet and stop — do not reword it and do not propose a waiver, both being the author's call, and an under-asserted bullet is not this verdict.
Style pass (--style)
Run scope.py --style with the same scope to lint each in-scope bullet against
the ASD-STE100 subset docs/specs.md adopts: one fact per sentence, at most 25
words, no rationale inside the promise. CI already fails a bullet the script
would flag, so on a green branch this pass is for a draft not yet committed.
The script flags; it cannot rewrite.
For each flagged bullet, propose a rewrite that keeps every clause of the
claim and drops only the reason (point at where the reason belongs, usually
AGENTS.md). A rewrite that keeps the meaning keeps its ~rev. If the bullet
cannot be split without changing what it claims, report that and stop.
Style is a separate section of the report and never a pair verdict. A well-written bullet can still be unassertable, and a 40-word one can still be entailed.
Never weaken
Propose, never edit. Specifically banned, because "make the audit clean" is the adversarial reading of this skill:
- Do not reword a bullet down to what the test happens to assert — the spec demands tests, never the reverse.
- Do not bump a
~revor touch a marker; both are the author's re-verify act, not the auditor's. - Do not propose a waiver for a bullet that is merely under-asserted.
- Do not replace the reading with a confidence score over the pairs, fanned out or otherwise. A cascade needs a threshold validated on labelled pairs of the kind it will meet; there are none here, and self-reported confidence from the judging model is not the third party this skill exists to be.
Report
Mismatches first, then unassertable bullets, then partials with the missing
assertion sketched as a snippet, entailed as a bare count. Open with
scope.py's denominator — 130 auditable, 5 in scope — since "all entailed"
means nothing without it. Cap findings at five and give the remainder as a
number. When every pair holds, say "all entailed" — promoting a nit to fill the
section is how the audit stops carrying signal. Style findings come last,
under their own count, capped at five like the rest.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

