verify-agent-claims
Skill0 starsChanged 50 days agoMITView on GitHub
Checks whether what a coding agent said it did is true — "fixed the bug", "added tests", "all tests pass", "removed dead code", "no behaviour change", "verified" — by splitting its commits, PR description or chat summary into claims and giving each a verdict (holds, doesn't hold, can't tell) with evidence, written for a founder who doesn't read code. Also unsticks an agent that keeps fixing the same thing. It finds which of two look-alike files is live, dead code kept alive by other dead code, and writes the CLAUDE.md rules that stop the loop. Ships a read-only, self-testing script. Use it whenever someone asks to verify, check or audit what Claude, Cursor, Codex or any AI agent did; says "did it really fix it", "can I trust this PR", "it says tests pass", "it keeps breaking the same thing", "it's going in circles"; is about to merge AI-written code they can't read; or inherited an AI-built codebase, even if they never say "verify".
1 other project uses a version of this.
Adopted by
Most recently taken up by
- comkocs/md-workflowtheir own version, first seen Oct 8, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- comkocs/md-workflow★ 3same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repositoryOne other file it gives its agents.
- AGENTS.mdAGENTS.md
The file
Show its rules and text
We split this skill into the rules it gives an agent
13 rules. About 0 are common, 10 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈10
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 53
test-selection config changed in the same range (load_tests, conftest.py, pytest.ini, testPathIgnorePatterns), which drops tests with no …
- Distinctive: line 55
test files deleted, or assertions going down.
- Distinctive: line 94
Tests pass: run the suite and report the count run, not the count written: skipped, todo and excluded files included.
- Distinctive: line 100
No behaviour change: run the same tests at both commits.
- Very specific: line 179
Import aliases are read automatically from every tsconfig.json / jsconfig.json (and tsconfig.*.json): compilerOptions.paths, baseUrl and …
- Distinctive: line 206
add one temporary log line to each copy and exercise the feature once.
- Distinctive: line 209
Remove the temporary log line afterwards.
- Very specific: line 216
Deletion by reference, not by name: delete the dead copy only after confirming nothing imports it through any channel (alias import, …
- Distinctive: line 230
If the repo has the Cairn Memory plugin, the settled question ("which auth is live") also belongs in its closed-questions ledger.
- Distinctive: line 238
Every claim names what changed specifically enough to check.
- Distinctive: line 239
Every file a claim says changed was changed.
- Distinctive: line 240
Every fix has a test that fails without it.
- Distinctive: line 246
No proof word ("verified", "works", "always") without evidence behind it.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Verify agent claims
A coding agent reports its work in sentences: fixed the login bug, added tests, all tests pass, removed dead code, no behaviour change. Each sentence is a claim, and a founder who can't read the diff has no way to tell which ones are true. The failures are rarely lies. They are claims made with no enforcer: a test that never imports the code it's named after, a fix in a file the build doesn't use, a "removal" that left the importer behind, a green suite that got green by skipping the case that failed.
This skill turns the agent's sentences into claims, checks each one against the repository, and reports a verdict per claim in words the founder can act on.
Scope
Covered here: is what the agent said true?, and getting an agent unstuck when it keeps circling the same bug.
Well served elsewhere, so point there rather than redoing them:
| Need | Use |
| A general code-quality review of a PR | Anthropic's pr-review-toolkit, /code-review |
| Security review | security-guidance, /security-review |
| Whether the numbers an AI product shows are real | Cairn Signals (ai-signals-audit) |
| Where rules and memory should live so agents stop drifting | Cairn Memory (memory-architecture) |
Mode 1: verify the claims
1. Collect what was said
Take the claims from wherever the agent made them: commit messages in the range, the PR description, or the agent's final chat summary. Ask for the summary if the user has it. Chat summaries hold the strongest claims ("everything works now") and never reach git.
2. Run the script
python <skill-dir>/scripts/verify_claims.py claims --repo <repo> --range <base>..<head> [--text summary.txt]
python <skill-dir>/scripts/verify_claims.py --self-testIt splits the text into claims and checks each against the diff:
- a file the claim says changed that the range never touched, or that exists nowhere;
- "added tests" with no test file changed;
- changed tests that import none of the changed code, so they can't fail for the right reason;
- a
.skip,.onlyorxfailadded while the claim is "tests pass"; - test-selection config changed in the same range (
load_tests,conftest.py,pytest.ini,testPathIgnorePatterns), which drops tests with no marker in the test itself; - test files deleted, or assertions going down;
- a deleted file still imported somewhere;
- a fix with no test showing it failed before and passes after;
- "no behaviour change" while existing assertions were rewritten;
- proof words ("verified", "works", "always") with nothing in the diff behind them.
Each commit's message is checked against that commit's own diff, and the summary against the whole range, so an edit that a later commit undid still shows.
It is a locator, not a judge. It says CONTRADICTED, UNPROVEN or NO CONTRADICTION FOUND, and none of those is a verdict until you've read the code. A claim with no contradiction can still be false: the script can't run the app.
--markdown prints the same result as a summary line and a table, ready for a PR comment.
The repository ships .github/workflows/verify-claims.yml, which does that on every pull
request: it checks the commits and the PR description and keeps one comment up to date. It
reports and never blocks a merge; it fails only if the script itself errors. Other
repositories can call it with uses: nicuk/did-ai-really-fix-it/.github/workflows/verify-claims.yml@main
(the file's header has the snippet). Offer it when the user keeps asking the same question
about every PR, and add it only if they want it.
Always run the orphan scan as well (Mode 2, step 1), even when nobody says "stuck". A fix whose changed file is orphaned, kept alive only by tests, or the unreachable half of a twin is a fix in dead code: its own test passes, and the running app never sees it. The claims check alone can't see that; the two together can.
3. Give each claim a verdict
For every claim, gather the cheapest evidence that could prove it false, then decide.
references/recipes.md has the check for each kind of claim, with commands. The core
moves are:
- A fix: find or write a test that fails on the commit before the fix and passes on
the commit with it. Run it at both commits in a separate worktree (
git worktree add), never by resetting the user's checkout. If the test passes on both, the claim isn't proven by it. - Tests added: does each new test import the code it's named after? Break that code on purpose (in the worktree) and confirm the test fails. A test that passes against broken code proves nothing.
- Tests pass: run the suite and report the count run, not the count written: skipped, todo and excluded files included. Compare the test count before and after the range.
- Removed / dead code: confirm nothing imports what was deleted, then run the orphan scan (mode 2) to see what the removal newly orphaned.
- Wired up / called by: trace a caller from a real entry point (a route, a page, a job, a CLI). A function with a caller that is itself unreachable is not wired.
- No behaviour change: run the same tests at both commits. Any edited assertion needs a reason.
| Verdict | Meaning |
| Holds | evidence that could have failed, didn't |
| Doesn't hold | the repository contradicts it; say what the truth is |
| Can't tell | the proof is outside what you can run (production, another service, a paid API); give the one command or query that would settle it |
Name what would disprove the verdict before you give it. A search describes where you looked, not what exists: a literal grep can't see a path built from a template string, and a repo search can't see a caller in another service.
Run things safely. Work in a git worktree or a copy, never by checking out or
resetting the user's branch. Keep every scratch file (probe scripts, before/after output)
inside the worktree or one folder you create for the check, never in /tmp or next to the
repo, and delete it when you're done. Mock every paid API. Cap loops and run under a timeout. If
something hangs, stop only the process you started, by its PID. Never kill processes by
name (taskkill /IM node.exe, pkill node): that stops the user's dev servers and other
agents.
4. Report for a founder
Lead with the count and the one claim that matters most. Plain words, no jargon the user didn't use first.
# What the agent said, checked
**<N> claims: <a> hold, <b> don't, <c> can't tell.** <One sentence on the claim that matters most.>
| # | The agent said | Verdict | What's actually true | Evidence |
|---|---|---|---|---|
| 1 | "Fixed the login redirect" | Doesn't hold | The fix is in `auth-old.ts`, which nothing loads; the live file is `auth.ts` | `file:line`; orphan scan |
## What to tell the agent
<ready-to-paste instructions for each claim that doesn't hold: what to do and how to prove it>
## Can't tell yet
<each open claim, with the one command or query that settles it>End with a ready-to-paste message for the agent. It is what the founder will actually use.
Mode 2: the agent is stuck
Signs: the same bug "fixed" three times; fixes that land and change nothing; two files with the same name; the agent editing code the app never runs. The usual cause is that the agent is working in a dead copy of the thing that is broken.
1. Find what's live and what's dead
python <skill-dir>/scripts/verify_claims.py orphans --repo <repo> [--alias @=src] [--entry "glob" ...] \
[--smoke "python -c \"import app.main\""]It builds the import graph and reports:
- orphans in rounds. Round 1 is files nothing imports. Round 2 is files only round 1 imports, and so on until nothing new appears. Dead code is often kept alive only by other dead code, so one pass finds the first layer and misses the rest;
- twins: one exported name defined in several files, with which copy is reachable from
an entry point. This is how an agent fixes
AuthProviderin the wrong file; - named but never imported: files mentioned only as a path string, for example in a config. Read these before deciding: the config may load them;
- kept alive only by tests: tests import these, but nothing a user reaches does.
Entry points default to Next.js, SvelteKit, Nuxt, Astro, Storybook, Node and Python
conventions (main.py, app.py, wsgi.py, manage.py), applied from every app or package
folder in a monorepo, plus any file with a __main__ block, anything package.json names
in main, bin or scripts, a Vite index.html's module script, test-runner setup files,
and package.module:app strings (uvicorn, gunicorn, Helm charts, Dockerfiles). .svelte,
.vue and .mdx files count as used, and their imports are followed. Pass --entry to add
others (workers, cron handlers). If more than a quarter of an app's code comes out
unreachable, the script warns for that app: an entry point is probably missing. Fix that
before believing the result. When neither copy of a twin is reachable, it says so rather
than guessing which is live.
Import aliases are read automatically from every tsconfig.json / jsconfig.json (and
tsconfig.*.json): compilerOptions.paths, baseUrl and relative extends, each applying
to the files under its own folder. The run prints the aliases it read. --alias adds more;
@ and ~ for the repo root are assumed only when no --alias is given (in a Nuxt app, ~
means that app's folder). Workspace packages resolve by their package.json name, SvelteKit's
$lib by convention, and a TypeScript ./x.js import to x.ts. Python packages under any
src/ (including packages/*/src and services/*/src) resolve by their top-level name, a
folder with a pyproject.toml is its own import root, and relative imports
(from .models import x) are followed.
The script says "unreachable", not "unused". It can't follow an import built from a string
at runtime (importlib.import_module("app." + name), require(variable)), a plugin system
or another service. Before deleting anything, start the app once. --smoke "COMMAND"
does it safely: it adds a temporary git worktree of HEAD in the system temp folder (never
inside the user's checkout, never on their branch), runs COMMAND there under a timeout
(--smoke-timeout, default 120 s), reports app starts or app failed to start with
the last lines of output, and removes the worktree even on failure or timeout. On a timeout
it stops only the process it started, by PID or process group. If the app fails to start,
the scan's conclusions can't be trusted until it does: a file it calls dead may be loaded at
runtime, or the app is already broken. Pick a command that exits once the app has loaded (an
import, a build, one request), not a server that runs forever. The worktree has committed
code only, with no node_modules or virtualenv, so include the install step if the app
needs one (--smoke "npm ci && npm run build"). The exit code is 3 when it fails to start.
2. Prove which one runs
For each twin the agent keeps editing, prove the live one at runtime, not by name:
- add one temporary log line to each copy and exercise the feature once;
- or break the suspected-dead copy in a worktree and confirm nothing changes.
Remove the temporary log line afterwards.
3. Stop the loop
Give the founder three things:
- The verdict: which copy is live, which is dead, and the evidence.
- Deletion by reference, not by name: delete the dead copy only after confirming
nothing imports it through any channel (alias import, relative import,
require, dynamicimport(), path strings in configs). Re-run the orphan scan after each deletion, with--smoke, since removing one layer often orphans the next. Delete in one commit per round, and run the build and tests after each. - Rules for the agent, for
CLAUDE.mdorAGENTS.md, so it doesn't happen again:
- The live <thing> is `<path>`. `<dead path>` is dead and scheduled for deletion; never edit it.
- Before fixing a bug, name the file the running app loads for it, and how you know
(a caller from a route or page, or a log line you saw fire).
- A fix is done when a test that failed before it passes after it. Say which test.If the repo has the Cairn Memory plugin, the settled question ("which auth is live") also belongs in its closed-questions ledger.
Scoring (0–10)
Score how trustworthy the agent's reporting is on this range, with evidence for every point. Half points for partly met. Mark N/A what doesn't apply and rescale.
- Every claim names what changed specifically enough to check.
- Every file a claim says changed was changed.
- Every fix has a test that fails without it.
- New tests import and exercise what they're named after.
- "Tests pass" matches the count that actually ran, with nothing newly skipped.
- Deleted code has no remaining importers, and the removal orphaned nothing unnoticed.
- "Wired" claims trace to a real entry point.
- "No behaviour change" survives the same tests at both commits.
- No proof word ("verified", "works", "always") without evidence behind it.
- Nothing the agent edited is dead code.
Why each check exists
references/incidents.md has the real incidents behind these checks, including a dead
auth system kept alive only by other dead code that took three rounds of deletion to
surface, a live and a dead AuthProvider with the same name, a contract documented as
"one function, two callers" that had zero callers, and twelve probes in one project that
failed because the probe was wrong, not the code. Read it when a check seems like
overkill.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

