Claim Ledger

What this measures

Whether what the agent said can be reproduced.

Every AI coding session ends with a report — "tests pass", "this fixes the NPE", "no other callers affected". Nothing binds those sentences to a check that can be run and can fail. The gap is not anecdotal. Agents failed to read every file they were asked to review in 67.9% of runs, and were misleading 80.4% of the time when they did (OverclaimBench, arXiv:2609.20812). 75.8% of self-assessed coding trajectories asserting completion were false (arXiv:2606.09863). In this repository, 30 of 114 commits exist to correct an earlier claim.

So the gate states one narrow thing:

Every outcome an agent asserts is reproduced by running the check that claim names — or the claim is reported unverified, and unverified fails.

A claim can be true, reproducible, and worthless, and this instrument says so in its own output. It only reports that the outcome was reproduced, not that it was worth reproducing.

Run it

scripts/elohim_run.py                    # fixture mode -- the gate
scripts/elohim_run.py --json             # the shard
scripts/check_traps.py                   # traps, re-derived independently

Fixture mode runs the committed claims file and is what the gate and the ledger pin. Applied mode runs your own:

instrument/claim_ledger.py --claims /path/to/your/claims.json

Your file needs {"author": "...", "claims": [{"id", "text", "check": {"argv", "expect_exit", "asserts"}}]}. expect_exit is the only assertion vocabulary in v1. Set ELOHIM_CLAIM_AUTHOR if you run it — a file authored by the run judging it is reported as self-authored and the verdict fails.

Applied mode is a report, not a gate: your file is not the pinned one, so claims_are_the_pinned_ones is false and the verdict is FAIL by construction.

Read the output

off_scope and overstated scope are warnings, never fatal. Judging whether a claim's scope is too narrow is a judgement about prose, and a judgement step would make the gate worse than the thing it gates.

Two constraints this is built around

1. No LLM anywhere in the measurement path. Across five judges and five prompt strategies no LLM judge exceeded AUROC 0.65, while TF-IDF detectors reached 0.83–0.95 at ~3,300× lower latency, and false acceptance rises with agent capability. no_llm_in_the_measurement_path re-derives this every run by walking the instrument's own AST — and check_traps.py derives it again, from the same bytes, with a parser of its own.

2. The instrument must not share failure modes with the run it judges. When one trajectory writes both the patch and the test, "their errors can agree and create false confidence" (ExecCritic, arXiv:2609.09133). So the expectations live in a pinned ledger, the instrument is pinned by checksum, and a control that edits the instrument turns the gate red.

Measured on a real session

The fixture corpus is two hand-written claims. scripts/replay.py measures the gate against a real one instead: 14 claims read by hand out of a 1.9 MB agent session, each labelled against that session's own recorded tool results.

RECALL 2/2 (100.0%)   PRECISION 2/2 (100.0%)   COVERAGE 2/14 (14.3%)
CONTRADICTED 0/0 (undefined)   UNVERIFIED 2   NO_EXIT_CODE 0

Zero of the 14 claims were contradicted by the agent's own transcript. The two defects are both UNVERIFIED: the agent asserted that CI had been red "for the 26th consecutive run" and that a ratchet figure "had been 655/1161" — and neither figure appears anywhere in the session's 165 recorded tool results. So recall on contradiction is 0/0, undefined; the 100% above is carried entirely by the unverified class, and must never be restated as a contradiction rate.

Three things that figure is not, all of which cost something to find:

• It does not prove this gate works. With replay.py neutered to echo the labels instead of reading the transcript, the output was byte-identical, exit code included, because every binding in this corpus happens to agree with its label. The evidence that the gate is wired to the transcript is tests/test_replay_negative_controls.py, not the number.
• It is one agent, one afternoon, fourteen claims. It bounds what happened in that session and supports nothing about agents in general.
• It reaches half the corpus. 434 of 895 tool results across 25 sessions carry no exit code at all, so the check adjudicates shell-shaped claims and nothing else.

There is no claim classifier in replay.py and there will not be one. Two were built and measured on this corpus first: a failure-keyword classifier called 15 of 19 real failures undisclosed at roughly a 79% false-positive rate, and a metric-noun binding reported 120 contradictions of which the sampled rows were 100% false positive. The human supplies the binding, the transcript supplies the truth, the script joins them. references/limits.md has both measurements and the rows that fooled them.

Traps

Nine, plus one fact with a control of its own. Seven produced a confident wrong answer before they were written down; the two over the label fixture were added after the recall measurement showed the ground truth had no gate at all. check_traps.py re-derives all of them with its own code and never imports the instrument, so a regression is visible rather than agreed with.

tests/negative_controls_claim_ledger.py proves every one of them can go red. Each control names the trap it targets and fails unless that trap broke — a control that merely breaks some trap passes for free, because every synthetic corpus also breaks the_stale_ledger.

References

• references/traps.md — each trap, the wrong answer it was, and its control, including the one that shipped vacuous and how it was caught
• references/limits.md — what an exit-code gate over one claims file cannot do