agentleFS
ShaishavMaisuriaShaishavMaisuria/research-paper-lifecycle-skills/skills/verify-claims/SKILL.md

verify-claims

Skill59 starsChanged 5 months agoApache-2.0View on GitHub

Audit load-bearing paper claims against evidence in results, tables, figures, experiments, or citations. Use for claim audits, overclaiming checks, first or state-of-the-art claims, significant-result wording, prose-table number mismatches, and camera-ready or rebuttal evidence checks.

13 other projects use a version of this.

Adopted by

Most recently taken up by

The file

Show its rules and text

We split this skill into the rules it gives an agent

13 rules. 0 are common, 9 are distinctive, and 4 are very specific to this project.

  • 0

    Common

    Most of the 13 edited versions we read keep these.

  • 9

    Distinctive

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run both before any draft, rebuttal, or camera-ready leaves the machine.

    line 20
  2. Distinctive:

    Before a rebuttal — reviewers' top complaint is unsupported or overclaimed contributions.

    line 26
  3. Distinctive:

    Before camera-ready or arXiv — last chance to soften an indefensible "first" or fix a number that drifted out of sync with a revised table.

    line 28
  4. Very specific:

    After polish-tables-figures regenerated a table, to confirm the prose still matches the new numbers.

    line 30
  5. Distinctive:

    Read references/claim-taxonomy.md for what each claim type requires as evidence and the standard reviewer attack on each.

    line 91
  6. Distinctive:

    Apply the overclaiming rules: Read references/overclaiming-rules.md.

    line 95
  7. Very specific:

    Numbers in prose: must equal the numbers in the tables/figures they summarize.

    line 107
  8. Distinctive:

    Never the fourth move: inventing evidence, a result, a citation, or a significance test that was not run.

    line 119
  9. Distinctive:

    Never fabricate evidence: No invented result, table number, citation, baseline, or significance test to "support" a claim.

    line 153
  10. Distinctive:

    Treat them as claims requiring proof, not rhetorical flourish.

    line 160
  11. Distinctive:

    When a claim's evidence is a citation, hand that reference to verify-citations.

    line 163
  12. Very specific:

    At start: read lessons.md to skip re-flagging claims the author already scoped or backed this cycle, and lead with any recurring …

    line 192
  13. Very specific:

    Create .paper-memory/ on demand if absent and offer to add it to the project .gitignore.

    line 204

“Common” is measured: kept by at least 6 in 10 of the 13 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Verify Claims

The claim-to-evidence gate. Reviewers attack the gap between what a paper asserts and what it shows — an unbacked "we are the first", a "state-of-the-art" with no comparison, a "significant improvement" with no test, a speedup stated in the text that the table does not support. This skill finds those candidate claims, makes the author map each to its evidence, and produces a claims matrix (claim · location · evidence · status) so the gap is visible and closeable before a reviewer finds it.

It is the content counterpart to verify-citations: that skill proves each reference in the .bib resolves to a real paper; this skill proves each load-bearing sentence in the prose is backed by something in this paper (a number, a figure, a table, an experiment) or a real citation. Run both before any draft, rebuttal, or camera-ready leaves the machine.

When to use

  • The user asks to verify / audit / check claims (not citations): "are my claims supported?", "am I overclaiming?", "do my results match my tables?"
  • Before a rebuttal — reviewers' top complaint is unsupported or overclaimed contributions; close the gaps first, or arm the rebuttal with the evidence.
  • Before camera-ready or arXiv — last chance to soften an indefensible "first" or fix a number that drifted out of sync with a revised table.
  • After polish-tables-figures regenerated a table, to confirm the prose still matches the new numbers.

Inputs

  • The paper's .tex (main file; the script follows \input/\include). Find it next to the .bib, or via the file with \documentclass.
  • The author, in the loop: the skill cannot decide whether a claim is true — it surfaces candidates and the author supplies (or admits the absence of) the evidence. Copilot, not pilot.
  • Optional: .paper-memory/profile.yml — risk_appetite and contribution_type set how hard the author wants to push novelty vs. hedge (see Memory).

Process

  1. Extract candidate claims. Deterministic work belongs to the script — do not eyeball the paper for claims:

    python3 scripts/claim_audit.py path/to/main.tex --json /tmp/claims.json

    It scans the prose (skipping math, comments, tables) for sentences carrying novelty markers ("first", "novel", "we are the only", "unlike prior work"), superiority markers ("outperforms", "state-of-the-art", "best", "superior"), magnitude/result markers ("significantly", "X% improvement", "Nx faster", "substantially"), and generalization markers ("always", "in all cases", "guarantees"). It also pulls every numeric token in the prose and every numeric cell in the tables, so you can cross-check. Each candidate gets a type, the sentence, and a file:line location.

    Useful variants:

    • --type novelty (or superiority, result, generalization) — focus one class of claim.
    • --numbers — emit only the prose-number vs. table-number cross-check list (for the "do my results match my tables?" question).
    • --context N — include N sentences of surrounding text per claim.
    • --min-confidence high — only the strongest-signal candidates (fewer false positives) when the paper is large.

    Exit codes: 0 no candidate claims found (rare — usually means the file parsed but is near-empty; check the input), 2 candidate claims were found (the normal case — they need author triage, not a "problem"), 1 operational failure (unreadable file, bad arguments). The script finds claims; it does not judge them — a nonzero 2 is the expected, healthy result, not a failure.

  2. Map each candidate to evidence — with the author. This is the core of the skill and it is NOT automatable: the script flags "we achieve state-of-the-art accuracy"; only the author (or the paper's own Table 3) can say which result backs it. For each candidate, establish:

    • Evidence — the specific result, table/figure number, experiment, theorem, or citation that supports it (a \ref/\label, a table cell, a section). "Section 4" is not evidence; "Table 3, row BERT-large, +2.1 F1 over the strongest baseline" is.
    • Status — SUPPORTED (evidence exists and matches), WEAK (evidence exists but is thinner than the claim — e.g. "significantly" with no significance test), UNSUPPORTED (no evidence found), MISMATCH (prose number disagrees with the table), or SCOPED (claim is fine once narrowed — see step 4).

    Read references/claim-taxonomy.md for what each claim type requires as evidence and the standard reviewer attack on each.

  3. Apply the overclaiming rules. Read references/overclaiming-rules.md. The high-frequency offenders:
    • "first" / "novel" needs a defensible scope and a literature check — an absolute "first to X" is a single-counterexample-away from a desk-level embarrassment. Prefer a scoped "first to X under constraint Y".
    • "state-of-the-art" / "outperforms" needs the comparison: which baselines, on which benchmark, by how much, and whether the baselines are current and fairly tuned.
    • "significantly" / "substantial" is a statistical word: it needs a test (and the test named), not just a bigger mean. If no test was run, the word should change, not the data.
    • Numbers in prose must equal the numbers in the tables/figures they summarize. The script's --numbers mode lists prose vs. table numerics; reconcile every mismatch (a revised table that left a stale sentence behind is the classic camera-ready bug).
  4. Decide the fix per claim, with the author. Three honest moves, never a fourth:
    • Back it — add/point to the missing evidence (a result already in the paper, a citation, an experiment to run).
    • Scope it — narrow the claim to what the evidence actually supports ("first" → "first under Y"; "always" → "in our experiments").
    • Cut it — remove the claim if it cannot be backed or scoped. Never the fourth move: inventing evidence, a result, a citation, or a significance test that was not run. An honest "this claim is unsupported" beats a fabricated backing.
  5. Write the claims matrix to paper-workspace/review/claims-matrix.md: one row per load-bearing claim — claim · location · type · evidence · status · suggested fix. Lead the chat summary with counts by status and the most dangerous open items (UNSUPPORTED and MISMATCH first). Append the run to INDEX.md.
  6. Re-run after fixes. Once the author edits, re-run claim_audit.py (and --numbers) to confirm softened/scoped claims no longer trip the markers and no prose/table mismatch remains. Stop when every load-bearing claim is SUPPORTED, SCOPED, or an explicitly author-accepted WEAK — not on an open-ended "keep improving" loop (cap at a couple of passes; report what remains open).

Output

  • claim_audit.py stdout (and --json): candidate claims with type, location, and the prose-vs-table numeric cross-check — a worklist, not a verdict.
  • paper-workspace/review/claims-matrix.md: the reviewable claims matrix the author fills in and acts on, plus a one-line INDEX.md entry.
  • A chat summary: counts by status, the riskiest open claims, and the per-claim fix (back / scope / cut) — never a fabricated backing.

Hard rules

  • The skill never decides a claim is true or false on its own. It has no oracle for "is this the first paper to do X" or "is this really state-of-the-art" — it surfaces the candidate and routes the judgment to the author and the paper's own evidence. Do not use the model's self-assessment as the verification signal (it measures plausibility, not correctness, and is worst exactly when most confident).
  • Never fabricate evidence. No invented result, table number, citation, baseline, or significance test to "support" a claim. If it cannot be backed, it must be scoped or cut.
  • Numbers are facts, not prose. A prose number that disagrees with its table is a MISMATCH to reconcile from the table (or the underlying result), never by editing the table to match a sentence the author likes better.
  • "first"/"SOTA"/"significant" are load-bearing words with evidentiary cost. Treat them as claims requiring proof, not rhetorical flourish.
  • This skill checks claim→evidence within this paper; it does not verify the .bib resolves — that is verify-citations. When a claim's evidence is a citation, hand that reference to verify-citations.
  • It reports and explains; it never edits the paper or submits anything. The author makes every back/scope/cut decision.

Adapt to your discipline

Defaults target CS venues (IEEE/ACM/ML). The marker lexicons and evidence expectations are field-specific: a theory paper's claims trace to theorems and proofs (not tables); an HCI paper's to study design and significance reporting; a survey's to coverage and taxonomy completeness rather than "outperforms". Edit references/claim-taxonomy.md and the marker lists in claim_audit.py (documented inline) for your field's claim vocabulary and standards of proof.

Bundled resources

  • scripts/claim_audit.py — extracts candidate claim sentences and the prose/table numeric cross-check from a .tex. Stdlib only; --help for all options. Run it; do not hand-scan the paper.
  • references/claim-taxonomy.md — the claim types, what evidence each needs, and the standard reviewer attack on each.
  • references/overclaiming-rules.md — the high-frequency overclaims ("first", "SOTA", "significant", stale numbers) and the back/scope/cut remedy for each.

Memory

Uses the shared .paper-memory/ convention (full spec: paper-memory-convention.md).

  • At start: read lessons.md to skip re-flagging claims the author already scoped or backed this cycle, and lead with any recurring overclaiming habit (e.g. "tends to write absolute 'first' claims; scope them up front"). Read profile.yml risk_appetite — conservative authors want every WEAK claim hedged; aggressive authors accept defensible WEAK claims they will fight for in review.
  • At end: append durable findings via reflect-and-improve's reflect_log.py append in the shared format - [YYYY-MM-DD] (verify-claims | <scope>) issue -> recommendation. A habit across the paper or across papers (e.g. unscoped novelty claims, "significant" without a test) is recurring; a single fixed sentence is this-paper. Never record a fabricated backing — only the pattern and the honest fix.
  • Create .paper-memory/ on demand if absent and offer to add it to the project .gitignore. Local-only; never uploaded or copied into this repo.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.