agentleFS
zirui-songzirui-song/claude-skills/skills/verify-numbers-pipeline/SKILL.md

verify-numbers-pipeline

Skill17 starsChanged 3 months agoMITView on GitHub

Use to make every number in a paper machine-checkable — inventories scripts to the tables they produce, builds a YAML manifest of every numeric claim in the manuscript prose, writes verify_numbers.py to re-parse estimation logs and diff them against the text, adds make all / make verify targets and a pre-commit hook that blocks stale numbers. Trigger phrases include "self-verifying pipeline", "check every number in the paper", "which sentences went stale", "make verify".

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run it against the current repo and give a ranked report of every stale number found — ranked by how central the claim is to the paper's …

    line 52

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Verify Numbers Pipeline

Builds the infrastructure that makes stale numbers impossible to commit. One-time setup per paper repository; verify-claims is the per-draft pass that runs on top of it.

1. Inventory

Map every Stata do-file and Python script to the .tex tables and figures it produces. Write it to pipeline_map.yaml: script → outputs → the log file it writes.

2. Claim manifest

claims.yaml — every numeric claim appearing in the manuscript prose (not just the tables): coefficients, standard errors, sample sizes, F-stats, percentages. Each entry:

- id: main_effect_bps
  text: "spreads fall by 42.5 basis points"
  tex_file: sections/results.tex
  value: -42.5
  tolerance: 0.05
  source_log: logs/main_spec.log
  line_pattern: 'post_treat\s+-?\d+\.\d+'

3. verify_numbers.py

Re-parses each source_log, extracts the value via line_pattern, asserts it matches value within tolerance. Prints a diff table of every mismatch: claim id, manuscript value, log value, delta, and the file:line on both sides. Non-zero exit on any mismatch.

A claim whose log is missing is a failure, not a skip — it means the number was never reproduced.

4. Make targets

  • make all — full pipeline, raw data → compiled PDF.
  • make verify — assertion layer only, no re-estimation. Fast enough to run constantly.

5. Pre-commit hook

Blocks the commit when make verify fails. Prints the diff table so the user sees which sentence went stale.

6. First run

Run it against the current repo and give a ranked report of every stale number found — ranked by how central the claim is to the paper's argument.

Do not fix the prose in this pass. Show the diff first; the user decides which side is right (the text may be current and the log stale, or vice versa).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.