paper-to-specs-and-tests
Skill2 starsChanged 22 days agoMITView on GitHub
Turn a scientific paper into a reproduction spec (spec/<paper_id>.md) and executable pytest reproducibility tests that check whether local experiments recover the paper's tables and main results. Fetches the published version of the paper (falling back on arXiv), finds and assesses the official code repo, decides whether to build on it or write an own implementation, and pins down the target numbers with sources and tolerances. Use this whenever the user wants to reproduce, replicate or re-implement a paper, start a reproduction, write a spec or plan for a paper, extract a paper's experimental setup or target results, or write tests against a paper's tables, even if they only paste an arXiv link, DOI or title and say "let's reproduce this".
In this repository3 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- implement-paperSkill
- test-paperSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
14 rules. About 0 are common, 11 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈11
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 37
Run the resolver to collect identifiers and candidate versions:
- Distinctive: line 47
Prefer the published (camera-ready / version-of-record) PDF, plus its supplementary material.
- Distinctive: line 49
If the published version is paywalled or can't be found, use the latest arXiv version and say so.
- Distinctive: line 51
Read tables from the arXiv LaTeX source when available.
- Distinctive: line 54
If both versions are available and their numbers differ, take targets from the published version and list the differences in spec section 9.
- Distinctive: line 59
Read the whole paper, including the appendix and the supplement.
- Distinctive: line 67
Every detail needed to implement and evaluate: architecture, objective, data and splits, preprocessing, optimizer and schedule, number of …
- Very specific: line 76
Read references/repo-assessment.md and follow it.
- Distinctive: line 96
Delete template comments and placeholder rows.
- Distinctive: line 106
Choose the targets: Every number in the main results table that this bench could plausibly run is a main target.
- Distinctive: line 111
If the paper reports variance (std / CI over seeds), use about 2σ, but not less than the last printed digit.
- Very specific: line 123
Proxy experiments: if a main experiment is out of compute reach, say so in section 4.3.
- Very specific: line 132
Use the same IDs, values, tolerances and sources as the spec.
- Distinctive: line 135
Keep the template's behaviour intact:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Paper → spec + reproducibility tests
This skill produces two things for one paper, and stops there (implementation and running experiments come later):
spec/<paper_id>.md: everything an implementer needs to reproduce the paper faithfully. That means the method, setup, hyperparameters, evaluation protocol and target numbers, each with its source, plus a decision on whether to build on the official code.tests/<paper_id>/test_<paper_id>.py: pytest tests that read the local experiment results and check whether they recover the paper's numbers.
The person reading the spec will be an agent or engineer who has not read the paper. Write for them: precise, sourced, and explicit about what is uncertain.
Conventions
paper_id:<first author's last name><year>_<short_keyword>, lowercase ASCII with underscores, e.g.vaswani2017_transformer,devlin2019_bert. It must be a valid Python identifier fragment because it names the test file. The same ID is used inspec/,tests/,results/andpapers/.- Downloaded paper files go in
papers/<paper_id>/source/, which is gitignored. - Experiments will write
results/<paper_id>/<experiment_id>.json, following the contract in spec section 7. The tests read only these files and never import implementation code, so the tests work whether the implementation is the official repo, a fork or a rewrite.
Workflow
1. Fetch the paper: published version first, arXiv as fallback
Run the resolver to collect identifiers and candidate versions:
python .claude/skills/paper-to-specs-and-tests/scripts/resolve_paper.py "<arXiv id | DOI | title | URL>"Then read references/fetching.md. It covers how to read the resolver's noisy
output, where each venue hosts its open-access camera-ready, and how to download
and extract the paper safely. In short:
- Prefer the published (camera-ready / version-of-record) PDF, plus its supplementary material. Camera-ready versions often change numbers.
- If the published version is paywalled or can't be found, use the latest arXiv version and say so.
- Read tables from the arXiv LaTeX source when available. It is far more reliable than PDF text extraction. Cross-check the values against the version the targets come from.
- If both versions are available and their numbers differ, take targets from the published version and list the differences in spec section 9.
2. Read the paper for reproduction
Read the whole paper, including the appendix and the supplement. Hyperparameters hide in footnotes, table captions and appendix tables. While reading, collect:
- The claims: what the paper says its method achieves, in a form that can be checked.
- The main results: the headline table(s) and figure(s) that back those
claims. These become
maintargets. Ablations and extra datasets aresecondary. - Every detail needed to implement and evaluate: architecture, objective, data and splits, preprocessing, optimizer and schedule, number of seeds, checkpoint selection, and the exact metric definitions.
- Gaps: anything an implementer would have to guess. These become assumptions (A1, A2, ...) in spec section 8, each with a reason and the impact if it is wrong.
3. Find and assess the official code
Read references/repo-assessment.md and follow it. The principle: start from
the official repository when one exists. Papers omit details the code makes
explicit, and building on the official code is usually the shortest path to a
faithful reproduction. Write an own implementation only when there is no
official code, or the assessment shows it is unusable: target experiments are
missing, it cannot be installed even after porting, or there is evidence it
does not produce the paper's numbers.
Inspect the repo by reading it (shallow clone into the scratchpad, or browse on GitHub). Do not run its code at this stage. Record the commit SHA you assessed.
The spec must state one of three decisions: Base on official repo, Official repo as reference only, or Own implementation. It must also state a fallback trigger: concrete conditions under which the implementer drops the official code and switches to an own implementation. That way the decision can be checked during implementation instead of being fixed at spec time.
4. Write the spec
Copy assets/spec_template.md to spec/<paper_id>.md and fill every section.
Delete template comments and placeholder rows. Rules that make the spec usable:
- Every number has a source: a paper location ("Table 3, row 'Ours', col
'mAP'"; "Sec. 4.2"; "App. B, Table 8"), an official-code location
(
configs/base.yaml:14), or an assumption ID. An implementer who doubts a value must be able to check it in a minute. - Copy target values exactly as printed, in the paper's units. Record unit
conversions in the results contract, not by changing the value (e.g. the paper
prints 76.3 (%) and the results file stores
top1_accas 0.763 → the target is 0.763 with source "Table 1: 76.3%"). - Choose the targets. Every number in the main results table that this bench
could plausibly run is a
maintarget. Pick a few representative numbers from each secondary table, not all of them. A target nobody will ever run only adds noise. - Set tolerances on purpose and write the rationale in section 6.1:
- If the paper reports variance (std / CI over seeds), use about 2σ, but not less than the last printed digit.
- Otherwise default to roughly ±1.0 point for percentage metrics (accuracy, F1, BLEU, mAP), ±2% relative for losses and perplexities, and judgment for anything else. Tighten for deterministic evaluations (evaluating a released checkpoint) and loosen for notoriously noisy setups (RL, small datasets).
- A tolerance is a statement about what counts as "reproduced". Don't widen it later to make a test pass. Record the miss instead.
- Add relative claims (section 6.2) for the paper's comparative statements: "ours > baseline", "the ablation hurts", "scaling helps". They often reproduce even when absolute numbers drift (different hardware, library versions), so they separate "the method works" from "the number matches".
- Proxy experiments: if a main experiment is out of compute reach, say so in
section 4.3. A scaled-down proxy can be added in section 6.3 with
priority="proxy"and its own expected result. It does not replace the paper target, and both targets stay in the spec.
5. Write the reproducibility tests
Copy assets/test_template.py to tests/<paper_id>/test_<paper_id>.py, set
PAPER_ID, the docstring, and fill TARGETS and CLAIMS from spec section 6.
Use the same IDs, values, tolerances and sources as the spec. The built-in
test_spec_lists_every_id fails if an ID is missing from the spec.
Keep the template's behaviour intact:
- Missing results → skip, not fail. The suite is committed before any experiment has run, and a skip means "not attempted yet".
- Results present but off → fail with a message that gives the local value, the paper value, the tolerance and the source.
- Results files are validated (
paper_id,experiment_id, metric present), so a wrong file path or a typo in a metric name fails loudly instead of silently.
Tailor beyond the template when the paper needs it, e.g. a target that is a
curve (check a few points), a result reported as "within X of" another, or a
metric where lower is better (express it as a Claim with greater=baseline).
6. Verify
From the repo root:
python -m pytest tests/<paper_id> -v # expect: spec-sync test passes, the rest skip
python .claude/skills/paper-to-specs-and-tests/scripts/smoke_test.py <paper_id>smoke_test.py runs the suite against fake results: at the paper's values
every target and claim must pass, and with values shifted by 3× tolerance every
target must fail. A claim that fails at the paper's own values means a wrong
direction, wrong units, or a claim the paper's numbers don't support. Fix the
cause before finishing.
Then reread the spec from the implementer's side. Could you start implementation from it without opening the paper? Is every target traceable to a table cell?
7. Report back
Tell the user, briefly:
- which paper version the targets come from, and where it was fetched
- the implementation decision and its main reason
- the number of main / secondary / proxy targets and claims
- the riskiest assumptions and any experiments that look out of compute reach
- the paths of the spec and the test file
Files in this skill
scripts/resolve_paper.py: queries arXiv, Semantic Scholar, DBLP, OpenAlex and Crossref for a paper's versions (stdlib only).scripts/smoke_test.py: checks that a generated test suite skips, passes and fails when it should.references/fetching.md: finding and downloading the published version or the arXiv fallback; venue table.references/repo-assessment.md: finding official code, the assessment rubric, and the decision rules.assets/spec_template.md: spec skeleton.assets/test_template.py: pytest skeleton.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

