agentleFS
MariusAAROSMariusAAROS/paper-reproduction/.claude/skills/paper-to-specs-and-tests/SKILL.md

paper-to-specs-and-tests

Skill2 starsChanged 22 days agoMITView on GitHub

Turn a scientific paper into a reproduction spec (spec/<paper_id>.md) and executable pytest reproducibility tests that check whether local experiments recover the paper's tables and main results. Fetches the published version of the paper (falling back on arXiv), finds and assesses the official code repo, decides whether to build on it or write an own implementation, and pins down the target numbers with sources and tolerances. Use this whenever the user wants to reproduce, replicate or re-implement a paper, start a reproduction, write a spec or plan for a paper, extract a paper's experimental setup or target results, or write tests against a paper's tables, even if they only paste an arXiv link, DOI or title and say "let's reproduce this".

The file

Show its rules and text

We split this skill into the rules it gives an agent

14 rules. About 0 are common, 11 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈11

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run the resolver to collect identifiers and candidate versions:

    line 37
  2. Distinctive:

    Prefer the published (camera-ready / version-of-record) PDF, plus its supplementary material.

    line 47
  3. Distinctive:

    If the published version is paywalled or can't be found, use the latest arXiv version and say so.

    line 49
  4. Distinctive:

    Read tables from the arXiv LaTeX source when available.

    line 51
  5. Distinctive:

    If both versions are available and their numbers differ, take targets from the published version and list the differences in spec section 9.

    line 54
  6. Distinctive:

    Read the whole paper, including the appendix and the supplement.

    line 59
  7. Distinctive:

    Every detail needed to implement and evaluate: architecture, objective, data and splits, preprocessing, optimizer and schedule, number of …

    line 67
  8. Very specific:

    Read references/repo-assessment.md and follow it.

    line 76
  9. Distinctive:

    Delete template comments and placeholder rows.

    line 96
  10. Distinctive:

    Choose the targets: Every number in the main results table that this bench could plausibly run is a main target.

    line 106
  11. Distinctive:

    If the paper reports variance (std / CI over seeds), use about 2σ, but not less than the last printed digit.

    line 111
  12. Very specific:

    Proxy experiments: if a main experiment is out of compute reach, say so in section 4.3.

    line 123
  13. Very specific:

    Use the same IDs, values, tolerances and sources as the spec.

    line 132
  14. Distinctive:

    Keep the template's behaviour intact:

    line 135

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Paper → spec + reproducibility tests

This skill produces two things for one paper, and stops there (implementation and running experiments come later):

  1. spec/<paper_id>.md: everything an implementer needs to reproduce the paper faithfully. That means the method, setup, hyperparameters, evaluation protocol and target numbers, each with its source, plus a decision on whether to build on the official code.
  2. tests/<paper_id>/test_<paper_id>.py: pytest tests that read the local experiment results and check whether they recover the paper's numbers.

The person reading the spec will be an agent or engineer who has not read the paper. Write for them: precise, sourced, and explicit about what is uncertain.

Conventions

  • paper_id: <first author's last name><year>_<short_keyword>, lowercase ASCII with underscores, e.g. vaswani2017_transformer, devlin2019_bert. It must be a valid Python identifier fragment because it names the test file. The same ID is used in spec/, tests/, results/ and papers/.
  • Downloaded paper files go in papers/<paper_id>/source/, which is gitignored.
  • Experiments will write results/<paper_id>/<experiment_id>.json, following the contract in spec section 7. The tests read only these files and never import implementation code, so the tests work whether the implementation is the official repo, a fork or a rewrite.

Workflow

1. Fetch the paper: published version first, arXiv as fallback

Run the resolver to collect identifiers and candidate versions:

python .claude/skills/paper-to-specs-and-tests/scripts/resolve_paper.py "<arXiv id | DOI | title | URL>"

Then read references/fetching.md. It covers how to read the resolver's noisy output, where each venue hosts its open-access camera-ready, and how to download and extract the paper safely. In short:

  • Prefer the published (camera-ready / version-of-record) PDF, plus its supplementary material. Camera-ready versions often change numbers.
  • If the published version is paywalled or can't be found, use the latest arXiv version and say so.
  • Read tables from the arXiv LaTeX source when available. It is far more reliable than PDF text extraction. Cross-check the values against the version the targets come from.
  • If both versions are available and their numbers differ, take targets from the published version and list the differences in spec section 9.

2. Read the paper for reproduction

Read the whole paper, including the appendix and the supplement. Hyperparameters hide in footnotes, table captions and appendix tables. While reading, collect:

  • The claims: what the paper says its method achieves, in a form that can be checked.
  • The main results: the headline table(s) and figure(s) that back those claims. These become main targets. Ablations and extra datasets are secondary.
  • Every detail needed to implement and evaluate: architecture, objective, data and splits, preprocessing, optimizer and schedule, number of seeds, checkpoint selection, and the exact metric definitions.
  • Gaps: anything an implementer would have to guess. These become assumptions (A1, A2, ...) in spec section 8, each with a reason and the impact if it is wrong.

3. Find and assess the official code

Read references/repo-assessment.md and follow it. The principle: start from the official repository when one exists. Papers omit details the code makes explicit, and building on the official code is usually the shortest path to a faithful reproduction. Write an own implementation only when there is no official code, or the assessment shows it is unusable: target experiments are missing, it cannot be installed even after porting, or there is evidence it does not produce the paper's numbers.

Inspect the repo by reading it (shallow clone into the scratchpad, or browse on GitHub). Do not run its code at this stage. Record the commit SHA you assessed.

The spec must state one of three decisions: Base on official repo, Official repo as reference only, or Own implementation. It must also state a fallback trigger: concrete conditions under which the implementer drops the official code and switches to an own implementation. That way the decision can be checked during implementation instead of being fixed at spec time.

4. Write the spec

Copy assets/spec_template.md to spec/<paper_id>.md and fill every section. Delete template comments and placeholder rows. Rules that make the spec usable:

  • Every number has a source: a paper location ("Table 3, row 'Ours', col 'mAP'"; "Sec. 4.2"; "App. B, Table 8"), an official-code location (configs/base.yaml:14), or an assumption ID. An implementer who doubts a value must be able to check it in a minute.
  • Copy target values exactly as printed, in the paper's units. Record unit conversions in the results contract, not by changing the value (e.g. the paper prints 76.3 (%) and the results file stores top1_acc as 0.763 → the target is 0.763 with source "Table 1: 76.3%").
  • Choose the targets. Every number in the main results table that this bench could plausibly run is a main target. Pick a few representative numbers from each secondary table, not all of them. A target nobody will ever run only adds noise.
  • Set tolerances on purpose and write the rationale in section 6.1:
    • If the paper reports variance (std / CI over seeds), use about 2σ, but not less than the last printed digit.
    • Otherwise default to roughly ±1.0 point for percentage metrics (accuracy, F1, BLEU, mAP), ±2% relative for losses and perplexities, and judgment for anything else. Tighten for deterministic evaluations (evaluating a released checkpoint) and loosen for notoriously noisy setups (RL, small datasets).
    • A tolerance is a statement about what counts as "reproduced". Don't widen it later to make a test pass. Record the miss instead.
  • Add relative claims (section 6.2) for the paper's comparative statements: "ours > baseline", "the ablation hurts", "scaling helps". They often reproduce even when absolute numbers drift (different hardware, library versions), so they separate "the method works" from "the number matches".
  • Proxy experiments: if a main experiment is out of compute reach, say so in section 4.3. A scaled-down proxy can be added in section 6.3 with priority="proxy" and its own expected result. It does not replace the paper target, and both targets stay in the spec.

5. Write the reproducibility tests

Copy assets/test_template.py to tests/<paper_id>/test_<paper_id>.py, set PAPER_ID, the docstring, and fill TARGETS and CLAIMS from spec section 6. Use the same IDs, values, tolerances and sources as the spec. The built-in test_spec_lists_every_id fails if an ID is missing from the spec.

Keep the template's behaviour intact:

  • Missing results → skip, not fail. The suite is committed before any experiment has run, and a skip means "not attempted yet".
  • Results present but off → fail with a message that gives the local value, the paper value, the tolerance and the source.
  • Results files are validated (paper_id, experiment_id, metric present), so a wrong file path or a typo in a metric name fails loudly instead of silently.

Tailor beyond the template when the paper needs it, e.g. a target that is a curve (check a few points), a result reported as "within X of" another, or a metric where lower is better (express it as a Claim with greater=baseline).

6. Verify

From the repo root:

python -m pytest tests/<paper_id> -v          # expect: spec-sync test passes, the rest skip
python .claude/skills/paper-to-specs-and-tests/scripts/smoke_test.py <paper_id>

smoke_test.py runs the suite against fake results: at the paper's values every target and claim must pass, and with values shifted by 3× tolerance every target must fail. A claim that fails at the paper's own values means a wrong direction, wrong units, or a claim the paper's numbers don't support. Fix the cause before finishing.

Then reread the spec from the implementer's side. Could you start implementation from it without opening the paper? Is every target traceable to a table cell?

7. Report back

Tell the user, briefly:

  • which paper version the targets come from, and where it was fetched
  • the implementation decision and its main reason
  • the number of main / secondary / proxy targets and claims
  • the riskiest assumptions and any experiments that look out of compute reach
  • the paths of the spec and the test file

Files in this skill

  • scripts/resolve_paper.py: queries arXiv, Semantic Scholar, DBLP, OpenAlex and Crossref for a paper's versions (stdlib only).
  • scripts/smoke_test.py: checks that a generated test suite skips, passes and fails when it should.
  • references/fetching.md: finding and downloading the published version or the arXiv fallback; venue table.
  • references/repo-assessment.md: finding official code, the assessment rubric, and the decision rules.
  • assets/spec_template.md: spec skeleton.
  • assets/test_template.py: pytest skeleton.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.