agentleFS
MariusAAROSMariusAAROS/paper-reproduction/.claude/skills/test-paper/SKILL.md

test-paper

Skill2 starsChanged 22 days agoMITView on GitHub

Assess a paper reproduction. Runs the paper's reproducibility tests (tests/<paper_id>/) against the local results (results/<paper_id>/), checks that the results are valid (provenance, stale or fake results, edited targets), gives a verdict (reproduced / partially / not reproduced / inconclusive), diagnoses likely root causes of every failed target with cheap read-only checks, and writes a short report to reports/<paper_id>.md. Use this whenever the user asks whether a reproduction worked, how close the results are to the paper, why numbers don't match the paper, to check, evaluate, grade, validate or report on a reproduction, or to run a paper's tests, even if they only say "did it reproduce?" or name the paper.

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 3 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Run it with an interpreter that has pytest, e.g.

    line 26
  2. Distinctive:

    Start from the mechanical verdict.

    line 55
  3. Very specific:

    Read references/root-causes.md.

    line 64
  4. Distinctive:

    Pattern: look across all targets before diving into one.

    line 67
  5. Distinctive:

    Cheap checks: read configs and diff them against the spec, grep run logs (finished all steps?

    line 72

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Test a paper reproduction

Input: a paper_id (or the paper's name, mapped to spec/<paper_id>.md). Output: reports/<paper_id>.md, a short report with a verdict, the results table, validity issues, and ranked root-cause hypotheses for every failure, plus the paper's row in the README's "Reproduced papers" table.

This skill is the referee. It doesn't fix anything: it reads, runs the tests, and runs cheap checks, and leaves spec, tests, results and code untouched. A referee who edits the scoresheet can't be trusted, and keeping diagnosis separate from implementation stops the assessment from drifting toward whatever the code produced. Fixes and reruns are next steps for the implement-paper skill. Spec changes go to the user.

1. Collect the evidence

python .claude/skills/test-paper/scripts/collect.py <paper_id> --out <scratchpad>/<paper_id>_evidence.json

Run it with an interpreter that has pytest, e.g. the repo's .venv/bin/python. It runs pytest, joins each target with its results file (local value, delta, |delta|/tolerance, seeds), and runs the validity checks:

  • invalid: fake results written by the smoke test, or _smoke experiments
  • stale: patches/ changed after the result was written
  • provenance: missing fields (file not written by write_results.py), uncommitted code at run time, official commit ≠ spec commit
  • target edits: paper_value or tolerance changed, or targets removed, since the test file's first commit, without a matching spec change-log entry
  • spec/test sync

It prints a summary and a mechanical verdict:

VerdictMeaning
REPRODUCEDevery main target passes on valid results, and every evaluated claim holds
PARTIALLY REPRODUCEDsome main targets pass or some claims hold, or not all main targets have run
NOT REPRODUCEDno main target passes and no claim holds
INCONCLUSIVEno valid main results, or a blocking validity problem

Proxy and secondary targets don't decide the verdict; report them anyway.

Then read the context the numbers need: the spec (§4–§7, §8 assumptions, §9 known discrepancies, change log), papers/<paper_id>/REPRODUCTION_LOG.md (deviations, environment, runs), and the list of patches.

2. Judge the verdict

Start from the mechanical verdict. You may downgrade it with a stated reason, e.g. REPRODUCED → PARTIALLY REPRODUCED or INCONCLUSIVE when a results file is valid on paper but the log shows the run was cut short, or a local value beats the paper suspiciously (possible leakage). Never upgrade a verdict past failing tests: the tolerance is the agreed definition of "reproduced", and the report should say a near-miss is near, not call it a pass.

3. Diagnose every failure

Read references/root-causes.md. It maps failure patterns to likely causes and lists cheap checks for each. Work in this order:

  1. Pattern: look across all targets before diving into one. Which experiments fail? By how much and in which direction? Do the claims still hold? Does Stage 0 (released checkpoints) match?
  2. Hypotheses: up to three per failing group, from the catalogue entries the pattern points to.
  3. Cheap checks: read configs and diff them against the spec, grep run logs (finished all steps? NaN? dataset sizes?), read the behaviour patches and deviations, recompute a metric from saved predictions on CPU, run the paper's unit tests, and check the official repo's issues. No training, no GPU evaluation, no edits. A hypothesis that needs a run becomes a next step with the command and its cost.
  4. Confidence: high only when a check confirmed it. Name what you ruled out.

Also look at passing results for a moment. A pass from invalid or stale results is not a pass, and a value far above the paper is a red flag even inside the tolerance.

4. Write the report

Fill assets/report_template.md into reports/<paper_id>.md (overwrite: the report reflects the current state; git keeps history). Keep it short, one screen if things pass and two at most with failures. The reader wants the verdict, the table, and the most likely reason for each gap, not the investigation diary. Concrete evidence beats prose: numbers, file:line, log lines.

Then update the paper's row in the README's "Reproduced papers" table (paper, venue/year, verdict, link to the report), adding the row if missing.

5. Tell the user

In chat: the verdict with its one-line reason, the main targets as local vs paper ± tol, the top root-cause hypothesis per failure, and the path to the report. If validity problems made the verdict inconclusive, lead with them, because they are the first thing to fix.

Files in this skill

  • scripts/collect.py: runs the tests, joins targets with results, runs the validity checks, and gives the mechanical verdict as JSON.
  • references/root-causes.md: failure patterns, a catalogue of causes with cheap checks, and how to rank hypotheses.
  • assets/report_template.md: report skeleton.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.