CLAUDE.md

A bench for faithfully reproducing scientific papers with Claude Code skills. See README.md for the user-facing overview.

How to answer

• Get straight to the point. Lead with the answer or the result, skip preamble and recaps, and keep explanations short.
• Illustrate whenever possible. Prefer a diagram, table or image to prose: mermaid diagrams in Markdown files, ASCII schemas in the terminal, plots (PNG) for results and learning curves, and tables for paper-vs-local numbers.

Pipeline

paper ──/paper-to-specs-and-tests──▶ spec/<id>.md + tests/<id>/
spec  ──/implement-paper───────────▶ papers/<id>/ + results/<id>/*.json
tests + results ──/test-paper──────▶ reports/<id>.md + README "Reproduced papers" row

Each stage only reads the previous stages' files. Skills live in .claude/skills/<name>/.

Rules that must hold

1. Never edit the spec without asking the user. Propose the change (with its source) via AskUserQuestion. After approval, edit the spec and append a row to its ## 10. Change log. Unapproved values may be used provisionally for cheap work, never for full runs.
2. Official code first. papers/<id>/official/ is a git submodule pinned to the spec's commit. Never commit edits inside it: save changes with papers/<id>/save_patch.sh <name> as patches/NNNN-<name>.patch, and restore with apply_patches.sh. Write an own implementation only when the spec's §2 decision says so.
3. Ask before full training runs. Smoke runs and released-checkpoint evaluations (Stage 0) are fine. Full runs need a compute estimate and the user's approval. The GPU is shared, so check nvidia-smi first.
4. Never stop a GPU process you didn't launch without asking the user, even to free memory for a run. Show the PID, its owner and command, then ask. You may stop your own processes, i.e. ones started in this session or recorded with their PID in a REPRODUCTION_LOG.md Runs table. Check that the PID and command still match first, since PIDs get reused, and log the stop. Running under the same Unix user doesn't make a process yours.
5. Results only via write_results.py. Never hand-write results/. The script records the provenance that test-paper checks.
6. Tests are the definition of "reproduced". Never widen a tolerance, change a paper_value or remove a target to make a test pass. Such a change needs a spec change-log entry approved by the user, otherwise test-paper makes the verdict INCONCLUSIVE.
7. test-paper is read-only. It diagnoses but never edits spec, tests, results or code. Fixes go back to implement-paper.
8. Report honestly. A near-miss is a fail. A value far above the paper is suspicious (possible leakage), not a success.
9. Detach every run longer than ~1 minute from the ssh session, so it survives a disconnect or the session shutting down. Launch it with setsid nohup <cmd> < /dev/null > <run_dir>/stdout.log 2> <run_dir>/stderr.log & (own session, no TTY, SIGHUP ignored), or in tmux. Write its PID to <run_dir>/pid and to the log's Runs table. Never run long jobs in the foreground of a tool call or as a plain background child of the shell.

Conventions

• paper_id is <first author last name><year>_<keyword> in snake_case, e.g. vaswani2017_transformer. It is shared by every file of the paper.
• Spec (spec/<id>.md, from paper-to-specs-and-tests/assets/spec_template.md): §2 implementation strategy, §6 targets (absolute, relative claims, proxies), §7 results contract, §8 assumptions, §10 change log. Every number cites its source (table, figure, section, config).
• Tests (tests/<id>/test_<id>.py): Target(id, experiment, metric, paper_value, tolerance, source, priority) and Claim(id, greater, lesser, min_gap, source). They skip when a results file is missing. Every target and claim ID must appear in backticks in the spec. REPRO_RESULTS_DIR overrides the results folder.
• Results (results/<id>/<experiment_id>.json, tracked in git): paper_id, experiment_id, metrics, per_seed, implementation (official / official-modified / own), commit, bench_dirty, official_commit, patches, patches_sha256, config, run_dir, written_at, notes. Smoke experiments have IDs ending in _smoke.
• Per-paper work log: papers/<id>/REPRODUCTION_LOG.md (status, pending spec questions, deviations, environment, runs). Read it first when resuming.
• Verdicts: REPRODUCED, PARTIALLY REPRODUCED, NOT REPRODUCED, INCONCLUSIVE. The model may downgrade a mechanical verdict, never upgrade it.

Environment

• Root .venv (Python 3.12, pytest) runs the bench tests and the skill scripts. Each paper has its own papers/<id>/.venv created by setup_env.sh (uv).
• GPU: NVIDIA RTX PRO 6000 Blackwell (98 GB, shared with other users). It needs torch ≥ 2.7 built with CUDA ≥ 12.8.
• Not installed: gh, pdftotext.
• Paper lookup: the arXiv API and OpenAlex work. Semantic Scholar rate-limits without S2_API_KEY, and DBLP is blocked from this machine. Papers with Code has shut down, so use huggingface.co/papers/<arxiv_id> to find code.
• Large artifacts (data, checkpoints, runs, papers/*/source/) are gitignored.

Commands

.venv/bin/python -m pytest tests/<id> -v                                            # quick check
.venv/bin/python .claude/skills/paper-to-specs-and-tests/scripts/smoke_test.py <id>  # validate a new test suite
.venv/bin/python .claude/skills/test-paper/scripts/collect.py <id> --out <file>     # evidence + verdict