Implement a paper from its spec

Input: a paper_id (or a paper the user names, mapped to its spec/<paper_id>.md). Output: working code in papers/<paper_id>/, results in results/<paper_id>/<experiment_id>.json, and an up-to-date papers/<paper_id>/REPRODUCTION_LOG.md, with pytest tests/<paper_id> as the judge.

The spec is the contract. It was written from the paper, and the tests are derived from it. Your job is to make the code meet the spec faithfully, not to make the tests green. A reproduction that honestly fails is useful; one that quietly changed the method, the protocol or the targets is worse than none.

0. Preconditions

• spec/<paper_id>.md and tests/<paper_id>/test_<paper_id>.py exist. If not, stop and offer the paper-to-specs-and-tests skill first.
• Read the whole spec and the test file before writing code. Pay most attention to: §2 (implementation decision and fallback trigger), §4 (setup), §5 (evaluation protocol), §6 (targets), §7 (results contract), §8 (assumptions).
• If papers/<paper_id>/REPRODUCTION_LOG.md already exists, this is a continuation: read it, pick up from its Status table and open spec questions, and don't redo finished steps.

1. Official code first

Use the official implementation whenever it is usable. It contains the details papers leave out, and reproductions built on it are both more faithful and cheaper. Concretely, by the spec's §2 decision:

• Base on official repo → bring it in as a submodule and get it running (porting it to a current stack if needed). Porting working code beats rewriting it.
• Official repo as reference only → write own code in src/, but still add the submodule (or keep a clone in the scratchpad) and read the official code for every detail the spec leaves open. Cite official/<file>:<line> in comments where you follow it.
• Own implementation → first spend a minute checking whether official code has appeared since the spec was written (the paper's page, the authors' GitHub, huggingface.co/papers/<arxiv_id>). If it has, that is a spec update: propose switching (step 3). Otherwise implement in src/.

Changing strategy (e.g. the spec's fallback trigger fires and you want to drop the official code) is a spec change too. Gather the evidence, then ask.

Read references/official-repo-workflow.md before touching the official code. It covers the submodule + patches workflow, the per-paper environment, porting old code, and reading third-party code safely.

2. Scaffold the workspace

python .claude/skills/implement-paper/scripts/scaffold_paper.py <paper_id> \
    [--official-url <github url>] [--sha <commit>] [--python <version>]

It creates papers/<paper_id>/ with official/ (submodule at the spec's pinned SHA, read from the spec header when --sha is omitted), patches/, src/, configs/, tests/, scripts/{setup_env,apply_patches,save_patch}.sh and REPRODUCTION_LOG.md. Then edit scripts/setup_env.sh to install what the code needs and run it. The .venv lives in papers/<paper_id>/.venv, so each paper gets the library versions its code needs.

Layout rules that keep reproductions reviewable:

• Changes to upstream code exist only as patches/NNNN-<name>.patch (create them with scripts/save_patch.sh <name>). One concern per patch.
• Own code, wrappers and entry points go in src/, configs in configs/ (one per experiment_id), and unit tests of own code in papers/<paper_id>/tests/.
• Data, checkpoints and run outputs go outside papers/ (data/, checkpoints/, runs/, all gitignored), never inside official/.

3. When the spec is not enough: propose, then ask

You will hit details the spec doesn't settle: a missing hyperparameter, an ambiguous preprocessing step, an official config that disagrees with the paper, a target that looks mis-copied. Resolve them in this order:

1. Look it up. Check the official code first (configs, defaults, training script), then the paper itself. Fetch it again if needed, following the paper-to-specs-and-tests skill's references/fetching.md (published version first, the arXiv LaTeX source for tables). Note exactly where the answer is.
2. Write the proposal: the spec section, the current text, the proposed text, the source, and what it affects (code only, or targets/tests too).
3. Ask the user before editing the spec. Show the proposal as a short diff in your message, then ask with AskUserQuestion. Options: Apply, Apply as an assumption (goes in §8 with its uncertainty), Keep the spec as is. Group related questions, up to 4 per call. Don't ask one at a time if several are pending.
4. Until the user answers, don't block on it when you can avoid it. Use the proposed value provisionally if the work is cheap to redo (it shows up as provisional in the log's Spec questions table), and continue with other parts. Block only when a wrong guess would waste real compute: never launch a full run on a provisional value.
5. After approval, edit the spec, append a line to its ## Change log section (create it at the end if missing: date, section, change, source, "approved by user"), and mark the question applied in the log. If the change touches §6 (targets, tolerances, claims), update TARGETS/CLAIMS in the test file to match and run python .claude/skills/paper-to-specs-and-tests/scripts/smoke_test.py <paper_id>.

Why so strict: the spec and tests are the independent yardstick. If the implementer can change them freely, the yardstick moves toward whatever the code produces, and a passing test no longer means anything. The user has to own every change to what "reproduced" means.

Spec changes covered by this rule include target values, tolerances, experiments, the protocol, the implementation strategy, the pinned commit and filled gaps. Don't ask about choices that are purely about code (file layout, refactors, logging), and don't put them in the spec.

4. Implement faithfully

• Follow the spec, not your taste. Don't fix what looks like a flaw in the method, swap in a "better" optimizer, or modernize the architecture. If the official code and the paper disagree, that is a spec question (step 3).
• Own code mirrors the paper: name things as the paper does, reference equations (# Eq. (3)) and sections in comments, and keep the method's structure visible. Add focused unit tests in papers/<paper_id>/tests/ for the parts that are easy to get subtly wrong: loss terms on hand-computed inputs, masking, shapes, schedules. When official code exists, compare your module's output against it on the same small input.
• Results: every experiment ends by writing its result file with

  python .claude/skills/implement-paper/scripts/write_results.py --paper-id <paper_id> \
      --experiment <experiment_id> --per-seed <metric>=v1,v2,v3 \
      --implementation official|official-modified|own --config <path> --run-dir <dir> --notes "<deviations>"

  It records provenance (bench commit, official commit, patch hash). Use the metric keys and units of spec §7 exactly. A unit mismatch (fraction vs %) is the most common way to fail a correct reproduction.
• Log as you go: every patch and deviation goes in the log's Deviations table, every spec question in Spec questions, and environment facts in Environment.

5. Run: cheap checks yourself, full runs after asking

Read references/running-experiments.md. In short:

1. Smoke run of each experiment pipeline on tiny data, written under a _smoke experiment ID. Measure throughput here.
2. Stage 0: when checkpoints are released, evaluate them with the official eval code and run the tests. If they don't match the paper, fix the eval/data pipeline before any training.
3. Compute estimate per experiment from measured throughput. Show the user a table and ask which full runs to launch (AskUserQuestion). Training jobs hold shared GPUs for hours or days, so that is the user's call.
4. Full runs on idle local GPUs (check nvidia-smi), detached with nohup, one run dir per run under runs/<paper_id>/<experiment_id>/, each recorded in the log.
5. After each run: evaluate on the spec's protocol, write results, then run python -m pytest tests/<paper_id> -v.

A failing test is information, not something to optimize away. Never widen a tolerance, select checkpoints or hyperparameters on the test set, drop bad seeds, or rerun until a seed lands in range. Look for the cause; if none is found, report the miss with what you checked.

6. Report

End each session with a short report to the user, and update the log's Status table to match:

• per experiment: stage reached, and local value vs paper value ± tolerance (from the pytest output)
• the implementation route actually used, and the patches and deviations that can affect numbers
• spec questions: applied, pending (what you need from the user), provisional
• what is still to run, with the compute estimate, and anything blocked

Files in this skill

• scripts/scaffold_paper.py: creates papers/<paper_id>/, adds the official submodule at the pinned SHA, and writes the env/patch scripts and the log.
• scripts/write_results.py: writes a results file in the spec's contract, with provenance.
• assets/log_template.md: REPRODUCTION_LOG.md skeleton.
• references/official-repo-workflow.md: submodule + patches, environments, porting old code, security.
• references/running-experiments.md: smoke runs, Stage 0, compute estimates, launching and monitoring on local GPUs, honest results.