reproducibility-check

Most reproducibility failures are not exotic. They are an undeclared dependency, a hard-coded path, or a file that was never committed.

Work from a clean checkout, always

Clone into a new directory. Not your working copy — your working copy has untracked files, cached builds and environment variables that the reader will not have.

git clone <url> /tmp/repro-check && cd /tmp/repro-check

If you cannot clone it fresh because something is gitignored that shouldn't be, you have already found the bug.

The four failure classes

1. Undeclared dependency — it works for you because it is installed globally. Check in a container with nothing preinstalled.
2. Hard-coded path — /Users/you/... or C:\Users\... anywhere in the source or config. Grep for the home directory literal.
3. Uncommitted file — a config, a fixture, a small data file. git status in your working copy after a successful run often reveals it.
4. Unpinned version — "latest" resolved differently today than it did when the results were produced. Check for a lockfile.

Pin the environment

Record the versions that actually worked: language runtime, package manager, OS, and any system libraries. A Dockerfile or a devcontainer is the strongest form of this, because it is executable rather than aspirational.

Verify the output, not just the exit code

A script that finishes without error has not necessarily reproduced anything. Check the produced artifact against the claimed result — the number in the paper, the committed reference output, the expected file.

Where the output is numeric and not bit-identical across platforms, state the tolerance rather than pretending to exactness.

Report as a sequence someone can replay

The useful output of this check is a list of commands, in order, from clone to verified result. If you cannot write that list, the artifact is not yet reproducible — regardless of whether it worked for you.