experiment-reproducibility

A result that cannot be regenerated from a command is an anecdote. Reproducibility is cheap at the start and impossible at the end, so this is set up before the first run that will appear in the thesis.

Layout

experiments/
  env/          environment.yml or requirements.txt with exact pins; Dockerfile if system deps matter
  configs/      one YAML per experiment: exp01-baseline.yaml, exp02-ours.yaml
  data/         raw/ (never modified), processed/ (generated by scripts), DATA.md (source, licence, version, checksum)
  src/          the code; run.py takes --config and --seed
  runs/         one directory per run: <exp>-<seed>-<timestamp>/ with manifest.json, metrics.json, log.txt
  results/      aggregated tables (CSV) generated from runs/ by aggregate.py
  figures/      generated from results/ by figure scripts (research-figures)
  Makefile      or justfile: `make exp01` runs all seeds; `make results` aggregates; `make figures`

Everything under runs/, results/, and figures/ is generated; regenerate rather than edit. Commit results/ and figures/ (they are small) so the thesis builds from a clean clone; do not commit runs/ unless small (use DVC or a release asset otherwise).

Environment

• Python: conda env export --no-builds or pip freeze into env/, with the Python version. Better: pyproject.toml with a lockfile (uv lock, poetry lock).
• Record the CUDA/driver version and the exact framework build when GPUs are involved; results differ across them.
• A Dockerfile when anything outside the language environment matters (system libraries, a database).
• The manifest records python --version, pip freeze hash, nvidia-smi output, and uname -a at run time regardless.

Determinism

• One --seed argument that seeds every source: random, numpy, torch (and torch.cuda.manual_seed_all), tf, data shuffling, and worker init. Use a helper seed_everything(seed) called first.
• Set deterministic flags where the framework offers them (torch.use_deterministic_algorithms(True), CUBLAS_WORKSPACE_CONFIG) and record whether they were on; some ops stay nondeterministic on GPU and the thesis should say so.
• Run every experiment with at least three seeds (five or ten when cheap); report mean and SD or CI (statistics-advisor). A single-seed number is not a result.

Configuration

• All parameters in the config file; nothing hard-coded, nothing from the command line except --config and --seed (and --out). The config is copied into the run directory unchanged.
• Named experiments, not edited-in-place configs: exp03-ours-lr0.001.yaml rather than changing exp02 and losing what produced the earlier result.

The run manifest

runs/<run>/manifest.json, written at start and completed at end:

{
  "experiment": "exp02-ours", "seed": 1, "started": "2026-09-21T10:00:00Z", "finished": "...",
  "git": { "commit": "a1b2c3d", "dirty": false, "branch": "main" },
  "config_sha256": "...", "data_version": "v3 (sha256 ...)",
  "env": { "python": "3.12.4", "packages_sha256": "...", "cuda": "12.4", "gpu": "A100 40GB", "host": "..." },
  "duration_s": 1832.4, "status": "ok"
}

Refuse to start a run that will appear in the thesis from a dirty working tree (git status --porcelain non-empty): commit first, so the commit hash means something. Log the refusal.

Data

data/DATA.md: source URL, download date, licence, version, checksum of the raw archive, the preprocessing script and its parameters, the resulting row counts and split sizes. Splits are made once with a seed and saved as index files; every run reads the same splits.

Aggregation and figures

aggregate.py reads every runs/*/metrics.json, joins with the manifests, and writes results/<table>.csv with one row per (experiment, seed) and a summary with mean, SD, n. Figure scripts read only results/. The thesis \inputs tables generated from results/ (benchmark-reporting), so a rerun updates the document.

One command

make all (or just all): environment check, all experiments for all seeds, aggregate, figures, tables. Document it in experiments/README.md with the expected total runtime and hardware. The examiner, or a future student, runs it and gets the thesis's numbers.

Checklist before a result enters the thesis

• Run from a clean commit, manifest present, seeds ≥ 3.
• Config committed, data version recorded, environment pinned.
• Table and figure generated from results/, not typed.
• The commit hash cited in the thesis appendix.