Experiment Discipline

Write results to disk, always

Every run writes a file under results/. Not to stdout, not to a variable that lives in the agent's context, not to a summary you will remember. The paper cites files. If it is not on disk, it did not happen.

results/
  exp1_results.json     # metrics, one object per condition
  exp1_raw.csv          # per-sample outputs, for later re-analysis
  environment.json      # versions, seeds, hardware

Include the seed and the library versions. A number without them cannot be reproduced, and reviewers ask.

Baselines are part of the experiment

A method with no comparison is not an evaluation. Implement at least one honest baseline, and give it the same tuning budget you gave your own method. A hobbled baseline is a well-dressed way of faking a result.

If a published baseline number exists and you did not rerun it, say it is quoted from the paper rather than measured by you. Mixing quoted and measured numbers in one table without saying which is which is a common, serious error.

Respect the compute backend the user chose

If the project is configured for a cluster, submit to the cluster. A trivial CPU check can run inline, but say which route you took in your notes. Otherwise the user, who deliberately selected Slurm, sees nothing in the queue and cannot tell whether their configuration is broken.

Observed: an agent ran every experiment inline because its instructions said "use Slurm for GPU jobs" and the experiment was CPU-only. That was a reasonable reading, but the decision was invisible, and it looked like a platform bug for days.

Use the project's own environment

Install into the project environment, never a shared or global one. See env-isolation. This is not tidiness. A single stray install into a shared environment once downgraded a core library and silently broke every user of that machine for five days.

Isolate what you run

If you execute third-party code, benchmark harnesses, or anything you did not write, run it in a container. It protects the host, and it makes the environment a fact you can report rather than a property of your laptop.

Failure is data

When a run fails, record the failure with its error, and move on to the next experiment rather than quietly dropping it. An experiment that disappears between the plan and the results section is the shape of a result that was inconvenient.

Before writing the paper

• Every planned experiment is either done or explicitly marked blocked
• Every result is a file under results/
• Seeds and versions are recorded
• Baselines got a fair shot
• Quoted numbers are labelled as quoted