Experiments writing

Iron rule

Every number printed in the paper (text, table, figure, caption, footnote, abstract) has an EVIDENCE.md row: ID | number as printed | command | log path | commit | machine | date | runs/median. Tables are generated from the data files behind those rows (e.g. with the crbench table generator), not typed. Put % EV:<ID> comments next to hand-placed numbers so a script can cross-check.

Procedure

1. Freeze the data. Ask experimenter for the final json/csv and the EVIDENCE rows. Anything measured after the freeze requires a new row and a note in DECISIONS.md.
2. Decide the scope sentence for every range ("on the 9 parameter sets at ≥128-bit estimated security"). Scope comes from the parameter/security table, generated.
3. Write the section following paper-playbook/sections/10_experiments.md.
4. Generate tables (below). Mark the best cell per column (bold, optional light colour); two-level headers with \multicolumn; captions define every abbreviation and symbol.
5. Write the fairness statements (below) that apply.
6. Write boundary/limitation remarks (below).
7. Cross-check script: crbench audit-tex paper/main.tex EVIDENCE.md (lib/bench), or extract every number from the PDF text (pdftotext -layout) and match it to EVIDENCE rows; unmatched numbers are errors unless they are parameters or citations. Record the result in SUBMISSION.md.

Table set

Layout: \scriptsize, \arraystretch ≈ 0.92, \tabcolsep ≈ 2.6pt; tables at [t]; table captions above tables, figure captions below figures.

Fair-comparison statements (use the ones that apply)

• Same pipeline: "All ratios are against <baseline> run in our pipeline with the same <radix/parameters/library commit>; only <stage> differs."
• Measured vs claimed: "Rerunning the artifact of [X] on our machine reproduces their claimed a–b× as c–d×; we report both."
• General caution before cross-paper tables: "Such comparisons must be read with care because of differences in libraries, hardware and failure probabilities; we mark numbers taken from the original papers with ◇."
• Metric orientation remark: "Our goal is throughput; for latency-critical applications <other scheme> remains preferable."
• Parallelism remark: "All our numbers are single-threaded. Evaluating independent instances in parallel helps every scheme equally, so single-thread numbers compare designs, not hardware."
• Security-level parity: "Both our parameters and the baseline's are estimated at λ bits with <estimator commit>; where the estimator has moved since the baseline was published, we re-estimate both."
• Baseline strength: "We use the fastest configuration of <library> for the baseline (<flags>), not the default one." (Reviewers and shepherds check this.)

Boundary / limitation statements

• Exclusions: "The k instances below the claimed security level are excluded from the headline ranges and flagged in Table <n>."
• Overheads: "The composed evaluator consumes 1.2–1.5× [ill.] the baseline's memory/capacity."
• Regime where we lose: "For t ≥ 7 [ill.], [X] is faster because ...; the regimes do not overlap." Say it before a reviewer does.
• Not applicable: "Our method requires <condition>; for instances violating it the implementation falls back to the baseline (log shows '<message>')."
• Measurement noise: "Absolute times vary with machine load across sessions; all ratios come from interleaved back-to-back runs in one session."

Measurement hygiene to report (and to follow)

• Interleave A/B runs (ABAB...) in the same session; report median and IQR/CI.
• No concurrent heavy jobs; use a bench lock; pin threads.
• Never mix numbers from different sessions in one ratio; if an earlier number is not reproducible, retract it explicitly (and say so in the rebuttal if already quoted).
• Decryption/output verification for every run.

Output

paper/sections/experiments.tex, generated tables in paper/tables/, the cross-check report in SUBMISSION.md.