Pre-Registration Protocol

Every test you run against a dataset spends statistical credibility. This skill makes the spending explicit, budgeted, and irreversible — so that when something finally passes, it means something.

When to use

• Before ANY test whose outcome will influence a ship / kill / iterate decision: backtests, A/B tests, ablations, feature evaluations, model comparisons, pricing experiments, conversion tests.
• Whenever a new "variant", "tweak", "quick sanity check", or "what if we just..." is proposed against a dataset that previous tests have already touched.
• When resuming a research program after any gap: the FIRST action is to read the trial log and check remaining budget, not to run code.
• When the user asks the agent to "optimize", "sweep", or "search" over parameters — a sweep of 50 configs is 50 trials, not 1.

When NOT to use

• Pure engineering verification: does the pipeline run, does it produce identical numbers twice, does the loader parse the file. Testing infrastructure is not testing a hypothesis.
• Genuinely exploratory profiling — ONLY if you write down, before looking, that nothing seen will be treated as evidence. Anything interesting found during exploration must be confirmed on data the exploration never touched, and that confirmation costs a trial.
• Live monitoring against thresholds that were themselves pre-registered. That is execution of a prior registration, not a new test.

The procedure

Execute these steps in order. Steps 1–6 happen BEFORE any outcome is computed. If any outcome has already been seen, the trial is exploratory by definition — label it so in the log and do not use it as evidence.

1. Write the hypothesis as one falsifiable sentence with a direction and a magnitude. Bad: "volume features might help." Good: "Adding feature X to model M improves out-of-sample metric Y by at least Z on window W." If you cannot state the magnitude, you are not ready to test — you are ready to explore, which is a different (and budgeted) activity.
2. Define the success gate: metric + threshold + evaluation window + multiple-testing correction, all numeric. Example: "OOS Sharpe >= 2.0 on the held-out 180 days, AND permutation p < per-trial alpha from step 5, AND the result survives realistic costs." A gate with an escape hatch ("or shows promise") is not a gate.
3. Define the kill gate and the outcome categories. Exactly three outcomes are allowed, and they are decided by the numbers, not by the write-up: SHIP (passes success gate), REJECT (fails), BACKLOG (passes some pre-named intermediate condition, e.g. "positive and orthogonal but below gate — re-test only on NEW data"). Define what BACKLOG requires now, or everything mediocre will end up there.
4. Fix the sample before running: exact data window, exact universe, exact train/test split. Write the file paths or query. Changing the window after seeing a result is a new trial against a contaminated dataset.
5. Declare the trial number against a fixed budget. Before the program starts, set N — the total number of confirmatory trials this dataset will ever support — and derive the per-trial threshold:

# Family-wise (Bonferroni) budget for a research program
alpha_family = 0.05
N_budget = 40                      # decided ONCE, before trial #1
per_trial_alpha = alpha_family / N_budget   # 0.00125

# Why a budget exists at all: unbudgeted search manufactures winners.
# 300 independent tests at p < 0.05 on the same data:
expected_false_positives = 300 * 0.05        # 15 "discoveries" from pure noise

Benjamini–Hochberg FDR is acceptable instead of Bonferroni if declared up front — but the budget N is the load-bearing part, not the correction formula. Every trial header reads "Trial #k of N=40".

6. Freeze the registration. Commit the pre-reg entry to version control BEFORE running, and reference that commit hash in the results entry. The freeze must be mechanical: a registration whose timestamp postdates the first result file is not a registration, it is a rationalization with formatting.
7. Run once. One execution of the registered test. If the code crashes, fix and rerun — bug fixes are free. Changing a parameter because the first number looked bad is not a bug fix; it is trial #k+1.
8. Record the outcome verbatim in the log: the actual numbers, the category (SHIP/REJECT/BACKLOG) forced by the gates, and one line of interpretation. Rejections get logged with the same care as ships — the rejection log is what stops you re-testing the same dead idea eight months later.
9. When the budget hits N: STOP testing on this dataset. Not "stop unless it's really promising." More search on the same data can only manufacture false positives; the remaining moves are (a) collect new data, (b) take a BACKLOG item to new data, (c) ship what passed, (d) close the program. A budget you extend when you hit it was never a budget.

Common failure modes

(a) "Just one more variant." A research program ran well over 300 strategy variants against the same 4-year dataset. Each individual test looked disciplined; collectively they were a search process, and search processes select for luck. The best-looking configuration was deployed with real money and delivered 25–50% below its backtest — the gap was not an execution bug, it was selection bias coming due, discovered only after live losses. Wrong way: treating each new variant as an independent, innocent question. Right way: every variant decrements the same counter; when the counter says 40/40, variant #41 does not exist. The honest expected live performance of a budget-exhausted program's best pick is well below its backtest — plan capital and expectations accordingly.

(b) Moving the goalposts. The gate said "OOS Sharpe >= 2.0"; the result came in at 1.6; the write-up said "promising — ship at reduced size." That is a REJECT being laundered into a SHIP by prose. Wrong way: renegotiating the gate while looking at the number it gates. Right way: a miss is a miss. If you now believe the gate itself was mis-set, that belief is a new hypothesis: register it as a new trial, with the new gate, on data the old result did not touch.

(c) The exploratory peek treated as confirmatory. You plot the data, notice a pattern, then "test" that pattern on the same data and get p = 0.003. That p-value is fiction — the hypothesis was chosen BY the data it was tested on. Wrong way: "we found it AND confirmed it in one afternoon." Right way: exploration and confirmation on disjoint data, always. If you peeked, say so in the log, and let the confirmation run on a window the peek never saw — and pay a trial for it.

(d) Budget accounting that only counts survivors. Formal write-ups undercount. The "quick sanity check" that influenced which variant you formalized was a test. The abandoned sweep was 30 tests. One honestly-kept log revealed the true trial count was roughly 3x the number of formal reports. Right way: the log counts every computation whose output a human looked at and reacted to.

(e) Retroactive registration. Writing the pre-reg after the result, dated before it. This is the one failure mode that is fraud rather than error. The mechanical freeze (step 6) exists so that future-you cannot do this even on a bad day.

Trial-log template

Keep one markdown file per program. Append-only. Example entry:

# Trial Log — <program name>
Budget: N = 40 confirmatory trials, alpha_family = 0.05, per-trial alpha = 0.00125
Dataset: <exact source, window, universe — frozen at program start>
Status: 37/40 used. 3 remaining.

---
## Trial #37 of N=40 — <short hypothesis name>
- **Registered:** 2026-04-23, commit `a1b2c3d` (pre-result)
- **Hypothesis:** <one falsifiable sentence with direction + magnitude>
- **Success gate:** <metric> >= <threshold> on <window>, p < 0.00125 (<test>)
- **Kill gate / categories:** SHIP if gate passes; BACKLOG iff <pre-named condition>; else REJECT
- **Sample:** <paths / query / split — fixed before run>
- **Result (verbatim):** <numbers as produced>
- **Category:** REJECT
- **One-line interpretation:** <why, for future-you>
- **Contamination notes:** <any peeks, reruns, or parameter touches — be honest>

Checklist

Before running:

• Hypothesis is one sentence, falsifiable, with direction and magnitude
• Success gate is numeric: metric + threshold + window + correction
• Kill gate and BACKLOG condition defined; only three outcomes possible
• Sample/window/split fixed and written down
• Trial numbered against the fixed budget ("#k of N") — sweeps counted per config
• Registration committed to version control before first result

After running:

• Outcome recorded verbatim; category forced by gates, not by prose
• Any peeks or mid-run changes disclosed in the log
• Budget counter updated; if N reached — no further tests on this dataset, ever