Pre-Analysis Plan

Overview

If you can run a hundred reasonable specifications and report the one you like, you haven't measured an effect — you've measured your own preferences with extra steps. A pre-analysis plan (PAP) is the commitment that stops this: write down what you will do, and what would count as the answer, before the outcomes are visible. It's the analytics counterpart of writing a spec before code — decide the contract first — but the stakes are higher: the temptation to retrofit the plan to the result is enormous and almost invisible after the fact.

Core principle: A specification chosen after seeing the outcome is not evidence. Decide the analysis while you are still ignorant of how it will turn out.

When you actually need this

Not every exploration needs a PAP — forcing one onto genuine EDA is theater (see question-framing and the exploration regime in data-contracts). It earns its weight when:

• The result will be used to make or defend a decision (ship the feature, keep the policy, publish the finding).
• You or the requester have a stake in the result coming out a particular way.
• The analysis is causal — identification claims are exactly where specification search does the most damage.
• The work will be scrutinized — a referee, a regulator, a skeptical exec, a replication.

If none of those hold, you're exploring; label it exploratory and move on.

What the plan locks (before seeing outcomes)

1. Hypotheses. Stated directionally and ranked: primary vs. secondary. Test the primary one; treat everything else as secondary or exploratory. Pre-commit the multiple-comparisons correction for secondary tests (Bonferroni / Benjamini–Hochberg / etc.) — uncorrected, they reintroduce the forking-paths problem the PAP exists to stop. Budget the families: one to a few primaries; a secondary family in the order of ten slots, not dozens. Individual event-time coefficients are exhibits of a pre-registered aggregate, not separate slots. A count that measures coverage, match rates, or data quality is a diagnostic, never a hypothesis — it does not enter a test family or a correction.
2. Estimand. The exact quantity (ATE/ATT/LATE/ITT), on the exact population, over the exact window. Reuse the question-framing brief.
3. Primary specification. One pre-committed model: functional form, controls, fixed effects, SE structure (and clustering level), inference method — the number you will report. Robustness specs support it; they don't replace it when you like them better. Write the primary specification to the presentation standard in causal-identification — equation with subscripts defined, level of variation, intuition, literature precedent — not an estimator name-drop. Inference defaults to analytic standard errors clustered at the level treatment is assigned, following the named precedent — if the precedent ships analytic clustered SEs, so do you. A bootstrap or other simulation-based inference is a deliberate departure to state and justify (the precedent has no analytic variance; the clustering structure has no analytic estimator), never a default: it multiplies runtime by the number of draws and buys nothing the precedent didn't already need.
4. Sample and exclusions. Inclusion criteria, exclusion rules, and how outliers and missing data are handled — decided now, by rule, not later by eye. "Drop obvious outliers" after seeing the data is a degree of freedom; "drop values beyond 3 IQR, pre-committed" is a rule.
5. Robustness suite. The alternative specs, placebo/falsification tests, and sensitivity analyses you commit to run regardless of whether the primary result survives them — you can't quietly drop the ones that disagree. Keep it small and targeted: two or three checks that probe the load-bearing assumption, not a catalogue; a pre-registered buffet is still a buffet. (causal-identification's design-specific diagnostics are mandatory and separate from this discretionary suite; "small" governs the discretionary specs only.)
6. Decision rule and power. What result leads to what action, and what counts as the effect being absent — define the null outcome so a null is a finding, not a prompt to keep digging. Commit too to the minimum detectable effect / power the design has: a null from an underpowered test isn't evidence of no effect, and saying so up front stops a noisy null from being read as a clean one.
7. Measured feasibility — before the lock, not after. Every item above is a claim about what the data can support, so measure it on non-outcome data first: treated units per cohort, identifying variation for each interaction the primary relies on (repeated pairs, within-group contrast support), cluster counts at the clustering level, and support for each outcome's denominator. Report the counts in the plan. A primary with no measured identifying variation is not lockable — rewrite it or drop it now, while doing so is design work rather than a deviation. This is blinding-safe: it touches sample structure, never outcomes.

Write it down and get sign-off before touching outcome data

The PAP's trigger line is stricter than the shared gate's (analysis-checkpoints owns the write→sign-off mechanics): sign-off fires at touching outcome data, not merely at "estimation." Loading the outcomes, plotting their distribution, or peeking at the treatment–outcome relationship is itself the blinding violation — once seen, every later "choice" is contaminated, so the user signs off while everyone is still blind. Persist the PAP to a file (e.g. docs/pre-analysis-plan.md) and register it in docs/analysis/index.yaml via analysis-state-management; don't write it and proceed on your own reading.

Confirmatory vs. exploratory — keep the line bright

You will discover interesting things you didn't pre-register. That's good — it's where new hypotheses come from. The sin is laundering them as confirmatory. Report them, clearly flagged as exploratory and hypothesis-generating, with the understanding that they need fresh data to confirm. A finding that has been both used to form a hypothesis and to test it has been counted twice.

Pick one label and let it set the machinery. If prior results have already been seen, the study is exploratory, and exploratory work gets lighter machinery: a stated primary, analytic SEs, a few labeled checks, reported as exploratory. Bolting a full confirmatory battery — large corrected families, bootstrapped inference, an exhaustive sensitivity suite — onto a study you have labeled exploratory buys no credibility; it spends days of compute to dress up a claim the label has already qualified, and the weight of the apparatus is what makes the reader miss the label. Confirmatory or exploratory, not both.

The garden of forking paths

Even with no conscious cheating, the sheer number of defensible choices — which controls, which window, which subgroup, how to handle outliers — means somewhere in that garden is a significant result, and you'll tend to wander toward it. The PAP prunes the garden to one path chosen in advance. When the data surprises you and a departure seems warranted, that is a checkpoint, not a judgment call you make on your own: stop, bring the proposed deviation and its rationale to the user, and report both the pre-registered and the revised analysis once they agree (see analysis-checkpoints). Deviation approved and disclosed is science; deviation taken silently — even with good intentions — is fishing.

Red flags — STOP

• You've seen the outcomes and now you're deciding which controls to include or which subgroup to feature.
• The robustness checks reported are exactly the ones that agreed with the headline — you can't say what happened to the others.
• "We'll know the right specification once we see the data" — for a confirmatory claim, that's the forking-paths trap.
• An exploratory finding is about to be presented with the confidence of a pre-registered test.
• No stated null — no result that would have counted as "no effect."
• A test family you could not list from memory. If you can't name the slots without rereading the plan, no reader will hold them either — and the correction is now doing the thinking for you.
• An outcome that is zero almost everywhere sitting in the family — a near-empty count is a data-quality diagnostic that wandered into the hypotheses; it cannot move and it will drag the whole family's correction.

Common rationalizations

The Process

1. Measure feasibility on non-outcome data, then lock the seven items — hypotheses (+ family budget and comparisons correction), estimand, primary spec, sample/exclusion rules, robustness suite, decision rule/power, measured feasibility. The counts come first: they decide which hypotheses are lockable. Use the question-framing brief and the sample rules data-contracts enforces.
2. Persist the PAP to a file and get explicit sign-off before touching outcome data — mandatory, not rhetorical: once outcomes are seen, the lock is gone.
3. Route to exactly one next step. Structural/counterfactual work → invoke structural-estimation for the model card. Otherwise → invoke executing-analysis-plans to carry out the locked plan (causal-identification runs the design diagnostics there).
4. If the outcomes surprise you and a departure tempts → STOP and invoke analysis-checkpoints — report pre-registered and revised analyses; never deviate silently.

The bottom line

Confirmatory claim  →  hypotheses, estimand, primary spec, sample rules, robustness suite, decision rule — all fixed before outcomes seen, with feasibility measured first
Otherwise           →  exploratory; label it so, keep the machinery light, and confirm on fresh data