Experiment Design

An experiment exists to change a decision. If no result would change what you do, don't run it. Design backward from the decision to the measurement.

Specify before you collect (pre-registration discipline)

1. Decision and threshold: "If metric X moves by ≥ δ, we ship." Define δ from business value, not statistical convenience.
2. One primary metric: the thing the decision hinges on. Everything else is a guardrail or exploratory. Multiple primaries = multiple comparisons = inflated false positives.
3. Hypothesis and mechanism: not just "B is better" but why B should move the metric. A result without a mechanism is a coincidence waiting to fail to replicate.
4. Power analysis up front: required sample size = f(baseline rate, minimum detectable effect, α, power). Running until significant ("peeking") manufactures false positives.

Causal validity is the whole game

• Randomize at the right unit: user, not request, if effects persist across requests (otherwise contamination). Cluster randomization when interference exists (marketplaces, social graphs — one user's treatment affects another's outcome → SUTVA violation).
• Control for confounders by design (randomization) not by post-hoc adjustment, which only handles confounders you measured.
• Guardrail metrics: latency, error rate, revenue, retention — catch a treatment that wins the primary metric while quietly harming the system.

Adversarial validation

• Peeking / optional stopping: did anyone look at results early and stop when significant? That inflates α from 5% to 20-30%. Use sequential testing or fixed horizons.
• Sample ratio mismatch (SRM): is the actual split 50/50? A skew signals a broken assignment/logging pipeline that invalidates everything. Check it first, always.
• Novelty / primacy effects: early lift may be curiosity that decays. Run long enough to capture the steady state.
• Simpson's paradox: an aggregate win can be a loss in every segment (or vice versa) under mix shift. Segment before trusting the average.
• Multiple comparisons: 20 metrics at α=0.05 yields ~1 false "win" by chance. Correct, or pre-declare the primary.

Non-obvious risks

• Statistical significance ≠ practical significance; a real but tiny effect can be "significant" at huge N and worthless.
• Survivorship and selection bias creep in when the treatment changes who shows up in the sample.
• Winner's curse: the effect of the variant you picked because it won is biased upward; expect regression on launch.

Output structure

Deliver: the decision + threshold, the single primary metric and mechanism, the randomization unit and why, the pre-computed sample size/duration, the guardrails, and the validity checks (SRM, peeking policy, segmentation plan) that must pass before the result is trusted.