statistics

Core Principles

1. Question first, method second — Define what you're trying to learn before selecting a test. The question determines the method, not the other way around.
2. Check assumptions before computing — Every test has assumptions (normality, independence, equal variance). Violating them silently produces misleading results.
3. Effect size matters more than p-values — A statistically significant result can be practically meaningless. Always report the magnitude of the effect alongside significance.
4. Confidence intervals over point estimates — A mean of 42 is less useful than "42 ± 5 (95% CI)". Always quantify uncertainty.
5. Don't test what you can describe — If the question is "what does the data look like?", use descriptive statistics. Hypothesis tests answer "is this difference real?", not "what is the data?"
6. One question, one test — Running multiple tests on the same data inflates false positives. If you must run multiple comparisons, correct for it (Bonferroni, Holm, or FDR).
7. Honest reporting — Report all results, not just significant ones. Pre-register your hypothesis when possible. State limitations clearly.

Method Selection Guide

Step 1: Classify Your Question

Step 2: Identify Your Data Structure

Descriptive Statistics

Use these to summarize and understand data before testing anything.

Central Tendency

Rule: If mean ≠ median by more than 10–15%, the data is skewed — prefer median.

Spread

Distribution Shape

Comparison Tests

Choosing the Right Test

How many groups?
├── 1 group vs. known value
│   ├── Normal → One-sample t-test
│   └── Not normal → Wilcoxon signed-rank
├── 2 groups
│   ├── Paired (same subjects)?
│   │   ├── Normal → Paired t-test
│   │   └── Not normal → Wilcoxon signed-rank
│   └── Independent?
│       ├── Normal, equal variance → Independent t-test
│       ├── Normal, unequal variance → Welch's t-test
│       └── Not normal → Mann-Whitney U
└── 3+ groups
    ├── Normal, equal variance → One-way ANOVA
    ├── Not normal → Kruskal-Wallis
    └── Repeated measures → Repeated measures ANOVA or Friedman test

Test Reference

Post-Hoc Tests (After ANOVA / Kruskal-Wallis)

If the omnibus test is significant (groups differ), use post-hoc tests to find which groups differ:

Correlation and Association

Correlation ≠ causation. Always state this explicitly when reporting. A correlation is a starting point for investigation, not a conclusion.

Effect Size

Always report effect size alongside significance. A tiny p-value with a tiny effect is not actionable.

Sample Size and Power

Before running an experiment, calculate the required sample size:

## Power Analysis

**Test**: <e.g., "Independent t-test">
**Desired power**: 0.80 (standard) or 0.90 (high confidence)
**Significance level (α)**: 0.05
**Minimum detectable effect (MDE)**: <e.g., "Cohen's d = 0.3" or "5% conversion lift">
**Required sample size per group**: <computed value>
**Expected duration to collect**: <e.g., "2 weeks at current traffic">

Rules of thumb:

• Power < 0.80 = underpowered; you'll likely miss real effects
• Larger effects need smaller samples; smaller effects need larger samples
• When in doubt, aim for more data — underpowered studies waste everyone's time

A/B Testing

A/B tests are the most common applied statistics scenario. Follow this workflow:

Before the Test

1. State the hypothesis: "Variant B will increase conversion rate by ≥ 5% relative to control"
2. Choose the metric: Primary metric (what you're optimizing) + guardrail metrics (what must not degrade)
3. Run power analysis: Calculate required sample size per variant
4. Set the duration: Don't peek at results before the required sample size is reached
5. Pre-register: Document hypothesis, metric, sample size, and success criteria before launching

During the Test

• Don't peek: Checking results daily and stopping early inflates false positives
• Don't change the test mid-flight: Adding variants, changing allocation, or modifying the feature during the test invalidates results
• Monitor guardrails: If error rates or latency spike, you have a quality issue, not a test result

After the Test

## A/B Test Results

**Test**: <name>
**Duration**: <start – end>
**Sample size**: Control: <n>, Variant: <n>

**Primary metric**: <metric name>
| Group | Value | 95% CI |
|-------|-------|--------|
| Control | <X> | [<lo>, <hi>] |
| Variant | <X> | [<lo>, <hi>] |

**Relative lift**: <X%> [<CI lo>, <CI hi>]
**p-value**: <X>
**Effect size**: <measure and value>

**Guardrail metrics**: <all stable / degradation in X>

**Decision**: <Ship variant | Keep control | Inconclusive — extend or redesign>
**Reasoning**: <why this decision, referencing effect size and practical significance>

Assumption Validation Checklist

Before applying any parametric test, verify:

Process

Step 1: State the Question

## Statistical Question

**Question**: <What are you trying to learn? Be specific.>
**Population**: <Who or what does this apply to?>
**Hypothesis** (if testing): 
  - H₀: <null hypothesis — no effect, no difference>
  - H₁: <alternative hypothesis — the effect you expect>
**Practical significance threshold**: <What size effect would actually matter?>

Step 2: Profile the Data

## Data Profile

**Source**: <where the data comes from>
**Sample size**: <n>
**Variables**: 
  - <var1>: <type> — <description>
  - <var2>: <type> — <description>
**Missing data**: <count, percentage, pattern>
**Distribution**: <per variable — normal, skewed, categorical frequencies>
**Assumption checks**: <normality, variance, independence — pass/fail>

Step 3: Select the Method

Use the selection guide above. Document:

## Method Selection

**Test**: <name>
**Why**: <reasoning tied to question type, data structure, and assumption checks>
**Alternative considered**: <what else could work and why it wasn't chosen>
**Correction applied**: <e.g., "Bonferroni for 3 pairwise comparisons" or "none — single test">

Step 4: Compute and Interpret

## Results

**Test statistic**: <name> = <value>
**p-value**: <value>
**Effect size**: <measure> = <value> (<small | medium | large>)
**Confidence interval**: [<lower>, <upper>] at <confidence level>

**Interpretation**: <Plain-language statement of what this means. Not just "p < 0.05 so we reject H₀" — state the practical implication.>

**Limitations**: <What this result does NOT tell us. Confounders, generalizability, assumptions that were borderline.>

Output Format

Every statistical analysis should produce:

1. Question — What you're trying to learn, with hypothesis if applicable
2. Data Profile — Summary of the data, assumption checks
3. Method Selection — Which test, why, and what alternatives were considered
4. Results — Test statistic, p-value, effect size, confidence interval
5. Interpretation — Plain-language meaning and practical significance
6. Limitations — What the result doesn't tell you

Regression Basics

Regression is the most common applied statistics method. Use this decision guide:

Before running regression, check:

• Linearity (scatter plot of residuals vs. fitted)
• Independence of residuals
• Homoscedasticity (constant variance of residuals)
• No multicollinearity (VIF < 5 for each predictor)
• Normality of residuals (for inference, not prediction)

Key outputs to report: R², adjusted R², coefficients with CIs, residual plots, and F-test p-value.

When to Consider Bayesian Methods

Frequentist methods (everything above) are the default. Consider Bayesian approaches when:

Practical tools: pymc, arviz for general Bayesian analysis; Bayesian A/B testing via bayesian-testing or built-in platform features (Optimizely, LaunchDarkly).

Default stance: Use frequentist methods unless one of the above signals is present. Don't switch to Bayesian for complexity's sake.

Tooling Reference

Common Pitfalls