A/B Testing

Rules

• Hash-based assignment: deterministic variant selection using hash(userId + experimentId) % 100 — no randomness drift between sessions
• Variant config: store experiments in a config object { id, name, variants: [{ id, weight }], status } — never hardcode variants
• Split traffic server-side: assign in middleware or API route, set cookie for consistency — client-side splits cause flicker
• Statistical significance: require 95% confidence (p < 0.05) before declaring a winner — use z-test for proportions
• Sample size: calculate minimum sample before starting — n = (Z^2 * p * (1-p)) / E^2 where E is minimum detectable effect
• One metric per experiment: define a primary metric upfront (conversion rate, click-through, revenue) — secondary metrics are exploratory only
• Mutual exclusion: users in one experiment should not overlap with conflicting experiments — use experiment layers
• Feature flags: wrap experiment code in feature flag checks — clean up losing variants after experiment ends
• Duration: run for at least 1-2 full business cycles (7-14 days minimum) to account for day-of-week effects
• Logging: track experiment_id, variant_id, user_id, timestamp for every impression and conversion event

Avoid

• Peeking at results before sample size is reached — inflates false positive rate
• Changing experiment parameters mid-run — invalidates statistical analysis
• Testing too many variants at once — increases required sample size exponentially
• Client-side variant assignment without cookies — causes flickering and inconsistent experiences