A/B testing

Design tests that produce winners which hold up after launch. The discipline is simple and usually skipped: one change, a sample size decided in advance, a fixed duration, and no peeking.

When to use it

• Testing a headline, CTA, pricing page layout, signup form field or email subject line.
• Choosing which of several ideas to test first.

Don't test when traffic is too low to reach the sample size within about two months, when the idea has no observation behind it, when the change is cheap to reverse (just ship it and watch), or when you'd be changing several things at once.

Inputs

• The page or flow, and its current conversion rate (baseline).
• Monthly traffic to that page or flow.
• The observation behind the idea: analytics, a /website-score finding, user feedback, sales objections.
• context/goals/ if you have it: so the success metric ties to what the business actually tracks.
• context/brand/: your brand tokens if you have them, so the variant changes only the thing being tested.

Agency users: swap context/ for clients/<client>/context/. If you don't have the baseline or traffic, ask for them. Without both, you can't size the test, so say so rather than guess.

Steps

1. Write the hypothesis in this form: "Because {observation}, we believe {one change} will cause {outcome, with direction and size} measured by {metric} within {timeframe}." Every part is required. Produces: one hypothesis per idea.
2. Score the backlog. Rate each idea 1 to 10 on impact (how much it moves the metric if true), confidence (how sure you are) and ease (how fast to ship and measure). Multiply the three. Test the highest first, and run no more than one or two tests per traffic source at a time. Produces: a ranked list.
3. Size the test with the table below, or any online sample size calculator at 80% power and 95% confidence. Divide by daily traffic per variant to get the duration. Over two months means don't test. Produces: a sample size and a duration.
4. Pre-commit. Write down the sample size, end date, primary metric, two or three secondary metrics, and one or two guardrail metrics that must not get worse (bounce rate, activation, revenue per visitor). Produces: a test plan, dated, before launch.
5. Design the variant. The control stays as it is. Variant B changes one element only, with everything around it identical, including colours, fonts and spacing. Produces: a spec or mockup per variant.
6. Run without peeking. Check only that data is flowing and events fire. Don't read results until the sample size is reached. Produces: a clean run.
7. Call the result. p under 0.05 is a result; above it is inconclusive, not a loser. Then ask if the lift is worth it in real terms, and check the guardrails. A win that breaks a guardrail is a loss. Produces: winner, loser or inconclusive, with the reason.
8. Record the learning, win or lose: hypothesis, planned and actual sample size and dates, result, absolute and relative lift, and what to test next. Keep these in one running log. Produces: a log entry.

Sample size per variant (80% power, 95% confidence, 50/50 split)

What good looks like

• The hypothesis names a real observation and a predicted size of effect.
• The plan was written and dated before the test went live.
• Across many tests, 20 to 30% produce a winner. Much higher means your ideas are too safe; much lower means they're not grounded in evidence.

Mistakes to avoid

• Stopping the test the moment it looks significant. That's how false winners happen.
• Changing headline, image and CTA together, then not knowing which one worked.
• Picking the success metric after seeing the results.
• Ignoring guardrails: a better click rate that lowers signups is a loss.
• Calling an inconclusive test a failure and dropping the idea without a note.

Output

Save to outputs/website/MMYY-ab-test-<name>.md:

# A/B test: <name>
Hypothesis: Because ..., we believe ... will cause ... measured by ... within ...
ICE: I _ x C _ x E _ = _
Baseline: _%   Lift to detect: _%   Sample per variant: _   Duration: _ days (ends <date>)
Primary metric: ...   Secondary: ...   Guardrails: ...
Control: ...   Variant B (one change): ...
Result (after end date): winner / loser / inconclusive, lift, learning, next test

Attribution

Adapted from the ab-testing skill in coreyhaines31/marketingskills (MIT, Corey Haines), including the sample size table: https://github.com/coreyhaines31/marketingskills