Experiment Designer

Purpose

Design a rigorous product or growth experiment: clear hypothesis, variants, success metrics, statistical plan, and guardrails before running an A/B test or controlled rollout.

Input: Decision to make, current baseline metric, available traffic/volume, constraints (time, risk)
Output: Experiment Specification with hypothesis, variant definitions, power analysis, runtime estimate, and ship criteria
Examples: See examples.md for worked input/output.

Workflow

Step 1: Frame the decision

If decision is undefined, stop — experiments without action thresholds waste traffic.

Step 2: Write falsifiable hypothesis

Format:

We believe [change] for [segment] will improve [metric] because [reason].
We will be wrong if [metric] moves opposite or guardrails breach.

Example:

We believe showing AI confidence scores for [SMB users] will improve
[7-day retention] because trust increases repeat use.

Step 3: Define variants

Rules:

• One primary change per experiment (isolate causality)
• Document exact UI/copy/model/config diff
• For AI: version prompts and models in experiment config

Add treatment B only with enough traffic for multi-arm test.

Step 4: Select metrics

Primary metric must map to a tracked event in analytics plan.

If events missing, stop — run product/analytics-instrumentation-planner first.

Step 5: Statistical design

Document formula inputs; if traffic insufficient, recommend:

• Longer runtime
• Higher MDE acceptance
• Sequential test with caution
• Qualitative study instead

Step 6: Runtime and segmentation

For AI quality experiments: stratify by query type if volume allows.

Step 7: Ship and rollback criteria

Pre-register criteria before launch to avoid peeking bias.

Step 8: Validate

Run Validation checklist.

Decision Rules

Validation

• Decision and ship criteria stated upfront
• Falsifiable hypothesis documented
• Control and treatment defined with single primary change
• Primary, secondary, and guardrail metrics named
• Sample size or runtime estimate with MDE
• Randomization unit specified
• Rollback plan for guardrail breach
• Analytics events exist for primary metric

Anti-patterns

• HiPPO after peeking — stopping early when results look good.
• Multiple changes — cannot attribute wins.
• Vanity metrics — clicks without retention/revenue tie.
• No guardrails — wins conversion, breaks latency or quality.
• Forever test — experiment becomes permanent split.

Best Practices

• Run power calculation before engineering work.
• Log experiment ID on all exposure and outcome events.
• Archive results in experiment registry (hypothesis, outcome, learning).
• Pair with growth/go-to-market-planner for launch experiments.
• For AI: include human review sample on quality guardrails.

Output Structure

# Experiment: [Name]

## Decision
[What we decide on win/loss]

## Hypothesis
[Falsifiable statement]

## Variants
| Variant | Change | Traffic % |
|---------|--------|-----------|

## Metrics
| Role | Metric | Baseline | MDE |
|------|--------|----------|-----|

## Statistics
[Sample size, duration, confidence]

## Segmentation
[Population, exclusions]

## Ship criteria
[Win / stop / rollback rules]

## Dependencies
[Analytics events, feature flags]

Next Skills