A/B Testing Engineer

When to Use

• Frame hypotheses, success criteria, and decision rules before launch
• Define primary, secondary, and guardrail metrics with clear ownership
• Design A/B, A/B/n, or high-level MVT experiments with correct randomization units
• Calculate sample size, power, MDE, and minimum runtime (seasonality, cycles)
• Plan allocation, bucketing, exposure logging, and SRM monitoring
• Align instrumentation with event taxonomy and analysis-ready tables
• Write pre-registered analysis plans (frequentist and Bayesian workflow level)
• Run readouts, interpret uncertainty, and recommend ship / iterate / kill / holdout
• Maintain experiment registry hygiene and program operations

When NOT to Use

• Build or operate warehouse pipelines, dbt models, or BI stack → analytics-data-engineer, data-warehouse-engineer
• Executive dashboards and KPI storytelling without experiment design → bi-analyst
• General ML modeling, causal inference beyond experiments, MLOps → data-scientist
• Model training/eval A/B for ML systems (offline/online model comparison) → ml-research-engineer-safeguards, data-scientist
• Marketing copy, campaigns, or channel strategy → communication-lead, cmo-advisor (if installed)
• Product roadmap, portfolio prioritization without measurement design → product-management-monetization, cpo-advisor (if installed)
• Weekly metrics review cadence without new experiment → metrics-review (if installed), bi-analyst
• Legal privacy, consent, or regulatory sign-off for tracking → compliance-engineer, privacy-research-engineer-safeguards
• Growth experiment ideation without statistical design → growth-marketer (if installed), marketing-analyst (if installed)

Related skills

Core Workflows

1. Hypothesis and metric contract

1. State problem, user segment, and expected mechanism
2. Lock one primary metric for the decision (pre-register)
3. List secondary (learning) and guardrail (safety) metrics
4. Define non-goals and segments that are out of scope for decision
5. Agree rollout criteria: win threshold, guardrail breach, neutral band

See references/hypothesis_metrics_and_design.md.

2. Design and power

1. Choose design type: A/B, A/B/n, factorial/MVT (high level), holdout
2. Set MDE from business impact, not statistical convenience
3. Compute per-variant sample size (α, power, one- vs two-sided)
4. Estimate duration: traffic, allocation %, seasonality, full cycles
5. Document exclusions, mutual exclusion with other tests, and cooldown rules

See references/sample_size_power_and_duration.md.

3. Randomization, instrumentation, and SRM

1. Pick randomization unit (user, account, device, session—justify match to analysis)
2. Define bucketing hash, sticky assignment, and cross-device policy
3. Map exposure events to assignment logs; verify join keys
4. Pre-launch QA: assignment distribution, latency, fallback behavior
5. Monitor SRM and invariant checks during the run

See references/randomization_instrumentation_and_srm.md.

4. Analysis, readout, and decision

1. Follow pre-registered analysis plan; intent-to-treat by default
2. Run primary analysis with CIs; pre-defined segment slices only
3. Apply multiplicity policy if multiple variants or metrics drive decisions
4. Compare Bayesian vs frequentist readout needs with stakeholders
5. Produce readout: effect, uncertainty, guardrails, recommendation, risks

See references/analysis_readout_and_decision.md.

5. Governance and program ops

1. Register experiment in catalog (owner, dates, links, status)
2. Enforce ethical constraints: no p-hacking playbooks, no optional peeking without correction
3. Archive artifacts: config, power calc, analysis notebook, decision log
4. Plan ramp, rollback, and long-term holdout if shipping

See references/governance_ethics_and_program_ops.md.

When to load references