Split-Test Evaluator

Purpose

Compare ad and landing page variants to determine likely winners, stop premature conclusions, and ensure statistical significance before making decisions.

When to Use This Skill

• When evaluating the results of an A/B test (e.g., two different ad creatives, headlines, or landing pages).
• When deciding whether to scale a winning variant or pause a losing one.
• When determining if a test has reached statistical significance.
• When planning a new split test to ensure proper design and measurement.

Inputs Required

• The hypothesis being tested (e.g., "A shorter form will increase conversion rate").
• The variable being tested (e.g., headline, image, CTA color).
• The success metric (e.g., Cost Per Lead, Conversion Rate, Click-Through Rate).
• The minimum data needed (e.g., 100 conversions per variant).
• The winning condition (e.g., 95% statistical significance).
• The next action after the result (e.g., scale the winner, test a new variable).

Step-by-Step Workflow

1. Review Test Design: Ensure the test is structured correctly (e.g., only one variable changed, sufficient sample size).
2. Analyze Data: Compare the performance of the variants against the success metric.
3. Calculate Significance: Determine if the difference in performance is statistically significant (using a calculator or formula).
4. Evaluate Downstream Impact: Check if the winning variant also performs well further down the funnel (e.g., does a higher CTR lead to more qualified leads?).
5. Make a Decision: Recommend whether to declare a winner, continue testing, or pause the test.

Output Format

Provide a structured evaluation report containing:

• Test Overview: Hypothesis, variable, and success metric.
• Results Summary: Performance data for each variant.
• Significance Analysis: Statistical confidence level and whether the test is conclusive.
• Downstream Impact: Any observed effects on subsequent funnel stages.
• Recommendation: Clear action (e.g., "Declare Variant A the winner and scale budget").

Decision Rules

• Do not suggest random changes. Keep testing structured.
• Never declare a winner prematurely based on small sample sizes or insignificant differences.
• Always consider the downstream impact; a winning ad (high CTR) might produce lower-quality leads than a losing ad (lower CTR).

Escalation Rules

• If a test shows a statistically meaningful loss (e.g., a new landing page significantly underperforms the control), escalate immediately to pause the test.
• If the test design is flawed (e.g., multiple variables changed simultaneously), flag it and recommend a redesign.

Templates & Resources

• templates/split-test-report.md: Template for presenting split test results.
• scripts/calculate-significance.py: (Optional) Script to calculate statistical significance.