Growth Experiments — Experiment Lead

When to use

Use before an intervention goes live and when interpreting its result. Own the experiment record and decision quality; Marketing creates assets, Sales handles outreach and qualification, and Developers implement production exposure.

Inputs

• Strategy hypothesis, ICP/cohort definition, intended business effect and cost ceiling.
• Baseline denominator, outcome rate/variance, revenue lag and minimum effect worth pursuing.
• Eligibility/exclusion rules, identity map, simultaneous tests and instrumented events.
• Control behavior, proposed treatment, business owner and authorized deployment scope.

Workflow

1. State a falsifiable mechanism. “For cohort X, intervention Y should change business outcome Z because mechanism M.” Separate exploratory observations from confirmatory hypotheses.
2. Choose the business metric. Define numerator, denominator, unit, attribution and maturity window. For B2B prospecting, use unique Sales-accepted opportunities per eligible account and cost per accepted opportunity; reply/open rates are diagnostics. For lifecycle tests, use retained activation, contribution or retained revenue as appropriate.
3. Assign the unit and control. Randomize by account for B2B shared exposure, keeping users and agreed parent-account clusters together. Predeclare stratification and assignment persistence. Specify the existing experience as control or a genuine holdout. Record concurrent tests and likely spillover. For nonrandomized pilots, state the confounding and limit the causal claim.
4. Plan information needs. Estimate sample needs from baseline, effect threshold, chosen inference method and uncertainty. State alpha/power or a justified Bayesian decision rule, clustering and multiple-comparison treatment. Do not invent sample sufficiency. If volume is too low, run a bounded feasibility pilot and reserve “inconclusive” for business impact.
5. Predeclare timing and decisions. Set enrollment dates, minimum/maximum duration, outcome follow-up, analysis time and scale/iterate/stop/inconclusive rules before exposure. Efficacy peeking cannot end a fixed-horizon test; any sequential method must be specified first. Safety and cost stops apply independently and do not establish efficacy.
6. Instrument and preflight. Confirm eligibility and exposure counts, expected allocation, event freshness and metric reconciliation with growth-engineering. Check sample-ratio mismatch and account leakage. Freeze the card, cohort policy and asset version; changes require a new version and a declared interpretation.
7. Run within scope. Route assets to Marketing and authorized outreach to Sales. Log deviations and guardrail events. Do not silently modify treatment, assignment or qualification definitions mid-test.
8. Analyze and decide. Analyze by assigned unit, including assigned accounts with no exposure unless a different estimand was predeclared. Report absolute rates, denominators, absolute/relative effects, uncertainty, missing outcomes and mature cohorts. Separate diagnostics from the primary result. Return scale, iterate, stop or inconclusive; send revenue feedback to strategy and qualification owners.

Output format

Experiment card — required artifact

ID / version / status / owner / decision date:
Hypothesis and mechanism:
Business metric: numerator / denominator / unit / source / maturity window:
Baseline and minimum worthwhile effect:
Eligibility / exclusions / ICP and score versions:
Randomization unit (account for B2B) / assignment / stratification:
Treatment asset/version / control or holdout / contamination risks:
Sample plan / method / uncertainty / multiple-comparison policy:
Enrollment duration / follow-up duration / fixed analysis date:
Predeclared decision: scale | iterate | stop | inconclusive criteria:
Guardrails: metric / threshold / owner / safety action:
Cost ceiling / attribution / cost per accepted opportunity definition:
Instrumentation QA / authorization already held / required handoffs:
Results: assigned denominators / outcomes / effect / uncertainty / lag:
Deviations / interpretation limits / decision / next owner:

A draft card still needs concrete definitions and dates to be executable. A finished card includes results and a decision, including inconclusive if the data cannot answer the question. “No significant difference” does not prove equivalence.

Model adaptations

• ABM / B2B services: account assignment, Sales acceptance and contribution guardrails; follow long sales cycles and delivery capacity.
• Product-led SaaS: account value and retained paid adoption; avoid nudges that increase meetings while harming activation. Keep same-company users together.
• Ecommerce: declare customer/visitor assignment, purchase and refund windows; measure contribution after discounts, fulfillment and returns. Protect inventory and customer experience.

Quality bar

• Hypothesis, business metric and mechanism can be falsified.
• Assignment unit, holdout/control and contamination handling are explicit.
• Sample, duration, outcome lag and decision rule were set before exposure.
• Guardrail stops do not masquerade as positive efficacy results.
• Denominators and uncertainty are shown; missing/immature data are visible.
• Inconclusive is an allowed decision; observational results are not called causal.
• Cost and downstream accepted/won/retained outcomes close the learning loop.

Example

Testing an account-specific operations diagnostic against existing outreach requires randomizing eligible accounts, holding Sales acceptance criteria constant, and predeclaring the accepted-opportunity window. Eight versus four accepted opportunities from forty accounts per arm is useful pilot evidence but does not by itself establish lift. growth playbook provides a fully specified, explicitly synthetic feasibility example with an inconclusive business-outcome decision.