agentleFS
MadalaVijayMadalaVijay/marketing-skills-deep/skills/experiment-design/SKILL.md

experiment-design

Skill0 starsChanged 2 months agoMITView on GitHub

Design a marketing test that can actually answer the question, and read the result honestly. Covers sample size, run length, stopping rules and the difference between a real effect and noise. Use before launching a test, or when asked whether a result is real.

32 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use list-quality-scorecard and positive-reply-scoring.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • creatop-gtmSkill

    list-quality-scorecard

    Check a lead list before it goes into a sequence. Gives a deliverability readout (firm) and a targeting readout (advisory). Surfaces issues, never blocks. Use before enrolling any list.

    Also in 3 projects that use this

  • creatop-gtmSkill

    positive-reply-scoring

    Classify replies and compute positive reply rate, the metric that predicts revenue. Use to tell whether a campaign is actually working, not just getting replies.

    Also in 3 projects that use this

The file

We split this skill into the rules it gives an agent

3 rules. 0 are common, 3 are distinctive, and 0 are very specific to this project.

  • 0

    Common

    Most of the 23 edited versions we read keep these.

  • 3

    Distinctive

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Ask for the minimum effect worth acting on, not the effect you hope for.

    line 13
  2. Distinctive:

    Long enough to cover the conversion lag: If people convert 8 days after first touch, a 7-day test measures the wrong thing

    line 35
  3. Distinctive:

    If you must monitor, monitor for harm — a variant losing badly enough to be worth killing — not for wins.

    line 46

“Common” is measured: kept by at least 6 in 10 of the 23 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Experiment design

Most marketing tests cannot detect the effect they are looking for. They run, produce a number, and the number gets believed.

Before the test: can it possibly answer?

Ask for the minimum effect worth acting on, not the effect you hope for. If a 5% lift would not change any decision, do not power the test for 5%.

Rough sample size for a conversion test, per variant:

n ≈ 16 × p × (1 − p) / (minimum detectable effect)²

where p is the current conversion rate as a decimal and the effect is absolute. Going from 2% to 2.4% is an absolute effect of 0.004, and needs roughly 300,000 per variant. Most teams discover this and quietly test for 50% lifts instead.

Report the number honestly. If the traffic does not exist, say the test cannot be run and suggest a different question: a bigger change, a higher-traffic surface, or a qualitative method.

Run length

  • Full weeks only. Weekday and weekend behaviour differ in nearly every consumer category. A test stopped on a Wednesday is a test of Wednesdays
  • Minimum two weeks for anything with a purchase cycle
  • Long enough to cover the conversion lag. If people convert 8 days after first touch, a 7-day test measures the wrong thing

Stopping rules, set in advance

Peeking is the most common way marketing tests lie. Checking daily and stopping when it looks significant produces false positives at several times the stated rate, because you get many chances to cross the line by chance.

Set the sample size and the end date before launch. Then look at the end.

If you must monitor, monitor for harm — a variant losing badly enough to be worth killing — not for wins.

Reading the result

  • Under ~30 conversions per variant, do not call it. Say the test is inconclusive
  • A non-significant result is not "no difference." It means the test could not detect one. Report the range the true effect could plausibly sit in
  • Check the segments only to generate hypotheses, never to rescue a null. Slicing until something is significant will always eventually succeed
  • Confirm the split was actually even and that both variants were served through the same period

What not to test

Small copy changes on low-traffic pages. The effect is too small and the traffic too thin, ever. Change something big enough to matter, or research it qualitatively instead.

Output

For a planned test: required sample per variant, expected run length at current traffic, the pre-registered stopping rule, and a plain statement of whether the test is feasible. For a completed test: the verdict, the plausible effect range, and whether the design permits the conclusion being drawn.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.