landing-page-ab-test-readout
Skill2 starsChanged 4 months agoMITView on GitHub
Decides whether an A/B result is a decision or a coincidence, and says plainly when the honest answer is that the test cannot conclude. Use when a test has run and someone wants to declare a winner.
In this repository26 other files it gives its agents.
- AGENTS.mdAGENTS.md
- above-the-fold-clarity-reviewSkill
- accessibility-conversion-blocker-checkSkill
- comparison-page-positioning-reviewSkill
- conversion-leak-finderSkill
- cta-clarity-checkSkill
- form-friction-finderSkill
- google-ads-landing-page-experience-reviewSkill
- hero-section-diagnosisSkill
- landing-page-copy-readability-passSkill
- landing-page-scale-readiness-checkSkill
- landing-page-triageSkill
- lead-form-sales-handoff-checkSkill
- mobile-conversion-reviewSkill
- objection-map-builderSkill
- offer-clarity-diagnosisSkill
- page-length-fit-checkSkill
- page-speed-impact-reviewSkill
- paid-traffic-message-match-auditSkill
- popup-and-overlay-timing-reviewSkill
- pricing-page-clarity-reviewSkill
- social-proof-strength-auditSkill
- thank-you-page-opportunity-auditSkill
- traffic-temperature-match-reviewSkill
- trial-vs-demo-path-decisionSkill
- trust-signal-auditSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
13 rules. About 0 are common, 12 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈12
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 23
If conversion counts per arm are missing, this skill does not run.
- Distinctive: line 25
Not for: deciding what to test next, or reading a test that has not finished.
- Distinctive: line 32
Check sample size, baseline CVR, absolute lift, relative lift, confidence and tracking quality.
- Distinctive: line 39
Use this table when the user provides A/B test data:
- Distinctive: line 49
Follow-up test: the one next test if the result is useful but incomplete.
- Very specific: line 53
Run significance.py first and quote its numbers.
- Distinctive: line 59
Never smooth an inconclusive result into a weak recommendation.
- Distinctive: line 63
Start with the helper's output, verbatim.
- Distinctive: line 101
Do not name a winner without running the helper.
- Distinctive: line 102
Do not name a winner below the refusal thresholds, whatever the observed difference.
- Distinctive: line 103
Do not report a segment result without that segment's counts.
- Distinctive: line 104
Do not soften inconclusive into a weak yes.
- Distinctive: line 105
Do not start, stop or configure a test.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Landing Page Ab Test Readout
Use this skill when
- an A/B test has finished or is being called early, and the per-variant sample sizes and conversion counts are available.
- somebody wants to declare a winner and ship it this week.
- a test was stopped early because the numbers looked good.
This skill reads a finished test and needs conversion counts per arm. It does not design tests or run them.
Required input
- per variant: sessions or visitors, conversions, and the conversion definition. Rates alone are not enough; the readout needs the counts.
- runtime in days, and the days of the week it covered.
- what changed between variants, and whether anything else changed on the site during the window.
- optional but decisive: segment splits by device and source, with their own counts.
- tracking confidence: was the same event measured identically in both arms.
- If conversion counts per arm are missing, this skill does not run. Ask for them. A readout built on rates cannot compute anything.
Not for: deciding what to test next, or reading a test that has not finished. This skill reads a result that exists.
There is a deterministic helper next to this skill: significance.py. Run it. Do not assess significance by eye.
Analysis workflow
- Restate the hypothesis, variant difference, audience, dates, traffic source and success metric.
- Check sample size, baseline CVR, absolute lift, relative lift, confidence and tracking quality.
- Compare results by segment: device, source, campaign, new vs returning and lead quality where available.
- Separate statistical signal from business significance and novelty effects.
- Return ship, continue, stop, rerun or inconclusive with the next test question.
Diagnostic rubric
Use this table when the user provides A/B test data:
| Segment | Variant A CVR | Variant B CVR | Absolute lift | Relative lift | Quality signal | Decision |
| All traffic / device / source / audience | Percent or missing-data marker | Percent or missing-data marker | Percentage points | Percent | SQL rate, form quality, revenue signal or missing-data marker | Ship / keep testing / segment / rerun / ignore |
Then call out:
- Trust level: sample size, tracking quality, runtime and segment consistency.
- Business readout: what changes for qualified leads, pipeline or revenue, if known.
- Follow-up test: the one next test if the result is useful but incomplete.
Decision rules
- Run
significance.pyfirst and quote its numbers. A verdict about noise without a computed p-value or interval is a vibe, and this is the one skill where that is unforgivable. - Refusal thresholds, applied before any winner is named. Under 100 conversions in either arm, or a runtime shorter than seven days, the verdict is
inconclusiveregardless of what the observed difference looks like. State which of the two conditions failed. Both numbers are this pack's own working defaults with no external source, including calling seven days one business cycle; say so whenever you quote them. - A test that stopped the moment it looked significant did not measure what it appears to have measured. Ask when the decision to stop was made, and say the result is compromised if it was made on the strength of the numbers.
- Segment findings need their own counts. A device split quoted without sessions per device is not evidence and should be dropped from the readout rather than reported at medium confidence.
- Report the interval, not only the point estimate. A lift of 1.1 points with an interval spanning zero is a decision not to act, expressed honestly.
- Shipping a variant that did not reach significance is a legitimate business decision. Say so, and say plainly that it must not then be recorded as a validated lift.
- Never smooth an inconclusive result into a weak recommendation. The description of this skill promises it will say a test cannot conclude, and that promise is the point.
Output format
Start with the helper's output, verbatim.
| Variant | Sessions | Conversions | Rate | Difference vs control | Interval | Significant |
| A / B | Count | Count | Computed | Percentage points | From the helper | Yes / no, at the stated level |
Then:
| Gate | Status |
| At least 100 conversions per arm | Pass / fail, with the counts |
| At least one full business cycle | Pass / fail, with the runtime |
| Same conversion definition in both arms | Pass / fail / unknown |
| Stop decision made before looking | Yes / no / unknown |
End with:
Verdict:winner / inconclusive / compromisedIf you ship it anyway:the sentence that must go in the record instead of "validated lift"Additional sample needed to decide:the number from the helper, or not applicableMissing data:what would change the verdict
Practical example
User: "Variant B wins. A: 1200 sessions, 36 conversions, 3.0 percent. B: 1180 sessions, 48 conversions, 4.07 percent. Ran 6 days. We want to ship B."
Assistant should run the helper first:
python3 significance.py --a 1200 36 --b 1180 48 --runtime-days 6and quote what it printed: the observed gap is +1.07pp, z = 1.412, two-sided p = 0.1581 at alpha 0.05, and the 95 percent interval on the difference runs from -0.42pp to +2.55pp, so it contains zero and the direction of the effect is not established. Detecting a difference this size at 80 percent power would need 4693 visitors per arm against the 1180 on hand, a shortfall of 3513. Both refusal gates also fail independently: 36 and 48 conversions are under the 100 per arm minimum, and six days is under one full business cycle.
Then the verdict: inconclusive, not "B wins". Shipping B anyway is a legitimate business decision, and the sentence that goes in the record is that it was shipped without evidence of a lift, never as a validated lift. No segment consistency and no tracking confidence were supplied, so whether the instrumentation matched between arms is unknown and would change this readout if it did not.
Guardrails
- Do not name a winner without running the helper.
- Do not name a winner below the refusal thresholds, whatever the observed difference.
- Do not report a segment result without that segment's counts.
- Do not soften inconclusive into a weak yes.
- Do not start, stop or configure a test.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

