agentleFS
kastriasanikastriasani/google-ads-skills/skills/google-ads-ab-testing/SKILL.md

google-ads-ab-testing

Skill1 starsChanged 5 months agoMITView on GitHub

Design valid Google Ads experiments (RSA ad copy, landing page, bid strategy) and read them for statistical significance. Use this whenever someone wants to set up or plan a Google Ads experiment or A/B test, run a campaign experiment or draft, test an RSA or ad variation, test a landing page, compare two bid strategies or tROAS/tCPA targets, work out the sample size or how long a test must run, check whether a result is statistically significant, compute observed lift and a p-value, or decide whether a test can be called or was stopped too early. Triggers include A/B test, experiment, split test, campaign experiment, draft and experiment, ad variation test, RSA test, landing page test, bid strategy test, sample size, test duration, statistical significance, p-value, confidence, MDE, minimum detectable effect, and lift. Runs standalone on pasted numbers; optional Google Ads API and GA4 sharpen it.

The file

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 2 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Check whether a finished test is statistically significant and what the lift is.

    line 25
  2. Very specific:

    If the request is to choose a bid strategy in the first place, use google-ads-bidding.

    line 28
  3. Very specific:

    Size it before launch: Run scripts/experiment_planner.py --baseline 0.05 --mde 0.15 --daily-clicks 600 (or --demo).

    line 38
  4. Distinctive:

    Sanity-check feasibility: If the required sample needs more than ~8 weeks at the campaign's click volume, raise the MDE, reduce variants, …

    line 46
  5. Very specific:

    Set it up correctly: Use Google Ads campaign experiments (from a draft), not two live campaigns.

    line 51
  6. Very specific:

    Read the result once, at the planned sample: Run scripts/experiment_planner.py --read --control-clicks 12000 --control-conv 600 …

    line 56

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Google Ads A/B Testing

Design experiments that can actually answer a question, then read them honestly. The two ways tests go wrong are bad design (changing more than one thing, unfair split) and bad statistics (too small, called early). This skill closes both: it sizes the experiment before launch, gives the setup checklist, and computes significance from the results.

When to use

Reach for this skill when the user wants to:

  • Plan or set up a Google Ads experiment - RSA/ad-variation, landing page, or bid strategy - using drafts and experiments.
  • Work out the required sample size and how long the test must run.
  • Check whether a finished test is statistically significant and what the lift is.
  • Decide whether a test can be called or was stopped too early (peeking).

If the request is to choose a bid strategy in the first place, use google-ads-bidding. If it is to forecast results, use google-ads-forecast.

Workflow

  1. Frame the test. Pin down the single variable being changed (one only), the primary metric (usually conversion rate), the guardrails that must not get worse (CPA, conversion value), and the baseline conversion rate of the control. See references/experiment-design.md for what is worth testing and in what order.
  2. Size it before launch. Run scripts/experiment_planner.py --baseline 0.05 --mde 0.15 --daily-clicks 600 (or --demo). It returns clicks needed per arm, the estimated duration, and a drafts-and-experiments setup checklist (50/50 split, hold constant, one-variable rule, minimum one week, pre-committed stop rule). Use scripts/sample_size_calculator.py for a quick sample/duration with a design-quality score across multiple scenarios.
  3. Sanity-check feasibility. If the required sample needs more than ~8 weeks at the campaign's click volume, raise the MDE, reduce variants, or test higher in the funnel. A test that cannot reach significance should not be launched - say so. references/significance.md has the sample-by-baseline table and the alternatives.
  4. Set it up correctly. Use Google Ads campaign experiments (from a draft), not two live campaigns. Change one variable, split 50/50, hold budget/targeting/ schedule constant, and exclude the Smart Bidding learning window for bid-strategy tests.
  5. Read the result once, at the planned sample. Run scripts/experiment_planner.py --read --control-clicks 12000 --control-conv 600 --variant-clicks 12000 --variant-conv 690 --needed-per-arm 14190. It reports observed CVRs, absolute and relative lift, the z-test p-value, whether it is significant, a WINNER/LOSER/INCONCLUSIVE verdict, and a warning if the sample is below what the test needed (called early).
  6. Decide and document. Apply the verdict, check practical significance and the guardrails, and record the learning. Do not peek-and-stop; honour the committed sample and minimum run length.

Inputs and data sources

Runs standalone: paste the baseline rate and the arm-level clicks/conversions, or hand the scripts the numbers directly. Each script has a --demo mode and a --json flag for piping into a report or sheet.

Optional sources that improve accuracy:

  • Google Ads API / UI - baseline conversion rate, daily clicks, and the experiment arms' clicks and conversions; experiment status and learning state.
  • GA4 - downstream conversion and value validation, segment-level conversion rates for segment-powered tests.
  • Google Sheets / BigQuery - log the hypothesis bank and results, and pipe --json output into a test repository.

Outputs

  • Clicks needed per arm and estimated test duration.
  • A drafts-and-experiments setup checklist (split, hold-constant, one-variable rule, stop rule).
  • A design-quality score flagging under-powered or impractically long tests.
  • Observed absolute and relative lift, p-value, and a significance verdict.
  • An early-call warning when the observed sample is below the planned sample.

References

  • references/experiment-design.md - Google Ads experiments setup, what to test, the one-variable rule, hypothesis format, split and duration.
  • references/significance.md - sample size, duration, p-value and confidence, and the common pitfalls (peeking, under-powering, wrong baseline, learning period).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.