Experiment

PREREQUISITE: Read ../_mureo-shared/SKILL.md for auth, security rules, output format, and Tool Selection (Read/Write on Code, mureo_strategy_* / mureo_state_* MCP on Desktop / Cowork).

Tool names below refer to the built-in platforms; a namespaced variant (see _mureo-shared → Namespaced built-in variants) is used with its prefix substituted.

Turn an ad-hoc change into a designed experiment. Most "we tried X and CPA moved" claims are noise — one variable was never isolated, the window was never fixed, and the decision was made by peeking. This skill enforces the discipline: a falsifiable hypothesis, exactly one variable, a success metric tied to a Goal, a pre-committed duration and sample floor, no peeking, and a per-variant verdict where inconclusive is a valid outcome.

Read ../_mureo-learning/SKILL.md first — its Observation Windows and Minimum Sample Sizes tables set the duration and sample floor this skill commits to, and its evidence lifecycle is the rulebook for the no-peeking rule below.

Prerequisites

• STRATEGY.md and STATE.json must exist (run the onboard skill first)

Steps

Before you start: Run the Diagnostic preamble from ../_mureo-shared/SKILL.md — load learning insights (mureo_learning_insights_get) and consult advisors (mureo_consult_advisor) before drawing conclusions.

0. Establish today: call mureo_state_get first, on every host (including Claude Code, where you would otherwise Read the file) and take server_now from its response — ISO 8601 with UTC offset, e.g. 2026-07-28T10:12:33+09:00. Its date is the only source of the current date for this run: the designed window's start_time / end_time, the observation_due you commit to in step 7, and the "has the window closed?" gate in step 9 are all measured from it. A stale date is not cosmetic here — it is how a test gets evaluated early, which is precisely the peeking failure step 8 forbids. Do not shell out (this skill must run in Bash-less headless hosts) and do not read the date off STATE.json — last_synced_at, reports.*.period and action_log timestamps are history, never evidence of what day it is now. Never write server_now into STATE.json: it is a response field, and a persisted copy becomes tomorrow's stale "today".
1. Hypothesis intake — force a falsifiable statement. Take the hypothesis from the operator, from a /creative-refresh finding, or from a VALIDATED learning insight, and rewrite it into a single testable sentence with three explicit parts: variable (the ONE thing that differs), expected effect (direction + rough magnitude), and success metric (the number that decides it). Reject vague inputs ("make it better") — push back until it is falsifiable, e.g. "Video creative (variable) will lower CPA by ≥15% (effect) vs the current static image, measured on cost_per_conversion (metric)." If two things would differ, split into two experiments or drop one.
2. Design — one variable, tied to a Goal. Confirm exactly one variable changes between variants (creative, headline, audience, placement, bid — not several at once; confounded designs are unattributable). Set the variant count (2 is the default; more variants need proportionally more sample and duration). Bind the success metric to a specific STRATEGY.md Goal so the winner means something (a CTR win that doesn't move the Goal's CPA/CV is not a win). Derive the minimum duration and sample floor from ../_mureo-learning/SKILL.md: creative/targeting changes need a 14-day window; the metric's sample floor (CPA: ≥30 conversions/variant; CTR: ≥1,000 impressions/variant; CVR: ≥200 clicks/variant) must be reachable within the window at current traffic — if it is not, say so and either extend the window or advise against running (an underpowered test wastes spend).
3. Discover platforms. Identify configured platforms from STATE.json platforms. Include any hosted official-MCP connector present in the session (e.g. TikTok, key tiktok_ads) and mcp__mureo__<plugin>_* plugin platforms — but note honestly that a true randomized split-test tool only exists for Meta below; on other platforms the experiment is a manual/approximate construction (see ../_mureo-shared/SKILL.md → Hosted-connector platforms and Plugin platforms). mureo BYOD data lives in the workspace byod/ directory and is reachable only through mureo MCP tools — do not look for raw CSVs in the project directory.
4. Setup — Meta (native split test). Meta has a real experiment engine. The variants are ad sets — the split-test tool references pre-existing ad sets and does not create them, so first make sure each variant exists as its own ad set (identical except for the one variable) via meta_ads_ad_sets_list / meta_ads_ad_sets_create. In BYOD mode the list response is wrapped as {"source": "byod_import", "as_of": <import time>, "data": [...]} — take the rows from data and treat them as of as_of, not as live. Then, after the approval gate (step 6), create the test with meta_ads_split_tests_create:
   • cells: one {name, adsets:[ad_set_id]} per variant; Meta splits traffic evenly.
   • objectives: [{type: COST_PER_RESULT | CONVERSIONS | REACH | CPC | CPM}] — pick the type matching the Goal metric from step 2.
   • start_time (must be in the future — i.e. after server_now) and end_time — honor the designed window; Meta requires ≥4 days between them.
   • confidence_level (80–99, default 95). Do not lower it just to force a faster verdict. Inspect a running test with meta_ads_split_tests_list / meta_ads_split_tests_get. If mureo's Meta tools are unavailable (MUREO_DISABLE_META_ADS=1 after mureo providers add meta-ads-official), fall back to the official meta-ads-official hosted MCP for whatever experiment surface it exposes, and self-apply the guardrail/confirm rules yourself (mureo's PolicyGate cannot see that call).
5. Setup — Google Ads (no split-test tool — honest manual paths). mureo ships no Google Ads campaign-experiment / drafts-and-experiments tool — there is no google_ads_experiment* in the tool surface, so do not invent one. Offer the operator the two honest alternatives and require explicit acknowledgment of their caveats before proceeding:
   • (a) RSA asset A/B (in-ad, lowest-risk). Add the challenger headline/description alongside the control in the same Responsive Search Ad and let Google's asset rotation gather per-asset data; read the split with google_ads_rsa_assets_analyze (per-asset performance_label + CTR) and google_ads_rsa_assets_audit. Caveat: this is asset-level rotation, not a randomized user split — Google serves assets non-uniformly (it favors likely winners), so it is directional, not a clean controlled test.
   • (b) Duplicated-campaign split (campaign-level). Duplicate the campaign (google_ads_campaigns_create + google_ads_ads_create), change only the one variable, split budget, and compare with google_ads_ad_performance_report / google_ads_ad_performance_compare. Caveats: the two campaigns compete in the same auction (they can cannibalize each other), Smart Bidding re-enters its learning phase on each, and there is no traffic-randomization guarantee — confounds are larger than a Meta split test. State these plainly and get the operator to accept them. If neither caveat is acceptable, recommend not running the Google Ads experiment rather than shipping a test the operator will over-trust.
6. Approval gate before ANY write. Setup steps 4–5 create/mutate platform state. Apply the Confirm Before Write Operations rule from ../_mureo-shared/SKILL.md: show the full design (variants, variable, split, budget, window, success metric, sample floor) and get explicit approval before creating any ad set, split test, or duplicated campaign. Respect Operation Mode and ## Guardrails (budget caps, blocked_operations); refuse a design that breaches them.
7. Record the design in action_log. After the test is created, append one mureo_state_action_log_append entry per variant (or one entry capturing all variants) with metrics_at_action = the baseline per variant (the control's current cpa / conversions / ctr / impressions with their period, and the challenger's starting zero/baseline), command="/experiment", a summary naming the hypothesis, and observation_due = the designed end date (not a default 7/14). This is the commitment device that makes step 8's no-peeking rule enforceable and lets daily-check pick the evaluation up automatically. Give every entry a display line: display_title (at most 40 characters) and display_summary (at most 120) — what this action WAS, in the operator's words, as plain text with no markdown (**bold** reaches a person as asterisks). The dashboard row shows those two and stops there; the full summary is drill-down only, so keep writing it as fully as the next agent needs and let the display line be the short one. Over either bound the append is refused, never truncated.
8. Interim rule — NO peeking-based decisions before the window closes. While the test runs, you may report interim numbers if asked, but you must refuse to call a winner or stop a variant early on them. State why: repeatedly checking and acting on a running test inflates the false-positive rate (the sequential-testing / "peeking" problem) — a lead that looks significant mid-flight reverses far more often than the confidence level implies. If a variant is causing genuine harm (runaway spend, a policy issue), that is an incident to escalate, not an experiment result — handle it via /rescue, and mark the experiment confounded.
9. Evaluation — only when the window has closed. Once observation_due has passed — compare it against server_now's date from step 0, never against another action_log timestamp:
   • Check sample-size adequacy FIRST. Pull per-variant results (meta_ads_split_tests_get for Meta; google_ads_ad_performance_report / google_ads_rsa_assets_analyze for the Google Ads paths). If any variant is below its sample floor from step 2, the result is inconclusive-underpowered — do not read the metric delta as signal. Say so and stop here (optionally extend the window if traffic allows).
   • Per-variant verdict. For each challenger, call mureo_outcome_evaluate — the deterministic verdict avoids eyeballing. When the variant has its own step-7 entry, pass evaluation_of = that entry's index (its metrics_at_action is the baseline, so do not pass before: refused with evaluation_of_conflicts_with_before) and after = the variant campaign's metrics object from mureo_state_get, unchanged (including period and fetched_at), and then append the evaluation record via mureo_state_action_log_append: evaluation_of = the same index, evaluation = the evaluation object the tool returned, exactly as returned, and action = "Outcome evaluated" (a neutral label — the table carries the verdict, so do not restate it in action; do not pass summary — mureo builds it from the table). When one entry captured all variants, pass before = the variant's baseline from step 7 and after = its final numbers instead. Cross-read Meta's own winner_cell_id / confidence_interval from meta_ads_split_tests_get where present.
   • Declare one of three outcomes, honestly: winner (challenger beats control on the success metric, above noise, sample-adequate), no-difference (within the noise band — a real and useful result: it means the variable doesn't matter here, stop spending attention on it), or inconclusive (underpowered or confounded). Inconclusive is a valid outcome — never manufacture a winner to close the ticket.
10. Close out. Apply the outcome (approval gate): promote the winner (shift budget to it / adopt the winning creative or copy), and stop the losing variant (meta_ads_split_tests_end if ending a Meta study early is warranted, or pause the losing ad set / ad). Show current-vs-proposed and confirm before each mutation.
11. Offer to save the finding via /learn. A closed experiment — winner, no-difference, or inconclusive-because-underpowered — is exactly the reusable know-how the knowledge base is for. Propose the generalized lesson (e.g. "Video beat static image by 22% CPA for this Persona — prefer video-first creative briefs") and, on approval, save it through the /learn flow (mureo learn add, per that skill's conventions — do not duplicate its steps here). Never record account IDs or personal data.

12. Persist the report summary (best-effort): Call mureo_state_report_set with report="experiment" and a concise summary object so the read-only dashboard can render the experiment without re-running you. Follow this convention:

    • generated_at: ISO 8601 timestamp of this run — use server_now
    • period: the experiment window (start_time..end_time) and its status (running / evaluated)
    • totals: the account's headline figures, using the canonical metric vocabulary — spend, conversions, cpa, ctr, clicks, impressions. This is the block the dashboard renders as figures. Raw numbers only: 773957, not "¥773,957"; 0.0466, not "4.66%" — one of those keys carrying a string is refused, because it sits where the view reads a figure and renders as nothing. A key outside the vocabulary may ride along: it is stored, just not shown as a headline number. Omit totals entirely if this run gathered no account-level figures.
    • kpis: per-variant {metric_at_baseline, metric_final, verdict} and the success metric name — the breakdown, not the headline row
    • flags: notable items (e.g. ["underpowered_variant_B", "google_ads_manual_split_caveats_accepted"])
    • narrative: the outcome (winner / no-difference / inconclusive, with confidence) and what you propose next, at most 400 characters — the tool refuses a longer one rather than truncating it (a sentence cut in half is worse than a long one). Do not restate the figures and do not list the findings here: numbers belong in totals, findings in flags.

    Reflect the FINAL state, and persist this LAST — after every action_log entry and any winner-promotion / loser-stop you applied this run. This is best-effort: if mureo_state_report_set is unavailable (e.g. a pure file-mode host without the context MCP), skip it silently — the rest of this skill still works.

13. Persist the display contract (best-effort): Call mureo_state_display_set with what the DASHBOARD shows. This is a different audience from the summary above — STATE.json is written for the next agent, and this small bounded section is the only thing the operator's screen reads. Write it in the SAME pass as the report, from the SAME figures, and reach no new verdict here: every value below is one you already decided, and this step only renders it.

    • source: your own skill name — REQUIRED whenever you write any section, at most 24 characters. The contract is replaced wholesale by whoever writes it last, so this is what lets the card say whose answer it is showing. generated_at is stamped by the server — do not compute it.
    • nav_message: the ONE line the operator should act on today — where the test stands, e.g. "Variant B wins on CPA — promote it and stop the control". One line, not a summary of the report.
    • highlights: at most 3 chips, and the tone must match the verdict you already gave — good for a target met or a clear win, watch for something to keep an eye on, bad for something deteriorating. Choose the three that matter; a fourth is refused and nothing is dropped for you. Map a finding's severity to a chip tone: action → bad / watch → watch / positive → good. info does NOT become a highlight — a neutral note would spend one of the 3 chips an action or a win needed, and it is still in the report for whoever wants it.
    • breakdown.campaigns / .adgroups: one row per entity you judged, carrying figures you ALREADY hold — spend, mcpa (the measured CPA) and target_cpa — plus state from the closed set (target_met / improving / watch / worsening / no_data) and a note of at most 40 characters. Use campaigns for the campaigns under test and adgroups for per-variant ad sets / ad groups; a design-only run with no results yet omits breakdown rather than inventing rows. Omit a figure you do not have: 0 states a perfect CPA rather than the absence of one, and no_data is the state for an entity with too little delivery to judge.
    • proposals: the recommendations you did NOT carry out, one entry each — title (what to do) plus body (why, in one line) plus date = server_now's date as YYYY-MM-DD. Something you DID apply this run is the same entry with status: "done". Record each of these as a decision too — mureo_decision_record with status: "proposed", your reasoning in rationale and the figures you judged it on in metrics. This section is the screen and is replaced whole by the next skill that writes it; the decision record is the one that survives, and it is what stops the next session re-proposing something the operator already turned down.
    • stated_values: label + figure chips ONLY ({"label": "…", "value": …}) — the design and the result: {"label": "variants", "value": 2}, {"label": "days run", "value": 14}, {"label": "CPA lift", "value": "-18%"}. No prose notes here. A sentence lands in a numeric column and the tool refuses it; anything that needs a sentence stays in the report's narrative above, exactly where it goes today.

    The bounds, verbatim from the tool: The dashboard reads THIS section and nothing else — keep your reasoning where it already goes. Every bound below refuses the write rather than truncating it, because a sentence cut in half is worse than a long one. nav_message: one line, at most 80 characters. highlights: at most 3 items of {tone, text}, tone one of good/watch/bad, text at most 60 characters. proposals: {title, body, status, date}, title at most 30 and body at most 80 characters, status one of proposed/done. breakdown.campaigns / breakdown.adgroups: rows of {name, spend, mcpa, target_cpa, state, note} — the three figures are raw numbers, state is one of target_met/improving/watch/worsening/no_data, note at most 40 characters. stated_values: {label, value}, label at most 24 characters and value a raw number or a string of at most 12 characters — a sentence there is refused, because it lands in a numeric column. Do NOT write the KPI funnel or the daily chart: mureo computes both from the stored totals.

    Over a bound the write is REFUSED — shorten and rewrite, do not re-send the same sentence trimmed. Lead with the point and drop the connectives; a noun phrase is fine (CPA 12% over target, not The CPA is currently running about 12% above the target we agreed). Trimming one character at a time and calling again spends the run's context on a bound one rewrite would have met.

    You may not be the first writer today. display is REPLACED WHOLE and the last writer wins — there is no merge. Before you write it, read the current one (mureo_state_get). Of what another skill wrote TODAY, carry exactly one thing into your own write: its proposals that are still live — not yet done, and not contradicted by what you just found. Everything else you write from your own run alone, because a screen assembled from two runs shows a moment that never happened. And carry over NOTHING ELSE: never copy another skill's nav_message, highlights, breakdown or stated_values, which would put its judgement under your name when you cannot vouch for it. Name yourself in source so the screen says whose answer it is.

    This is best-effort: if mureo_state_display_set is unavailable (e.g. a pure file-mode host without the context MCP), skip it silently — the rest of this skill still works.