Evidence-Based Marketing Decisions

A decision framework for AI agents managing marketing accounts through mureo. This skill teaches agents to distinguish signal from noise, avoid premature optimization, and only commit to strategy changes backed by sufficient evidence.

Why This Matters

Marketing data is noisy. A campaign's CPA can swing 30% day-to-day from random variation alone. Without statistical rigor, agents will:

• Chase noise: "CPA dropped yesterday, the keyword change worked!" (It might just be Tuesday.)
• Oscillate: Undo Monday's changes on Wednesday because metrics dipped, then redo them Friday.
• Overfit: Draw conclusions from 12 conversions when 50+ are needed for reliability.
• Contaminate: Attribute an improvement to one change when three changes happened simultaneously.

The antidote: observe, wait, verify, then act.

The Evidence Lifecycle

Every action that modifies a campaign enters this lifecycle. The agent tracks it via action_log entries in STATE.json.

Action taken (e.g., add negative keywords)
    │
    ├── Record observation_due + a baseline (metrics_at_action) in action_log
    │
    ▼
[OBSERVING]  ── Do NOT draw conclusions yet
    │             Wait for the observation window to pass
    │
    ├── Observation window elapsed, collect current metrics
    │
    ▼
[CANDIDATE]  ── "This looks like it worked" or "This didn't help"
    │             But one observation is NOT enough
    │
    ├── Wait for a second observation period to confirm
    │
    ▼
[VALIDATED]  ── Consistent improvement across 2+ observation periods
    │             NOW you can recommend a strategy change
    │
    ▼
[APPLIED]    ── User approved, STRATEGY.md updated

At any stage:
[REJECTED]   ── Not significant, contradicted, or confounded by concurrent actions

Critical rule: OBSERVING and CANDIDATE findings are NOT actionable. Only VALIDATED insights should influence strategy.

Observation Windows

Different actions need different wait times before evaluation:

Do NOT evaluate an action before its observation window has passed.

The why column is platform-conditional; the window is not. A platform that does not select delivery by a bid has no learning phase to wait out, but its numbers still need the same time to accumulate — keep the window and drop the learning-reset explanation, rather than narrating a re-learn it cannot have.

Minimum Sample Sizes

Before drawing any conclusion about a metric change, verify sufficient data:

If the sample size is insufficient, the finding stays in OBSERVING. Do NOT promote it.

The outcome evaluator enforces these floors itself. mureo_outcome_evaluate and the automatic observation closure judge ONE primary metric (the decision's expected effect marked primary, else the campaign's bidding target, else CPA when both sides have conversions) and only when BOTH sides reach the floor for it, on the raw counts:

Below the floor the verdict is inconclusive with reason: insufficient_volume (volume_unknown when the count is missing; zero_baseline when the baseline is zero). The noise band (default ±10%) is outcome_noise_pct. Count metrics (conversions, clicks, impressions, spend) are compared per day, so a 7-day baseline against a 30-day snapshot is not read as growth. The automatic observation closure records such a result as inconclusive and closes the observation; it is not "no effect". To judge the change anyway, re-evaluate it manually with mureo_outcome_evaluate over a longer window, once both sides reach the floor.

Declare the primary effect. A single expected effect is primary by itself. When you declare several, mark exactly one "primary": true — the metric the change is meant to move ([{"metric": "spend", "direction": "down", "primary": true}, {"metric": "conversions", "direction": "hold"}] for a budget cut that must not cost conversions); the others are reference rows. A declaration of several without exactly one primary is refused with expected_primary_required. Use "direction": "hold" for "must not move": it is judged held inside the noise band and regressed outside it. A target decides when given — reaching it is improved even inside the band, and a move the right way that misses it is inconclusive (target_not_reached). A target on a count metric (spend, conversions, clicks, impressions) is the total over the evaluation window, not a per-day figure.

How to Evaluate an Outcome

When the observation window for an action has passed:

Step 1: Collect Before/After Metrics

metrics_at_action (recorded when action was taken):
  CPA: 5,200  |  Conversions: 45  |  Clicks: 1,200  |  CTR: 3.2%

metrics_now (collected after observation window):
  CPA: 4,100  |  Conversions: 58  |  Clicks: 1,400  |  CTR: 3.5%

Step 2: Check Sample Size

• Conversions before: 45, after: 58 → both > 30 → sufficient for CPA evaluation
• Clicks before: 1,200, after: 1,400 → both > 200 → sufficient for CVR evaluation

Step 3: Assess Magnitude and Consistency

Ask these questions:

1. Is the change large enough to matter?
   • CPA change < 5%: probably noise, even if "statistically significant"
   • CPA change 5-15%: meaningful if consistent
   • CPA change > 15%: strong signal, but verify it persists
2. Is the change consistent across the observation period?
   • Did CPA improve every day, or did one great day skew the average?
   • Look at daily/weekly trends, not just period totals
3. Were there confounding factors?
   • Did another change happen to the same campaign during the window?
   • Did seasonality, a holiday, or external event affect the data?
   • Did competitive pressure change? On an auction platform (Google Ads, Meta) that is a competitor entering or leaving the auction; on a platform that selects delivery some other way, ask what its own model exposes instead.
4. Does the direction match the hypothesis?
   • If you added negative keywords expecting CPA to drop, did CPA actually drop?
   • An improvement in an unexpected metric may be coincidence

Step 4: Classify the Finding

Step 5: Report with Confidence Level

When reporting findings, always state the evidence level:

GOOD (high confidence):
  "CPA improved 18% (5,200 → 4,100) after negative keyword cleanup.
   Observed over 14 days with 58 conversions. Consistent daily trend.
   Confirmed across 2 observation periods. [VALIDATED]"

BAD (jumping to conclusions):
  "CPA improved! The keyword changes are working great!"
   (No sample size, no observation window, no consistency check.)

Noise Guards for Each Command

/daily-check

• DO: Report current metrics and flag anomalies
• DO: Check if any action's observation_due date has passed and evaluate outcomes
• DO NOT: Recommend strategy changes based on day-to-day fluctuations
• SAY: "CPA is 12% above target today, but this is within normal daily variance. Monitoring."

/rescue

• DO: Take emergency action when metrics are clearly critical (>30% off target for 7+ days)
• DO: Give every change a baseline (mureo fills a campaign's metrics_at_action when omitted)
• DO NOT: Trigger rescue based on a single bad day
• THRESHOLD: At least 3 consecutive days of critical metrics before rescue actions

/search-term-cleanup, /creative-refresh, /budget-rebalance

• DO: Set observation_due when making changes; omit metrics_at_action to let mureo fill it
• DO: Consult past VALIDATED learnings before proposing new changes
• DO NOT: Reverse a previous action that is still in OBSERVING
• SAY: "The budget increase from 4/1 is still in its observation window (due 4/8). I recommend waiting before making further budget changes to this campaign."

/goal-review

• DO: Distinguish between metric movements backed by evidence vs noise
• DO: Highlight VALIDATED learnings that explain goal progress
• DO NOT: Attribute goal progress to specific actions without checking the evidence lifecycle
• SAY: "CPA improved 15% toward goal. This aligns with the VALIDATED finding from negative keyword cleanup on 3/20."

/weekly-report

• DO: Include an "Evidence Pipeline" section showing pending observations and validated insights
• DO: Rate confidence in reported improvements (low/medium/high based on lifecycle status)
• DO NOT: Present OBSERVING findings as confirmed wins

/competitive-scan

• DO: Look at trends over 4+ weeks, not week-to-week changes
• DO NOT: React to a single week's impression share dip

Recording Outcomes in action_log

When executing any write operation, record the context needed for future evaluation. On a campaign-level entry, omit metrics_at_action: mureo fills it from the campaign's STATE.json metrics, period and fetched_at included. Passed by hand, it is that campaign's metrics from mureo_state_get unchanged; figures the snapshot lacks (MTD spend, a segment) still take canonical names (../_mureo-strategy/SKILL.md → Performance Metrics) and a period:

{
  "timestamp": "2026-04-01T10:30:00+09:00",
  "action": "Added 15 negative keywords",
  "platform": "google_ads",
  "campaign_id": "12345",
  "command": "/search-term-cleanup",
  "summary": "Excluded informational queries misaligned with Persona",
  "metrics_at_action": {
    "cpa": 5200,
    "conversions": 45,
    "clicks": 1200,
    "impressions": 37500,
    "ctr": 0.032,
    "spend": 234000,
    "period": "LAST_30_DAYS"
  },
  "observation_due": "2026-04-15"
}

If the append answers baseline.source: "none" with a reason (baseline_stale, no_current_metrics, current_metrics_undated, platform_not_in_state, campaign_not_in_state), nothing was filled: append the entry again with the campaign's figures from mureo_state_get passed as metrics_at_action.

Which metrics to record by hand depends on the action type:

Consulting Past Evidence

Before proposing any action, check the action_log for:

1. Pending observations: Actions whose observation_due has not passed. Do not stack changes on the same campaign.
2. Past outcomes: Actions whose observation_due has passed. Evaluate and report the result.
3. Patterns: If a type of action has been VALIDATED before (e.g., "negative keyword cleanup consistently improves CPA by 10-20%"), reference it when proposing the same action again.
4. Failures: If a type of action was previously REJECTED (e.g., "device bid adjustments had no significant impact"), note this when the same action is proposed.

Anti-Patterns to Avoid

Summary

1. Every write action → record observation_due + a baseline (metrics_at_action, filled when omitted)
2. Before the window passes → DO NOT evaluate, DO NOT reverse
3. After the window passes → collect current metrics, check sample size
4. Sufficient data + meaningful change → CANDIDATE
5. Confirmed in 2nd period → VALIDATED → recommend strategy update
6. Not confirmed → REJECTED → document and move on

Be patient. Be rigorous. Let the data speak — but only when it has enough to say.