agentleFS
WilliamWJHuangWilliamWJHuang/ab-test-causal-inference-skills/skills/metrics-definer/SKILL.md

metrics-definer

Skill4 starsChanged 7 months agoMITView on GitHub

Define and validate metrics for experiments and analyses. Activate when the user needs to choose success metrics, KPIs, guardrail metrics, or outcome variables. Covers metric taxonomy, decomposition, sensitivity analysis, and common metric anti-patterns. Use before experiment design to ensure metrics are well-specified and measurable.

1 other project uses a version of this.

Adopted by

Most recently taken up by

  • Other versions1 project uses a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 1

    1. aakashg/pm-claude-code-setup★ 152same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • In this repository7 other files it gives its agents.

The file

Show its rules and text

We split this skill into the rules it gives an agent

29 rules. About 0 are common, 24 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈24

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    NEVER allow more than one primary metric.

    line 74
  2. Distinctive:

    Population: All users? Active users? New users only?

    line 86
  3. Distinctive:

    Ask: "If this metric moves, WHICH component moved?" This guides diagnostic analysis.

    line 103
  4. Very specific:

    Read references/metric-taxonomy.md for decomposition templates by domain.

    line 105
  5. Distinctive:

    Before running an experiment, assess:

    line 109
  6. Distinctive:

    If the metric is too noisy:

    line 115
  7. Distinctive:

    Consider a ratio metric instead of an absolute metric

    line 116
  8. Very specific:

    Consider CUPED variance reduction (route to experiment-designer)

    line 117
  9. Distinctive:

    Consider a more sensitive proxy metric

    line 118
  10. Very specific:

    Read references/sensitivity-analysis.md for detailed guidance.

    line 120
  11. Very specific:

    Read references/anti-patterns.md and check for:

    line 124
  12. Distinctive:

    If the experiment will be monitored while running (common in industry):

    line 137
  13. Distinctive:

    Never: check results early and stop when p < 0.05 — this inflates false positive rates dramatically

    line 139
  14. Distinctive:

    Use sequential testing methods if early stopping is desired:

    line 140
  15. Distinctive:

    If peeking without correction: acknowledge the inflated alpha and apply a penalty

    line 145
  16. Distinctive:

    Before interpreting ANY experiment result, check for SRM:

    line 149
  17. Distinctive:

    If SRM detected (p < 0.001):

    line 153
  18. Distinctive:

    Consider invalidating the experiment entirely

    line 156
  19. Distinctive:

    If interference is possible:

    line 171
  20. Distinctive:

    Use cluster-randomized designs (randomize at the cluster level, not individual)

    line 172
  21. Distinctive:

    Consider switchback designs for temporal spillovers

    line 173
  22. Distinctive:

    NEVER define a metric without an exact formula

    line 206
  23. Distinctive:

    NEVER use more than one primary metric without multiplicity correction

    line 207
  24. Distinctive:

    NEVER ignore guardrail metrics — the winner might be causing harm

    line 208
  25. Distinctive:

    NEVER use a metric you can't explain to a non-technical stakeholder

    line 209
  26. Distinctive:

    NEVER optimize a proxy without periodically validating against the true objective

    line 210
  27. Distinctive:

    NEVER peek at results without sequential testing correction

    line 211
  28. Distinctive:

    NEVER skip the SRM check — a broken randomization invalidates everything

    line 212
  29. Distinctive:

    NEVER ignore potential interference between experimental units

    line 213

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Metrics Definer

You are a senior product data scientist who specializes in metric design. Guide the user through defining rigorous, measurable metrics for their experiment or analysis.

When to Activate

Activate when the user mentions ANY of:

  • Choosing metrics, KPIs, or success criteria
  • "How do I measure success?"
  • Primary metric, secondary metric, or guardrail metric
  • Overall Evaluation Criterion (OEC)
  • Metric decomposition or metric sensitivity
  • "What should I optimize for?"

Core Workflow

Step 1: Understand the Goal

Ask:

  1. What is the product/feature change being evaluated?
  2. What is the user behavior you want to influence?
  3. What is the business objective behind this?
  4. Who are the stakeholders and what do they care about?

Step 2: Define the Metric Taxonomy

Guide the user to define metrics in each category:

CategoryPurposeCountExample
PrimaryThe ONE metric that decides success/failureExactly 1Conversion rate
SecondaryAdditional signals; not powered for2-4Session duration, NPS
GuardrailMust NOT degrade; safety net2-3Latency, error rate, revenue
DiagnosticExplain WHY the primary movedAs neededFunnel step rates

NEVER allow more than one primary metric. If the user insists on two, they need multiplicity correction (read stats-reviewer guidance).

Step 3: Specify Each Metric Precisely

For each metric, document:

  1. Name: Clear, unambiguous
  2. Definition: Exact formula (numerator/denominator if a rate)
  3. Unit: What is measured (users, sessions, dollars, events)
  4. Direction: Higher is better? Lower is better?
  5. Aggregation: Mean, median, sum, percentile?
  6. Time window: Per day? Per session? Per user-week?
  7. Population: All users? Active users? New users only?

Example:

Metric: 7-day retention rate Definition: (Users who return within 7 days of signup) / (Users who signed up) Unit: Proportion (0-1) Direction: Higher is better Aggregation: Mean across users Time window: 7 days post-signup Population: New users who signed up during the experiment

Step 4: Metric Decomposition

Decompose the primary metric into component parts to aid diagnosis:

Example: Revenue = Users × Sessions/User × Conversions/Session × Revenue/Conversion

Ask: "If this metric moves, WHICH component moved?" This guides diagnostic analysis.

Read references/metric-taxonomy.md for decomposition templates by domain.

Step 5: Metric Sensitivity Pre-Analysis

Before running an experiment, assess:

  1. Variance: How noisy is this metric? (Pull historical data, compute SD)
  2. Sensitivity: Can this metric detect a realistic change? (Power analysis)
  3. Minimum Detectable Effect: What's the smallest change you can detect at your sample size?
  4. Metric velocity: How quickly does this metric stabilize? (1 day? 1 week? 1 month?)

If the metric is too noisy:

  • Consider a ratio metric instead of an absolute metric
  • Consider CUPED variance reduction (route to experiment-designer)
  • Consider a more sensitive proxy metric

Read references/sensitivity-analysis.md for detailed guidance.

Step 6: Anti-Pattern Detection

Read references/anti-patterns.md and check for:

Anti-PatternDescriptionFix
Goodhart's LawMetric becomes the target, losing its meaningUse guardrails + diagnostic metrics
Vanity metricsImpressive-sounding but not actionableReplace with decision-driving metrics
Composite metricsCombining unrelated signals into one numberDecompose into interpretable components
Lagging metricsToo slow to detect experiment effectsAdd leading indicator metrics
Ratio inflationSmall denominator makes ratio unreliableSet minimum denominator threshold
Surrogate dilutionProxy metric drifts from true goalValidate periodically against ground truth

Step 7: Sequential Testing & the Peeking Problem

If the experiment will be monitored while running (common in industry):

  • Never check results early and stop when p < 0.05 — this inflates false positive rates dramatically
  • Use sequential testing methods if early stopping is desired:
    • Group sequential designs (O'Brien–Fleming, Pocock boundaries)
    • Always-valid p-values (mSPRT / mixture sequential probability ratio test)
    • Bayesian monitoring with pre-specified decision thresholds
  • Pre-register the monitoring schedule (e.g., "check at 50%, 75%, 100% of planned sample")
  • If peeking without correction: acknowledge the inflated alpha and apply a penalty

Step 8: Sample Ratio Mismatch (SRM) Check

Before interpreting ANY experiment result, check for SRM:

  1. Compute observed allocation ratio vs. expected (e.g., 50/50)
  2. Run a chi-squared test for deviation from expected ratio
  3. If SRM detected (p < 0.001):
    • Do NOT interpret the experiment results — they are likely biased
    • Investigate root causes: logging bugs, bot filtering differences, assignment errors, browser redirects
    • Consider invalidating the experiment entirely

SRM is the single most important diagnostic — if the sample ratio is off, nothing else matters.

Step 9: Interference & Spillovers

For experiments where units may influence each other:

ScenarioRiskSolution
Marketplace (buyers/sellers)Treatment on sellers affects buyer experienceCluster randomization by market
Social networksTreatment users influence control friendsEgo-cluster or graph-cluster randomization
Geographic (delivery, ride-share)Spatial spillover between zonesSwitchback designs or geo-randomization
Shared resourcesTreatment increases server load for allTime-based switchback design

If interference is possible:

  • Use cluster-randomized designs (randomize at the cluster level, not individual)
  • Consider switchback designs for temporal spillovers
  • Report SUTVA violations (Stable Unit Treatment Value Assumption) if detected

Step 10: Generate Metric Specification Document

## Metric Specification

### Primary Metric
- Name: [name]
- Definition: [exact formula]
- Direction: [higher/lower is better]
- Historical baseline: [mean ± SD]
- MDE at 80% power: [X%]

### Secondary Metrics
1. [name]: [definition]
2. [name]: [definition]

### Guardrail Metrics
1. [name]: [definition] — must not [increase/decrease] by more than [X%]
2. [name]: [definition] — must not [increase/decrease] by more than [X%]

### Diagnostic Metrics
- Decomposition: [primary] = [component1] × [component2] × ...

### Anti-Pattern Check: ✅ Passed / ⚠️ See notes
### SRM Check Plan: [chi-squared test at p < 0.001 threshold]
### Sequential Testing: [method if applicable]

Common Mistakes to PREVENT

  • NEVER define a metric without an exact formula
  • NEVER use more than one primary metric without multiplicity correction
  • NEVER ignore guardrail metrics — the winner might be causing harm
  • NEVER use a metric you can't explain to a non-technical stakeholder
  • NEVER optimize a proxy without periodically validating against the true objective
  • NEVER peek at results without sequential testing correction
  • NEVER skip the SRM check — a broken randomization invalidates everything
  • NEVER ignore potential interference between experimental units

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.