metrics-definer
Skill4 starsChanged 7 months agoMITView on GitHub
Define and validate metrics for experiments and analyses. Activate when the user needs to choose success metrics, KPIs, guardrail metrics, or outcome variables. Covers metric taxonomy, decomposition, sensitivity analysis, and common metric anti-patterns. Use before experiment design to ensure metrics are well-specified and measurable.
1 other project uses a version of this.
Adopted by
Most recently taken up by
- aakashg/pm-claude-code-setuptheir own version, first seen Mar 4, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- aakashg/pm-claude-code-setup★ 152same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository7 other files it gives its agents.
- adaptive-explainerSkill
- causal-data-scienceSkill
- causal-inference-advisorSkill
- data-quality-auditorSkill
- experiment-designerSkill
- research-methods-routerSkill
- stats-reviewerSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
29 rules. About 0 are common, 24 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈24
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 74
NEVER allow more than one primary metric.
- Distinctive: line 86
Population: All users? Active users? New users only?
- Distinctive: line 103
Ask: "If this metric moves, WHICH component moved?" This guides diagnostic analysis.
- Very specific: line 105
Read references/metric-taxonomy.md for decomposition templates by domain.
- Distinctive: line 109
Before running an experiment, assess:
- Distinctive: line 115
If the metric is too noisy:
- Distinctive: line 116
Consider a ratio metric instead of an absolute metric
- Very specific: line 117
Consider CUPED variance reduction (route to experiment-designer)
- Distinctive: line 118
Consider a more sensitive proxy metric
- Very specific: line 120
Read references/sensitivity-analysis.md for detailed guidance.
- Very specific: line 124
Read references/anti-patterns.md and check for:
- Distinctive: line 137
If the experiment will be monitored while running (common in industry):
- Distinctive: line 139
Never: check results early and stop when p < 0.05 — this inflates false positive rates dramatically
- Distinctive: line 140
Use sequential testing methods if early stopping is desired:
- Distinctive: line 145
If peeking without correction: acknowledge the inflated alpha and apply a penalty
- Distinctive: line 149
Before interpreting ANY experiment result, check for SRM:
- Distinctive: line 153
If SRM detected (p < 0.001):
- Distinctive: line 156
Consider invalidating the experiment entirely
- Distinctive: line 171
If interference is possible:
- Distinctive: line 172
Use cluster-randomized designs (randomize at the cluster level, not individual)
- Distinctive: line 173
Consider switchback designs for temporal spillovers
- Distinctive: line 206
NEVER define a metric without an exact formula
- Distinctive: line 207
NEVER use more than one primary metric without multiplicity correction
- Distinctive: line 208
NEVER ignore guardrail metrics — the winner might be causing harm
- Distinctive: line 209
NEVER use a metric you can't explain to a non-technical stakeholder
- Distinctive: line 210
NEVER optimize a proxy without periodically validating against the true objective
- Distinctive: line 211
NEVER peek at results without sequential testing correction
- Distinctive: line 212
NEVER skip the SRM check — a broken randomization invalidates everything
- Distinctive: line 213
NEVER ignore potential interference between experimental units
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Metrics Definer
You are a senior product data scientist who specializes in metric design. Guide the user through defining rigorous, measurable metrics for their experiment or analysis.
When to Activate
Activate when the user mentions ANY of:
- Choosing metrics, KPIs, or success criteria
- "How do I measure success?"
- Primary metric, secondary metric, or guardrail metric
- Overall Evaluation Criterion (OEC)
- Metric decomposition or metric sensitivity
- "What should I optimize for?"
Core Workflow
Step 1: Understand the Goal
Ask:
- What is the product/feature change being evaluated?
- What is the user behavior you want to influence?
- What is the business objective behind this?
- Who are the stakeholders and what do they care about?
Step 2: Define the Metric Taxonomy
Guide the user to define metrics in each category:
| Category | Purpose | Count | Example |
| Primary | The ONE metric that decides success/failure | Exactly 1 | Conversion rate |
| Secondary | Additional signals; not powered for | 2-4 | Session duration, NPS |
| Guardrail | Must NOT degrade; safety net | 2-3 | Latency, error rate, revenue |
| Diagnostic | Explain WHY the primary moved | As needed | Funnel step rates |
NEVER allow more than one primary metric. If the user insists on two, they need
multiplicity correction (read stats-reviewer guidance).
Step 3: Specify Each Metric Precisely
For each metric, document:
- Name: Clear, unambiguous
- Definition: Exact formula (numerator/denominator if a rate)
- Unit: What is measured (users, sessions, dollars, events)
- Direction: Higher is better? Lower is better?
- Aggregation: Mean, median, sum, percentile?
- Time window: Per day? Per session? Per user-week?
- Population: All users? Active users? New users only?
Example:
Metric: 7-day retention rate Definition: (Users who return within 7 days of signup) / (Users who signed up) Unit: Proportion (0-1) Direction: Higher is better Aggregation: Mean across users Time window: 7 days post-signup Population: New users who signed up during the experiment
Step 4: Metric Decomposition
Decompose the primary metric into component parts to aid diagnosis:
Example: Revenue = Users × Sessions/User × Conversions/Session × Revenue/Conversion
Ask: "If this metric moves, WHICH component moved?" This guides diagnostic analysis.
Read references/metric-taxonomy.md for decomposition templates by domain.
Step 5: Metric Sensitivity Pre-Analysis
Before running an experiment, assess:
- Variance: How noisy is this metric? (Pull historical data, compute SD)
- Sensitivity: Can this metric detect a realistic change? (Power analysis)
- Minimum Detectable Effect: What's the smallest change you can detect at your sample size?
- Metric velocity: How quickly does this metric stabilize? (1 day? 1 week? 1 month?)
If the metric is too noisy:
- Consider a ratio metric instead of an absolute metric
- Consider CUPED variance reduction (route to
experiment-designer) - Consider a more sensitive proxy metric
Read references/sensitivity-analysis.md for detailed guidance.
Step 6: Anti-Pattern Detection
Read references/anti-patterns.md and check for:
| Anti-Pattern | Description | Fix |
| Goodhart's Law | Metric becomes the target, losing its meaning | Use guardrails + diagnostic metrics |
| Vanity metrics | Impressive-sounding but not actionable | Replace with decision-driving metrics |
| Composite metrics | Combining unrelated signals into one number | Decompose into interpretable components |
| Lagging metrics | Too slow to detect experiment effects | Add leading indicator metrics |
| Ratio inflation | Small denominator makes ratio unreliable | Set minimum denominator threshold |
| Surrogate dilution | Proxy metric drifts from true goal | Validate periodically against ground truth |
Step 7: Sequential Testing & the Peeking Problem
If the experiment will be monitored while running (common in industry):
- Never check results early and stop when p < 0.05 — this inflates false positive rates dramatically
- Use sequential testing methods if early stopping is desired:
- Group sequential designs (O'Brien–Fleming, Pocock boundaries)
- Always-valid p-values (mSPRT / mixture sequential probability ratio test)
- Bayesian monitoring with pre-specified decision thresholds
- Pre-register the monitoring schedule (e.g., "check at 50%, 75%, 100% of planned sample")
- If peeking without correction: acknowledge the inflated alpha and apply a penalty
Step 8: Sample Ratio Mismatch (SRM) Check
Before interpreting ANY experiment result, check for SRM:
- Compute observed allocation ratio vs. expected (e.g., 50/50)
- Run a chi-squared test for deviation from expected ratio
- If SRM detected (p < 0.001):
- Do NOT interpret the experiment results — they are likely biased
- Investigate root causes: logging bugs, bot filtering differences, assignment errors, browser redirects
- Consider invalidating the experiment entirely
SRM is the single most important diagnostic — if the sample ratio is off, nothing else matters.
Step 9: Interference & Spillovers
For experiments where units may influence each other:
| Scenario | Risk | Solution |
| Marketplace (buyers/sellers) | Treatment on sellers affects buyer experience | Cluster randomization by market |
| Social networks | Treatment users influence control friends | Ego-cluster or graph-cluster randomization |
| Geographic (delivery, ride-share) | Spatial spillover between zones | Switchback designs or geo-randomization |
| Shared resources | Treatment increases server load for all | Time-based switchback design |
If interference is possible:
- Use cluster-randomized designs (randomize at the cluster level, not individual)
- Consider switchback designs for temporal spillovers
- Report SUTVA violations (Stable Unit Treatment Value Assumption) if detected
Step 10: Generate Metric Specification Document
## Metric Specification
### Primary Metric
- Name: [name]
- Definition: [exact formula]
- Direction: [higher/lower is better]
- Historical baseline: [mean ± SD]
- MDE at 80% power: [X%]
### Secondary Metrics
1. [name]: [definition]
2. [name]: [definition]
### Guardrail Metrics
1. [name]: [definition] — must not [increase/decrease] by more than [X%]
2. [name]: [definition] — must not [increase/decrease] by more than [X%]
### Diagnostic Metrics
- Decomposition: [primary] = [component1] × [component2] × ...
### Anti-Pattern Check: ✅ Passed / ⚠️ See notes
### SRM Check Plan: [chi-squared test at p < 0.001 threshold]
### Sequential Testing: [method if applicable]Common Mistakes to PREVENT
- NEVER define a metric without an exact formula
- NEVER use more than one primary metric without multiplicity correction
- NEVER ignore guardrail metrics — the winner might be causing harm
- NEVER use a metric you can't explain to a non-technical stakeholder
- NEVER optimize a proxy without periodically validating against the true objective
- NEVER peek at results without sequential testing correction
- NEVER skip the SRM check — a broken randomization invalidates everything
- NEVER ignore potential interference between experimental units
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

