srm-check
Skill304 starsChanged 2 months agoMITView on GitHub
Automatically detect Sample Ratio Mismatch (SRM) in experiment or A/B test data before any analysis proceeds. SRM is a critical randomization integrity check — if the treatment/control split deviates significantly from the expected ratio, the experiment is compromised and results cannot be trusted. This skill acts as a safety gate that blocks analysis when randomization is broken.
In this repository65 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- always-compareSkill
- analysis-designSkill
- analyst-coreSkill
- archaeologySkill
- archive-analysisSkill
- auth-preflightSkill
- businessSkill
- causalSkill
- chart-to-driveSkill
- close-the-loopSkill
- codex-reviewSkill
- compare-datasetsSkill
- connect-dataSkill
- connect-snowflakeSkill
- context-compareSkill
- context-traceSkill
- data-inspectSkill
- data-mapSkill
- data-profilingSkill
- data-quality-checkSkill
- datasetsSkill
- distribution-profilerSkill
- evalSkill
- evaluate-graderSkill
- experiment-briefSkill
- experimentSkill
- exploreSkill
- exportSkill
- forecastSkill
- google-doc-exportSkill
- google-slides-exportSkill
- guardrailsSkill
- historySkill
- improve-contextSkill
- knowledge-bootstrapSkill
- log-correctionSkill
- maintain-contextSkill
- metric-specSkill
- metricsSkill
- monitor-evalsSkill
- notion-exportSkill
- paceSkill
- patternsSkill
- question-framingSkill
- question-routerSkill
- reliabilitySkill
- resume-pipelineSkill
- run-pipelineSkill
- runsSkill
- score-analysisSkill
- session-handoffSkill
- setup-dev-contextSkill
- setup-notionSkill
- setupSkill
- setup-slackSkill
- setup-snowflakeSkill
- stakeholder-communicationSkill
- stress-testSkill
- switch-datasetSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 8 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈8
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 41
Why SRM matters: If randomization is broken, treatment and control groups are NOT comparable.
- Very specific: line 51
Metadata: check for experiment config files in .knowledge/experiments/
- Very specific: line 58
If no metadata: default to equal allocation (50/50 for 2 groups, 33/33/33 for 3 groups, etc.)
- Distinctive: line 63
Use a chi-squared goodness-of-fit test:
- Distinctive: line 117
If any period shows WARNING or BLOCK, investigate before trusting the aggregate result.
- Distinctive: line 121
If the SRM check returns BLOCK:
- Distinctive: line 151
Always include the SRM check result at the top of any experiment analysis:
- Distinctive: line 165
If PASS: continue analysis.
- Distinctive: line 166
If WARNING: add a note to all findings: "Note: Marginal SRM detected (p = X.XX).
- Distinctive: line 167
If BLOCK: halt and report as described above.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Skill: SRM Check (Sample Ratio Mismatch)
Purpose
Automatically detect Sample Ratio Mismatch in experiment data before any analysis proceeds. SRM is a randomization integrity check — if the treatment/control split deviates significantly from the expected ratio, the experiment is compromised and results cannot be trusted. This skill acts as a safety gate that blocks analysis when randomization is broken.
When to Use
Apply this skill when:
- Loading any experiment or A/B test dataset — auto-fire on detection of treatment/control columns (e.g.,
variant,group,treatment,arm,experiment_group) - Before any treatment effect calculation — SRM must pass before comparing outcomes
- When the Experiment Analyzer agent starts — first step of any experiment analysis workflow
This skill auto-fires on experiment data detection. Do NOT wait to be asked.
Instructions
What Is SRM?
Sample Ratio Mismatch (SRM) occurs when the observed ratio of users in treatment vs. control deviates significantly from the expected ratio. For a 50/50 experiment with 10,000 users, you expect ~5,000 in each group. If you see 5,500 vs. 4,500, something is wrong with randomization.
SRM CHECK
━━━━━━━━━━
Expected ratio: 50/50 (or whatever was designed)
Observed ratio: [actual counts]
Test: Chi-squared goodness-of-fit
Decision: PASS (proceed) or BLOCK (halt analysis)Why SRM matters: If randomization is broken, treatment and control groups are NOT comparable. Any observed difference in outcomes could be caused by the broken randomization, not the treatment. SRM is the single most important validity check in experimentation.
Detection Logic
Step 1: Identify the experiment column
Scan the dataset for columns that indicate experiment assignment. Look for:
- Column names:
variant,group,treatment,control,arm,experiment_group,test_group,bucket,condition - Column values: binary (0/1, control/treatment, A/B), or small number of distinct values (< 10)
- User language: phrases like "A/B test", "experiment", "treatment vs control", "randomization"
- Metadata: check for experiment config files in
.knowledge/experiments/
If no experiment indicators are found: this skill does not apply. Continue with the user's request without mentioning SRM.
Step 2: Determine expected ratio
- If metadata is available (experiment config, brief, or user-specified): use the designed ratio
- If no metadata: default to equal allocation (50/50 for 2 groups, 33/33/33 for 3 groups, etc.)
- Ask the user if uncertain: "This looks like an experiment with [N] groups. Was the intended split equal, or was there a specific allocation ratio?"
Step 3: Run the SRM test
Use a chi-squared goodness-of-fit test:
from helpers.stats.experiment_stats import srm_check
# observed counts per group
result = srm_check(
observed_counts=[n_control, n_treatment],
expected_ratios=[0.5, 0.5], # or designed ratio
)
# Returns: chi2, p_value, verdict (PASS/WARNING/BLOCK), interpretation
chi2 = result["chi2"]
p_value = result["p_value"]
verdict = result["verdict"]For segmented SRM diagnosis (find WHICH segments have broken randomization):
from helpers.stats.experiment_stats import srm_diagnose
diag = srm_diagnose(assignments_df, segments=["platform", "country"])
# Returns per-segment SRM results to identify the source of the mismatchStep 4: Interpret and decide
| p-value | Verdict | Action |
| p > 0.05 | PASS | Randomization looks clean. Proceed with analysis. |
| 0.01 < p ≤ 0.05 | WARNING | Marginal SRM. Proceed with caution. Flag in report. |
| p ≤ 0.01 | BLOCK | SRM detected. HALT analysis. Investigate before proceeding. |
Step 5: Temporal stability check (if applicable)
If the experiment ran for multiple days/weeks, check SRM over time:
- Group data by time period (day, week, or other natural breakpoint based on experiment duration)
- Run SRM check for each period using the same chi-squared test
- Flag any period where p ≤ 0.05 (indicates intermittent SRM)
Why this matters: SRM can start mid-experiment due to bugs, infrastructure changes, or experiment modifications. A clean aggregate SRM might mask periods where randomization was broken.
Output format:
### SRM Over Time
| Period | Expected | Observed | Chi-squared | p-value | Verdict |
|--------|----------|----------|-------------|---------|---------|
| Week 1 | 50/50 | 5,012/4,988 | 0.058 | 0.81 | PASS |
| Week 2 | 50/50 | 5,300/4,700 | 36.0 | <0.001 | BLOCK |If any period shows WARNING or BLOCK, investigate before trusting the aggregate result.
When SRM is Detected (BLOCK)
If the SRM check returns BLOCK:
- HALT all downstream analysis — do NOT compute treatment effects, segment analyses, or recommendations. Results from a broken experiment are unreliable.
- Report the finding clearly:
## SRM CHECK: BLOCKED
| Group | Expected | Observed | Deviation |
|-------|----------|----------|-----------|
| Control | [N] ([X]%) | [N] ([X]%) | [+/- Y%] |
| Treatment | [N] ([X]%) | [N] ([X]%) | [+/- Y%] |
**Chi-squared test:** χ² = [value], p = [value]
**Verdict:** Sample Ratio Mismatch detected. Analysis halted.- Suggest investigation paths:
- Was there a bug in the assignment logic? (Check if certain user types were excluded from one group)
- Did the feature fail to load for some users? (Treatment users who never saw the treatment get dropped)
- Was there a data pipeline issue? (Events from one group not logged properly)
- Was the experiment stopped and restarted? (Can cause ratio imbalances)
- Offer options:
- "Investigate the cause of the SRM before analyzing results"
- "Proceed with analysis anyway (NOT RECOMMENDED — results may be invalid)"
- "Filter to a time window where the ratio was clean (if SRM started at a specific point)"
Output Format
Always include the SRM check result at the top of any experiment analysis:
## Validity Check: Sample Ratio
| Metric | Value |
|--------|-------|
| Expected ratio | [X/Y] |
| Observed | Control: [N] ([X]%), Treatment: [N] ([Y]%) |
| Chi-squared | χ² = [value] |
| p-value | [value] |
| **Verdict** | **PASS / WARNING / BLOCK** |If PASS: continue analysis. If WARNING: add a note to all findings: "Note: Marginal SRM detected (p = X.XX). Results should be interpreted with caution." If BLOCK: halt and report as described above.
Anti-Patterns
- Never skip the SRM check — even if the split "looks close enough," run the statistical test. A 51/49 split on 100K users is a significant SRM.
- Never proceed with analysis after a BLOCK unless the user explicitly overrides — invalid randomization means invalid results
- Never assume equal allocation without checking — some experiments use 80/20 or 90/10 splits by design
- Never check SRM only once — if the data spans multiple time periods (days/weeks), check SRM per-period to detect if the issue started at a specific point
- Never dismiss SRM as "close enough" — SRM is binary. Either randomization worked or it didn't. There is no "close enough" for randomization integrity.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

