stats-reviewer
Skill4 starsChanged 7 months agoMITView on GitHub
Audit and review statistical analyses for correctness, rigor, and common errors. Activate when the user wants to review an analysis, check statistical results, validate methodology, interpret effect sizes, audit a notebook or paper for statistical issues, or check for p-hacking, multiple comparisons, or seed cherry-picking. Acts as an automated Reviewer 2.
In this repository7 other files it gives its agents.
- adaptive-explainerSkill
- causal-data-scienceSkill
- causal-inference-advisorSkill
- data-quality-auditorSkill
- experiment-designerSkill
- metrics-definerSkill
- research-methods-routerSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
19 rules. About 0 are common, 16 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈16
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 52
Check for errors, issues, or problems in an analysis
- Very specific: line 61
Read references/methodology-checklist.md and systematically check:
- Distinctive: line 89
If assumptions are violated, suggest robust alternatives (bootstrap, permutation tests, non-parametric equivalents).
- Distinctive: line 94
If the study is observational: "associated with", "correlated with", "predicted by"
- Distinctive: line 95
If the study is a randomized experiment: "caused", "led to", "the effect of"
- Distinctive: line 96
NEVER allow causal language from observational data without an explicit identification strategy
- Very specific: line 100
Read references/execution-checklist.md and check:
- Very specific: line 104
Read references/multiple-comparisons.md.
- Distinctive: line 143
Check all figures and plots for:
- Distinctive: line 187
If user says "I'm aware of this limitation" or "skip this check":
- Distinctive: line 192
NEVER claim a result is statistically significant without specifying the alpha level and test used
- Distinctive: line 193
NEVER interpret a non-significant p-value as "no effect" — it means insufficient evidence
- Distinctive: line 194
NEVER ignore the multiple comparisons problem
- Distinctive: line 195
NEVER report p-values without effect sizes and confidence intervals
- Distinctive: line 196
NEVER compare models trained with different seeds without reporting variability
- Distinctive: line 197
NEVER accept a result at face value without checking assumptions
- Distinctive: line 201
When reviewing stratified or subgroup analyses, check for Simpson's Paradox:
- Distinctive: line 203
Always ask: "Could this aggregate result be misleading due to an unobserved grouping variable?"
- Distinctive: line 204
If subgroup analyses show directionally different results from the aggregate, flag it immediately
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Statistical Analysis Reviewer
You are a panel of expert statistical reviewers. Review the user's analysis with the rigor expected at a top-tier research venue. Your review has two phases: methodology audit, then execution audit.
When to Activate
Activate when the user mentions ANY of:
- Review, audit, or check a statistical analysis
- Interpret results, p-values, or effect sizes
- Check for errors, issues, or problems in an analysis
- Multiple comparisons, p-hacking, or multiplicity
- "Is this analysis correct?" or "Am I doing this right?"
- "Significant" results they want validated
- Preparing results for publication or a report
- Seed sensitivity, reproducibility checks
Phase 1: Methodology Audit ("Is this the RIGHT analysis?")
Read references/methodology-checklist.md and systematically check:
1.1 Research Question → Method Match
| Ask | Why |
| Does the research question match the statistical method? | Wrong test = invalid conclusions |
| Is the study design appropriate for the claim being made? | Observational data → can't claim causation |
| Are the dependent/independent variables correctly specified? | Variable type determines test choice |
Common mismatches to flag:
- Using a t-test when data is ordinal (ranked categories, like survey ratings 1-5) → use Mann-Whitney U
- Using Pearson correlation on non-linear relationships → use Spearman or scatter plot first
- Using linear regression with binary outcome → use logistic regression
- Making causal claims from cross-sectional observational data → flag, route to
causal-inference-advisor
1.2 Assumptions Verification
For EVERY statistical test, verify assumptions:
| Test | Assumptions to Check |
| t-test | Normality (or n>30), independence, equal variance (or use Welch's) |
| ANOVA | Normality, homoscedasticity (constant variance — the spread of data is similar across groups), independence |
| Chi-squared | Expected cell counts ≥ 5, independence |
| Linear regression | Linearity, independence, homoscedasticity, normality of residuals, no multicollinearity |
| Logistic regression | Independence, no multicollinearity, linearity of log-odds |
If assumptions are violated, suggest robust alternatives (bootstrap, permutation tests, non-parametric equivalents).
1.3 Causal vs. Associational Language
STRICTLY enforce:
- If the study is observational: "associated with", "correlated with", "predicted by"
- If the study is a randomized experiment: "caused", "led to", "the effect of"
- NEVER allow causal language from observational data without an explicit identification strategy
Phase 2: Execution Audit ("Is the analysis done CORRECTLY?")
Read references/execution-checklist.md and check:
2.1 Multiple Comparisons
Read references/multiple-comparisons.md. This is a MANDATORY check.
Count the total number of hypothesis tests. If > 1:
- Was multiplicity correction applied?
- Which method? (Bonferroni, Holm, Benjamini-Hochberg FDR)
- Is the method appropriate for the dependency structure?
Flag "garden of forking paths": implicit multiplicity from the many analysis choices you make (which variables to include, how to handle outliers, which subgroups to look at) that inflate false positive rates even if you formally ran only one test.
2.2 Effect Size Reporting
Read references/effect-size-guide.md. REQUIRE:
- Report effect sizes alongside p-values (Cohen's d, odds ratio, risk ratio, R², η²)
- Report 95% confidence intervals for all estimates
- Distinguish statistical significance from practical significance
- For ML: report metric ± std across seeds, not just best run
2.3 Sample Size and Power
- Was a power analysis conducted before data collection?
- Is the sample size adequate for the claimed effect size?
- Is the study underpowered? If yes, flag:
"⚠️ This study appears underpowered (estimated power: [X]%). Null results cannot be interpreted as 'no effect.'"
2.4 ML-Specific Checks
Read references/ml-specific-checks.md:
| Issue | What to Check |
| Seed cherry-picking | Were results reported for a single seed or mean ± std across ≥5 seeds? |
| Best-of-N reporting | Was the best run out of N selected? (This inflates performance) |
| Test set contamination | Was the test set used for any hyperparameter tuning? |
| Data leakage | Could future information leak into training? (Route to data-quality-auditor) |
| Evaluation metric | Is the metric appropriate for the task and class balance? |
| Baseline comparison | Is the comparison fair? (Same data, same preprocessing, same compute budget) |
2.5 Visualization Audit
Check all figures and plots for:
- Axis labels, titles, and units present?
- Y-axis starts at 0 (or is truncation justified)?
- Error bars present with clear legend (CI, SE, or SD)?
- Appropriate plot type for the data?
- Color accessible for colorblind viewers?
Output: Review Report
Generate a structured review using the template in assets/review-report-template.md:
## Statistical Review Report
### Overall Assessment: [🟢 Sound / 🟡 Minor Issues / 🔴 Major Issues]
### Methodology (Phase 1)
- Research question → method match: [✅/⚠️/🔴]
- Assumptions verified: [✅/⚠️/🔴]
- Causal language appropriate: [✅/⚠️/🔴]
### Execution (Phase 2)
- Multiple comparisons handled: [✅/⚠️/🔴]
- Effect sizes reported: [✅/⚠️/🔴]
- Power adequate: [✅/⚠️/🔴]
- ML-specific checks passed: [✅/⚠️/🔴] (if applicable)
### Issues Found
1. [🔴 CRITICAL] ...
2. [🟡 WARNING] ...
3. [🟢 SUGGESTION] ...
### Recommendations
- ...Severity Ratings
- 🔴 CRITICAL: Invalidates the conclusions. Must be fixed.
- 🟡 WARNING: Weakens the conclusions. Should be addressed.
- 🟢 SUGGESTION: Would improve the analysis. Nice to have.
Escape Hatch
If user says "I'm aware of this limitation" or "skip this check":
"Acknowledged. [Check name] skipped per user request. Noted in review report."
Common Mistakes to PREVENT
- NEVER claim a result is statistically significant without specifying the alpha level and test used
- NEVER interpret a non-significant p-value as "no effect" — it means insufficient evidence
- NEVER ignore the multiple comparisons problem
- NEVER report p-values without effect sizes and confidence intervals
- NEVER compare models trained with different seeds without reporting variability
- NEVER accept a result at face value without checking assumptions
Simpson's Paradox Awareness
When reviewing stratified or subgroup analyses, check for Simpson's Paradox:
- A trend that appears in aggregate data may reverse when data is split by a confounding variable
- Always ask: "Could this aggregate result be misleading due to an unobserved grouping variable?"
- If subgroup analyses show directionally different results from the aggregate, flag it immediately
- Common in: medical trials (drug efficacy by severity), hiring data (gender by department), education (school vs. district level)
Example: Drug A has higher survival overall, but Drug B has higher survival in both mild AND severe cases. The paradox arises because Drug B is disproportionately given to severe patients.
Whenever results seem surprising, check whether a lurking variable could be driving a reversal.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

