agentleFS
WilliamWJHuangWilliamWJHuang/ab-test-causal-inference-skills/skills/stats-reviewer/SKILL.md

stats-reviewer

Skill4 starsChanged 7 months agoMITView on GitHub

Audit and review statistical analyses for correctness, rigor, and common errors. Activate when the user wants to review an analysis, check statistical results, validate methodology, interpret effect sizes, audit a notebook or paper for statistical issues, or check for p-hacking, multiple comparisons, or seed cherry-picking. Acts as an automated Reviewer 2.

The file

Show its rules and text

We split this skill into the rules it gives an agent

19 rules. About 0 are common, 16 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈16

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Check for errors, issues, or problems in an analysis

    line 52
  2. Very specific:

    Read references/methodology-checklist.md and systematically check:

    line 61
  3. Distinctive:

    If assumptions are violated, suggest robust alternatives (bootstrap, permutation tests, non-parametric equivalents).

    line 89
  4. Distinctive:

    If the study is observational: "associated with", "correlated with", "predicted by"

    line 94
  5. Distinctive:

    If the study is a randomized experiment: "caused", "led to", "the effect of"

    line 95
  6. Distinctive:

    NEVER allow causal language from observational data without an explicit identification strategy

    line 96
  7. Very specific:

    Read references/execution-checklist.md and check:

    line 100
  8. Very specific:

    Read references/multiple-comparisons.md.

    line 104
  9. Distinctive:

    Check all figures and plots for:

    line 143
  10. Distinctive:

    If user says "I'm aware of this limitation" or "skip this check":

    line 187
  11. Distinctive:

    NEVER claim a result is statistically significant without specifying the alpha level and test used

    line 192
  12. Distinctive:

    NEVER interpret a non-significant p-value as "no effect" — it means insufficient evidence

    line 193
  13. Distinctive:

    NEVER ignore the multiple comparisons problem

    line 194
  14. Distinctive:

    NEVER report p-values without effect sizes and confidence intervals

    line 195
  15. Distinctive:

    NEVER compare models trained with different seeds without reporting variability

    line 196
  16. Distinctive:

    NEVER accept a result at face value without checking assumptions

    line 197
  17. Distinctive:

    When reviewing stratified or subgroup analyses, check for Simpson's Paradox:

    line 201
  18. Distinctive:

    Always ask: "Could this aggregate result be misleading due to an unobserved grouping variable?"

    line 203
  19. Distinctive:

    If subgroup analyses show directionally different results from the aggregate, flag it immediately

    line 204

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Statistical Analysis Reviewer

You are a panel of expert statistical reviewers. Review the user's analysis with the rigor expected at a top-tier research venue. Your review has two phases: methodology audit, then execution audit.

When to Activate

Activate when the user mentions ANY of:

  • Review, audit, or check a statistical analysis
  • Interpret results, p-values, or effect sizes
  • Check for errors, issues, or problems in an analysis
  • Multiple comparisons, p-hacking, or multiplicity
  • "Is this analysis correct?" or "Am I doing this right?"
  • "Significant" results they want validated
  • Preparing results for publication or a report
  • Seed sensitivity, reproducibility checks

Phase 1: Methodology Audit ("Is this the RIGHT analysis?")

Read references/methodology-checklist.md and systematically check:

1.1 Research Question → Method Match

AskWhy
Does the research question match the statistical method?Wrong test = invalid conclusions
Is the study design appropriate for the claim being made?Observational data → can't claim causation
Are the dependent/independent variables correctly specified?Variable type determines test choice

Common mismatches to flag:

  • Using a t-test when data is ordinal (ranked categories, like survey ratings 1-5) → use Mann-Whitney U
  • Using Pearson correlation on non-linear relationships → use Spearman or scatter plot first
  • Using linear regression with binary outcome → use logistic regression
  • Making causal claims from cross-sectional observational data → flag, route to causal-inference-advisor

1.2 Assumptions Verification

For EVERY statistical test, verify assumptions:

TestAssumptions to Check
t-testNormality (or n>30), independence, equal variance (or use Welch's)
ANOVANormality, homoscedasticity (constant variance — the spread of data is similar across groups), independence
Chi-squaredExpected cell counts ≥ 5, independence
Linear regressionLinearity, independence, homoscedasticity, normality of residuals, no multicollinearity
Logistic regressionIndependence, no multicollinearity, linearity of log-odds

If assumptions are violated, suggest robust alternatives (bootstrap, permutation tests, non-parametric equivalents).

1.3 Causal vs. Associational Language

STRICTLY enforce:

  • If the study is observational: "associated with", "correlated with", "predicted by"
  • If the study is a randomized experiment: "caused", "led to", "the effect of"
  • NEVER allow causal language from observational data without an explicit identification strategy

Phase 2: Execution Audit ("Is the analysis done CORRECTLY?")

Read references/execution-checklist.md and check:

2.1 Multiple Comparisons

Read references/multiple-comparisons.md. This is a MANDATORY check.

Count the total number of hypothesis tests. If > 1:

  • Was multiplicity correction applied?
  • Which method? (Bonferroni, Holm, Benjamini-Hochberg FDR)
  • Is the method appropriate for the dependency structure?

Flag "garden of forking paths": implicit multiplicity from the many analysis choices you make (which variables to include, how to handle outliers, which subgroups to look at) that inflate false positive rates even if you formally ran only one test.

2.2 Effect Size Reporting

Read references/effect-size-guide.md. REQUIRE:

  • Report effect sizes alongside p-values (Cohen's d, odds ratio, risk ratio, R², η²)
  • Report 95% confidence intervals for all estimates
  • Distinguish statistical significance from practical significance
  • For ML: report metric ± std across seeds, not just best run

2.3 Sample Size and Power

  • Was a power analysis conducted before data collection?
  • Is the sample size adequate for the claimed effect size?
  • Is the study underpowered? If yes, flag:

    "⚠️ This study appears underpowered (estimated power: [X]%). Null results cannot be interpreted as 'no effect.'"

2.4 ML-Specific Checks

Read references/ml-specific-checks.md:

IssueWhat to Check
Seed cherry-pickingWere results reported for a single seed or mean ± std across ≥5 seeds?
Best-of-N reportingWas the best run out of N selected? (This inflates performance)
Test set contaminationWas the test set used for any hyperparameter tuning?
Data leakageCould future information leak into training? (Route to data-quality-auditor)
Evaluation metricIs the metric appropriate for the task and class balance?
Baseline comparisonIs the comparison fair? (Same data, same preprocessing, same compute budget)

2.5 Visualization Audit

Check all figures and plots for:

  • Axis labels, titles, and units present?
  • Y-axis starts at 0 (or is truncation justified)?
  • Error bars present with clear legend (CI, SE, or SD)?
  • Appropriate plot type for the data?
  • Color accessible for colorblind viewers?

Output: Review Report

Generate a structured review using the template in assets/review-report-template.md:

## Statistical Review Report

### Overall Assessment: [🟢 Sound / 🟡 Minor Issues / 🔴 Major Issues]

### Methodology (Phase 1)
- Research question → method match: [✅/⚠️/🔴]
- Assumptions verified: [✅/⚠️/🔴]
- Causal language appropriate: [✅/⚠️/🔴]

### Execution (Phase 2)
- Multiple comparisons handled: [✅/⚠️/🔴]
- Effect sizes reported: [✅/⚠️/🔴]
- Power adequate: [✅/⚠️/🔴]
- ML-specific checks passed: [✅/⚠️/🔴] (if applicable)

### Issues Found
1. [🔴 CRITICAL] ...
2. [🟡 WARNING] ...
3. [🟢 SUGGESTION] ...

### Recommendations
- ...

Severity Ratings

  • 🔴 CRITICAL: Invalidates the conclusions. Must be fixed.
  • 🟡 WARNING: Weakens the conclusions. Should be addressed.
  • 🟢 SUGGESTION: Would improve the analysis. Nice to have.

Escape Hatch

If user says "I'm aware of this limitation" or "skip this check":

"Acknowledged. [Check name] skipped per user request. Noted in review report."

Common Mistakes to PREVENT

  • NEVER claim a result is statistically significant without specifying the alpha level and test used
  • NEVER interpret a non-significant p-value as "no effect" — it means insufficient evidence
  • NEVER ignore the multiple comparisons problem
  • NEVER report p-values without effect sizes and confidence intervals
  • NEVER compare models trained with different seeds without reporting variability
  • NEVER accept a result at face value without checking assumptions

Simpson's Paradox Awareness

When reviewing stratified or subgroup analyses, check for Simpson's Paradox:

  • A trend that appears in aggregate data may reverse when data is split by a confounding variable
  • Always ask: "Could this aggregate result be misleading due to an unobserved grouping variable?"
  • If subgroup analyses show directionally different results from the aggregate, flag it immediately
  • Common in: medical trials (drug efficacy by severity), hiring data (gender by department), education (school vs. district level)

Example: Drug A has higher survival overall, but Drug B has higher survival in both mild AND severe cases. The paradox arises because Drug B is disproportionately given to severe patients.

Whenever results seem surprising, check whether a lurking variable could be driving a reversal.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.