agentleFS
TrecekTrecek/useful-claude-skills/.claude/skills/exp-lens-error-budget/SKILL.md

exp-lens-error-budget

Skill1 starsChanged 8 months agoMITView on GitHub

Analyze statistical error budget showing Type I/II errors, power, minimum detectable effect, multiplicity corrections, and sequential monitoring. Statistical lens answering "Are error risks sized and controlled?

The file

We split this skill into the rules it gives an agent

9 rules. About 0 are common, 7 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈7

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …

    line 30
  2. Very specific:

    BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

    line 38
  3. Distinctive:

    Look for: sample_size, power, n_samples, effect_size, alpha, significance

    line 51
  4. Distinctive:

    Look for: test, compare, pvalue, p_value, bonferroni, fdr, holm, adjust, correct

    line 56
  5. Distinctive:

    Look for: sequential, interim, early_stop, monitor, peek, accumulate, streaming

    line 61
  6. Distinctive:

    Look for: threshold, alpha, significance, confidence, critical, reject, accept

    line 66
  7. Distinctive:

    Look for: effect, difference, improvement, minimum, meaningful, practical, clinical

    line 71
  8. Very specific:

    Write the analysis to: temp/exp-lens-error-budget/exp_diag_error_budget_{YYYY-MM-DD_HHMMSS}.md

    line 113
  9. Distinctive:

    Before creating the diagram, verify:

    line 226

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Error Budget Experimental Design Lens

Philosophical Mode: Statistical Primary Question: "Are error risks sized and controlled?" Focus: Type I/II Errors, Power, Minimum Detectable Effect, Multiplicity, Sequential Monitoring

When to Use

  • Need to verify statistical power before running an experiment
  • Multiple comparisons are performed without a stated correction strategy
  • Sequential testing or interim analysis is in use without defined stopping rules
  • User invokes /exp-lens-error-budget or /make-experiment-diag error

Critical Constraints

NEVER:

  • Modify any source code files
  • Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
  • Accept default alpha=0.05 without checking whether it is appropriate for the decision context

ALWAYS:

  • Enumerate every statistical test and account for its error contribution
  • Distinguish per-test error rates from family-wise error rates
  • Flag any sequential peeking without a formal stopping rule as a critical defect
  • Evaluate whether the minimum detectable effect is practically meaningful, not just statistically chosen
  • BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

Analysis Workflow

Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

Sample Size & Power

  • Find sample size calculations and power analyses
  • Identify assumed effect sizes and the alpha used for sizing
  • Look for: sample_size, power, n_samples, effect_size, alpha, significance

Multiple Comparisons

  • Find how many statistical tests are performed in total
  • Identify any multiplicity correction strategy
  • Look for: test, compare, pvalue, p_value, bonferroni, fdr, holm, adjust, correct

Sequential Analysis

  • Find whether data is analyzed as it arrives or at scheduled interim looks
  • Identify formal stopping boundaries if any exist
  • Look for: sequential, interim, early_stop, monitor, peek, accumulate, streaming

Decision Thresholds

  • Find significance levels, confidence intervals, and decision rules
  • Identify how raw test results are translated into accept/reject decisions
  • Look for: threshold, alpha, significance, confidence, critical, reject, accept

Effect Size Context

  • Find what effect sizes are practically meaningful for the decision
  • Identify whether the minimum detectable effect is set by business context or convention
  • Look for: effect, difference, improvement, minimum, meaningful, practical, clinical

Step 2: Build the Error Budget

Construct a complete accounting of error risks:

  1. What is the Type I error rate per test and family-wise across all tests?
  2. What is the power for the minimum practically meaningful effect?
  3. How many tests are performed and how is multiplicity handled (or not)?
  4. Is there sequential monitoring, and what are the stopping boundaries?
  5. Is each error tolerance justified by decision context, or is it a convention default?

Step 3: Analyze Error Allocation

CRITICAL — Analyze Error Allocation: For every statistical test:

  • Is the error rate justified by the decision context (cost of false positive vs. false negative), or is it a default (alpha=0.05)?
  • Are different tests allocated different error tolerances based on their importance?
  • Does the power calculation assume an effect size that is realistic and practically significant?
  • Is the family-wise error rate acceptable given the number of simultaneous decisions?

Rate each test's error budget alignment:

  • ALIGNED: Error tolerances match the stakes of the decision
  • CONVENTIONAL: Standard values used without explicit justification
  • MISALIGNED: Error tolerances are inconsistent with decision consequences

Step 4: Create Optional Decision-Flow Diagram

This lens is primarily argumentative and produces tables. An optional decision-flow diagram may be added when the test structure is non-trivial.

Direction: TB (decision flows from data through tests to conclusions)

Minimal diagram structure: Data sources → Statistical tests → Decision thresholds → Conclusions, with error rates labeled on edges

Node Styling:

  • cli class: Data sources
  • handler class: Statistical tests
  • detector class: Significance thresholds
  • output class: Decisions / conclusions
  • gap class: Uncorrected multiplicity or missing stopping rules

Step 5: Write Output

Write the analysis to: temp/exp-lens-error-budget/exp_diag_error_budget_{YYYY-MM-DD_HHMMSS}.md


Output Template

# Error Budget Analysis: {System / Experiment Name}

**Lens:** Error Budget (Statistical)
**Question:** Are error risks sized and controlled?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Error Budget

| Test | Alpha | Power | MDE | Multiplicity Correction |
|------|-------|-------|-----|------------------------|
| {test name} | {alpha} | {power %} | {minimum detectable effect} | {bonferroni / fdr / none} |

## Operating Characteristics Summary

| Metric | Value | Status |
|--------|-------|--------|
| Total tests | {N} | |
| Family-wise Type I error (uncorrected) | {value} | {OK / WARNING / CRITICAL} |
| Family-wise Type I error (corrected) | {value} | {OK / WARNING / CRITICAL} |
| Minimum power across tests | {value} | {OK / WARNING / CRITICAL} |
| Sample size adequacy | {adequate / underpowered / unknown} | |

## Sequential Monitoring Plan

{If sequential monitoring is used:}

| Look | Timing | Stopping Boundary | Alpha Spent |
|------|--------|------------------|-------------|
| {interim N} | {calendar / information fraction} | {boundary value} | {alpha} |

{If no sequential monitoring: "No interim analyses planned — full sequential peeking risk applies if data is inspected before planned end."}

## Decision Alignment Assessment

| Test | Decision Stakes | Error Tolerance Justification | Alignment |
|------|----------------|------------------------------|-----------|
| {test} | {what decision it drives} | {why this alpha/power} | {ALIGNED / CONVENTIONAL / MISALIGNED} |

## Optional Decision-Flow Diagram

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 40, 'rankSpacing': 50, 'curve': 'basis'}}}%%
flowchart TB
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    subgraph Data ["DATA SOURCES"]
        DS1["Data Source 1<br/>━━━━━━━━━━<br/>N observations"]
        DS2["Data Source 2<br/>━━━━━━━━━━<br/>N observations"]
    end

    subgraph Tests ["STATISTICAL TESTS"]
        T1["Primary Test<br/>━━━━━━━━━━<br/>alpha={a} power={p}"]
        T2["Secondary Test<br/>━━━━━━━━━━<br/>alpha={a} power={p}"]
        MULTI["Multiplicity<br/>━━━━━━━━━━<br/>correction applied?"]
    end

    subgraph Thresholds ["DECISION THRESHOLDS"]
        THR["Significance Gate<br/>━━━━━━━━━━<br/>p < alpha"]
    end

    subgraph Decisions ["CONCLUSIONS"]
        DEC1["Accept Variant<br/>━━━━━━━━━━<br/>ship / launch"]
        DEC2["Reject Variant<br/>━━━━━━━━━━<br/>do not ship"]
        UNCORR["Uncorrected Tests<br/>━━━━━━━━━━<br/>inflated Type I error"]
    end

    DS1 -->|"alpha={a}"| T1
    DS2 -->|"alpha={a}"| T2
    T1 --> MULTI
    T2 --> MULTI
    MULTI -->|"corrected"| THR
    MULTI -.->|"no correction"| UNCORR
    THR -->|"p < alpha"| DEC1
    THR -->|"p >= alpha"| DEC2

    class DS1,DS2 cli;
    class T1,T2 handler;
    class MULTI,THR detector;
    class DEC1,DEC2 output;
    class UNCORR gap;

Color Legend:

ColorCategoryDescription
Dark BlueDataData sources feeding statistical tests
OrangeTestsStatistical tests with error parameters
RedThresholdsSignificance gates and decision boundaries
Dark TealDecisionsConclusions reached from tests
YellowGapsUncorrected multiplicity or missing stopping rules

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for experimental lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-severity-testing` - For assessing whether tests are capable of detecting the errors they claim to test for
- `/exp-lens-variance-stability` - For diagnosing variance inflation that undermines power calculations

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.