exp-lens-error-budget
Skill1 starsChanged 8 months agoMITView on GitHub
Analyze statistical error budget showing Type I/II errors, power, minimum detectable effect, multiplicity corrections, and sequential monitoring. Statistical lens answering "Are error risks sized and controlled?
In this repository59 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- arch-lens-c4-containerSkill
- arch-lens-concurrencySkill
- arch-lens-data-lineageSkill
- arch-lens-deploymentSkill
- arch-lens-developmentSkill
- arch-lens-error-resilienceSkill
- arch-lens-module-dependencySkill
- arch-lens-operationalSkill
- arch-lens-process-flowSkill
- arch-lens-repository-accessSkill
- arch-lens-scenariosSkill
- arch-lens-securitySkill
- arch-lens-state-lifecycleSkill
- audit-archSkill
- audit-bugsSkill
- audit-cohesionSkill
- audit-defense-standardsSkill
- audit-frictionSkill
- audit-implSkill
- audit-testsSkill
- design-guardsSkill
- dry-walkthroughSkill
- elaborate-phaseSkill
- exp-lens-benchmark-representativenessSkill
- exp-lens-causal-assumptionsSkill
- exp-lens-comparator-constructionSkill
- exp-lens-estimand-claritySkill
- exp-lens-exploratory-confirmatorySkill
- exp-lens-fair-comparisonSkill
- exp-lens-governance-riskSkill
- exp-lens-iterative-learningSkill
- exp-lens-measurement-validitySkill
- exp-lens-pipeline-integritySkill
- exp-lens-randomization-blockingSkill
- exp-lens-reproducibility-artifactsSkill
- exp-lens-sensitivity-robustnessSkill
- exp-lens-severity-testingSkill
- exp-lens-unit-interferenceSkill
- exp-lens-validity-threatsSkill
- exp-lens-variance-stabilitySkill
- id-slopSkill
- implement-worktreeSkill
- investigateSkill
- make-arch-diagSkill
- make-experiment-diagSkill
- make-groupsSkill
- make-planSkill
- make-reqSkill
- make-scenariosSkill
- mermaidSkill
- plan-arcSkill
- rectifySkill
- review-approachSkill
- update-architectureSkill
- update-reqsSkill
- update-specsSkill
- verify-diagSkill
- verify-proposed-diagSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 7 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 30
Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …
- Very specific: line 38
BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY
- Distinctive: line 51
Look for: sample_size, power, n_samples, effect_size, alpha, significance
- Distinctive: line 56
Look for: test, compare, pvalue, p_value, bonferroni, fdr, holm, adjust, correct
- Distinctive: line 61
Look for: sequential, interim, early_stop, monitor, peek, accumulate, streaming
- Distinctive: line 66
Look for: threshold, alpha, significance, confidence, critical, reject, accept
- Distinctive: line 71
Look for: effect, difference, improvement, minimum, meaningful, practical, clinical
- Very specific: line 113
Write the analysis to: temp/exp-lens-error-budget/exp_diag_error_budget_{YYYY-MM-DD_HHMMSS}.md
- Distinctive: line 226
Before creating the diagram, verify:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Error Budget Experimental Design Lens
Philosophical Mode: Statistical Primary Question: "Are error risks sized and controlled?" Focus: Type I/II Errors, Power, Minimum Detectable Effect, Multiplicity, Sequential Monitoring
When to Use
- Need to verify statistical power before running an experiment
- Multiple comparisons are performed without a stated correction strategy
- Sequential testing or interim analysis is in use without defined stopping rules
- User invokes
/exp-lens-error-budgetor/make-experiment-diag error
Critical Constraints
NEVER:
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Accept default alpha=0.05 without checking whether it is appropriate for the decision context
ALWAYS:
- Enumerate every statistical test and account for its error contribution
- Distinguish per-test error rates from family-wise error rates
- Flag any sequential peeking without a formal stopping rule as a critical defect
- Evaluate whether the minimum detectable effect is practically meaningful, not just statistically chosen
- BEFORE creating any diagram, LOAD the
/mermaidskill using the Skill tool - this is MANDATORY
Analysis Workflow
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Sample Size & Power
- Find sample size calculations and power analyses
- Identify assumed effect sizes and the alpha used for sizing
- Look for: sample_size, power, n_samples, effect_size, alpha, significance
Multiple Comparisons
- Find how many statistical tests are performed in total
- Identify any multiplicity correction strategy
- Look for: test, compare, pvalue, p_value, bonferroni, fdr, holm, adjust, correct
Sequential Analysis
- Find whether data is analyzed as it arrives or at scheduled interim looks
- Identify formal stopping boundaries if any exist
- Look for: sequential, interim, early_stop, monitor, peek, accumulate, streaming
Decision Thresholds
- Find significance levels, confidence intervals, and decision rules
- Identify how raw test results are translated into accept/reject decisions
- Look for: threshold, alpha, significance, confidence, critical, reject, accept
Effect Size Context
- Find what effect sizes are practically meaningful for the decision
- Identify whether the minimum detectable effect is set by business context or convention
- Look for: effect, difference, improvement, minimum, meaningful, practical, clinical
Step 2: Build the Error Budget
Construct a complete accounting of error risks:
- What is the Type I error rate per test and family-wise across all tests?
- What is the power for the minimum practically meaningful effect?
- How many tests are performed and how is multiplicity handled (or not)?
- Is there sequential monitoring, and what are the stopping boundaries?
- Is each error tolerance justified by decision context, or is it a convention default?
Step 3: Analyze Error Allocation
CRITICAL — Analyze Error Allocation: For every statistical test:
- Is the error rate justified by the decision context (cost of false positive vs. false negative), or is it a default (alpha=0.05)?
- Are different tests allocated different error tolerances based on their importance?
- Does the power calculation assume an effect size that is realistic and practically significant?
- Is the family-wise error rate acceptable given the number of simultaneous decisions?
Rate each test's error budget alignment:
- ALIGNED: Error tolerances match the stakes of the decision
- CONVENTIONAL: Standard values used without explicit justification
- MISALIGNED: Error tolerances are inconsistent with decision consequences
Step 4: Create Optional Decision-Flow Diagram
This lens is primarily argumentative and produces tables. An optional decision-flow diagram may be added when the test structure is non-trivial.
Direction: TB (decision flows from data through tests to conclusions)
Minimal diagram structure: Data sources → Statistical tests → Decision thresholds → Conclusions, with error rates labeled on edges
Node Styling:
cliclass: Data sourceshandlerclass: Statistical testsdetectorclass: Significance thresholdsoutputclass: Decisions / conclusionsgapclass: Uncorrected multiplicity or missing stopping rules
Step 5: Write Output
Write the analysis to: temp/exp-lens-error-budget/exp_diag_error_budget_{YYYY-MM-DD_HHMMSS}.md
Output Template
# Error Budget Analysis: {System / Experiment Name}
**Lens:** Error Budget (Statistical)
**Question:** Are error risks sized and controlled?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Error Budget
| Test | Alpha | Power | MDE | Multiplicity Correction |
|------|-------|-------|-----|------------------------|
| {test name} | {alpha} | {power %} | {minimum detectable effect} | {bonferroni / fdr / none} |
## Operating Characteristics Summary
| Metric | Value | Status |
|--------|-------|--------|
| Total tests | {N} | |
| Family-wise Type I error (uncorrected) | {value} | {OK / WARNING / CRITICAL} |
| Family-wise Type I error (corrected) | {value} | {OK / WARNING / CRITICAL} |
| Minimum power across tests | {value} | {OK / WARNING / CRITICAL} |
| Sample size adequacy | {adequate / underpowered / unknown} | |
## Sequential Monitoring Plan
{If sequential monitoring is used:}
| Look | Timing | Stopping Boundary | Alpha Spent |
|------|--------|------------------|-------------|
| {interim N} | {calendar / information fraction} | {boundary value} | {alpha} |
{If no sequential monitoring: "No interim analyses planned — full sequential peeking risk applies if data is inspected before planned end."}
## Decision Alignment Assessment
| Test | Decision Stakes | Error Tolerance Justification | Alignment |
|------|----------------|------------------------------|-----------|
| {test} | {what decision it drives} | {why this alpha/power} | {ALIGNED / CONVENTIONAL / MISALIGNED} |
## Optional Decision-Flow Diagram
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 40, 'rankSpacing': 50, 'curve': 'basis'}}}%%
flowchart TB
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Data ["DATA SOURCES"]
DS1["Data Source 1<br/>━━━━━━━━━━<br/>N observations"]
DS2["Data Source 2<br/>━━━━━━━━━━<br/>N observations"]
end
subgraph Tests ["STATISTICAL TESTS"]
T1["Primary Test<br/>━━━━━━━━━━<br/>alpha={a} power={p}"]
T2["Secondary Test<br/>━━━━━━━━━━<br/>alpha={a} power={p}"]
MULTI["Multiplicity<br/>━━━━━━━━━━<br/>correction applied?"]
end
subgraph Thresholds ["DECISION THRESHOLDS"]
THR["Significance Gate<br/>━━━━━━━━━━<br/>p < alpha"]
end
subgraph Decisions ["CONCLUSIONS"]
DEC1["Accept Variant<br/>━━━━━━━━━━<br/>ship / launch"]
DEC2["Reject Variant<br/>━━━━━━━━━━<br/>do not ship"]
UNCORR["Uncorrected Tests<br/>━━━━━━━━━━<br/>inflated Type I error"]
end
DS1 -->|"alpha={a}"| T1
DS2 -->|"alpha={a}"| T2
T1 --> MULTI
T2 --> MULTI
MULTI -->|"corrected"| THR
MULTI -.->|"no correction"| UNCORR
THR -->|"p < alpha"| DEC1
THR -->|"p >= alpha"| DEC2
class DS1,DS2 cli;
class T1,T2 handler;
class MULTI,THR detector;
class DEC1,DEC2 output;
class UNCORR gap;Color Legend:
| Color | Category | Description |
| Dark Blue | Data | Data sources feeding statistical tests |
| Orange | Tests | Statistical tests with error parameters |
| Red | Thresholds | Significance gates and decision boundaries |
| Dark Teal | Decisions | Conclusions reached from tests |
| Yellow | Gaps | Uncorrected multiplicity or missing stopping rules |
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/make-experiment-diag` - Parent skill for experimental lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-severity-testing` - For assessing whether tests are capable of detecting the errors they claim to test for
- `/exp-lens-variance-stability` - For diagnosing variance inflation that undermines power calculationsDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

