exp-lens-sensitivity-robustness
Skill1 starsChanged 8 months agoMITView on GitHub
Create Sensitivity & Robustness experimental design analysis identifying load-bearing analytic choices and untested perturbations. Robustness lens answering "Which assumptions are load-bearing?
In this repository59 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- arch-lens-c4-containerSkill
- arch-lens-concurrencySkill
- arch-lens-data-lineageSkill
- arch-lens-deploymentSkill
- arch-lens-developmentSkill
- arch-lens-error-resilienceSkill
- arch-lens-module-dependencySkill
- arch-lens-operationalSkill
- arch-lens-process-flowSkill
- arch-lens-repository-accessSkill
- arch-lens-scenariosSkill
- arch-lens-securitySkill
- arch-lens-state-lifecycleSkill
- audit-archSkill
- audit-bugsSkill
- audit-cohesionSkill
- audit-defense-standardsSkill
- audit-frictionSkill
- audit-implSkill
- audit-testsSkill
- design-guardsSkill
- dry-walkthroughSkill
- elaborate-phaseSkill
- exp-lens-benchmark-representativenessSkill
- exp-lens-causal-assumptionsSkill
- exp-lens-comparator-constructionSkill
- exp-lens-error-budgetSkill
- exp-lens-estimand-claritySkill
- exp-lens-exploratory-confirmatorySkill
- exp-lens-fair-comparisonSkill
- exp-lens-governance-riskSkill
- exp-lens-iterative-learningSkill
- exp-lens-measurement-validitySkill
- exp-lens-pipeline-integritySkill
- exp-lens-randomization-blockingSkill
- exp-lens-reproducibility-artifactsSkill
- exp-lens-severity-testingSkill
- exp-lens-unit-interferenceSkill
- exp-lens-validity-threatsSkill
- exp-lens-variance-stabilitySkill
- id-slopSkill
- implement-worktreeSkill
- investigateSkill
- make-arch-diagSkill
- make-experiment-diagSkill
- make-groupsSkill
- make-planSkill
- make-reqSkill
- make-scenariosSkill
- mermaidSkill
- plan-arcSkill
- rectifySkill
- review-approachSkill
- update-architectureSkill
- update-reqsSkill
- update-specsSkill
- verify-diagSkill
- verify-proposed-diagSkill
The file
We split this skill into the rules it gives an agent
12 rules. About 0 are common, 10 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈10
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 30
Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …
- Distinctive: line 31
Treat "untested" as equivalent to "robust"
- Very specific: line 38
BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY
- Distinctive: line 50
Look for: choice, option, default, parameter, threshold, method, alternative
- Distinctive: line 54
Look for: ablation, without, remove, disable, vary, sweep, drop
- Distinctive: line 58
Look for: normalize, tokenize, augment, crop, resize, filter, clean, impute
- Distinctive: line 62
Look for: learning_rate, batch_size, epochs, dropout, hidden_size, temperature, alpha, beta
- Distinctive: line 66
Look for: shift, domain, transfer, cross, out_of_distribution, generalize, different
- Distinctive: line 85
Untested: No evidence either way
- Distinctive: line 89
If a diagram adds value, create a simplified flowchart.
- Very specific: line 104
Write the analysis to: temp/exp-lens-sensitivity-robustness/exp_diag_sensitivity_robustness_{YYYY-MM-DD_HHMMSS}.md
- Distinctive: line 211
Before creating the diagram, verify:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Sensitivity & Robustness Experimental Design Lens
Philosophical Mode: Robustness Primary Question: "Which assumptions are load-bearing?" Focus: Ablation Structure, Preprocessing Sensitivity, Metric Sensitivity, Hyperparameter Sensitivity, Distribution Shift
When to Use
- Results may depend on specific preprocessing choices
- Need to verify robustness of conclusions across conditions
- Ablation study seems incomplete or cherry-picked
- User invokes
/exp-lens-sensitivity-robustnessor/make-experiment-diag sensitivity
Critical Constraints
NEVER:
- Modify any source code or experiment files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Treat "untested" as equivalent to "robust"
ALWAYS:
- Build a full sensitivity matrix (choices x perturbation types)
- Classify every analytic choice as load-bearing, minor, or untested
- Flag cases where the most impactful choices are the least tested
- Distinguish between ablations that were run and choices that were simply fixed
- BEFORE creating any diagram, LOAD the
/mermaidskill using the Skill tool - this is MANDATORY
Analysis Workflow
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Analytic Choices Made
- Find all decision points in the analysis pipeline
- Look for: choice, option, default, parameter, threshold, method, alternative
Ablation Coverage
- Find which factors have been ablated
- Look for: ablation, without, remove, disable, vary, sweep, drop
Preprocessing Variations
- Find preprocessing steps that could be done differently
- Look for: normalize, tokenize, augment, crop, resize, filter, clean, impute
Hyperparameter Sensitivity
- Find which hyperparameters were tuned vs fixed
- Look for: learning_rate, batch_size, epochs, dropout, hidden_size, temperature, alpha, beta
Distribution/Environment Variations
- Find evidence of testing under different conditions
- Look for: shift, domain, transfer, cross, out_of_distribution, generalize, different
Step 2: Build the Sensitivity Matrix
Rows = analytic choices. Columns = perturbation types (remove, change, stress).
For each cell: Does the conclusion survive the perturbation?
Step 3: Classify Analytic Choices
CRITICAL — Analyze Assumption Load: For every analytic choice:
- What happens if this choice were made differently?
- Is there evidence from ablations, sweeps, or prior literature?
- Are the most impactful choices the least tested?
Classify each choice as:
- Load-bearing: Conclusion changes if this choice changes
- Minor: Conclusion is robust to changes in this choice
- Untested: No evidence either way
Step 4: Create the Optional Perturbation Diagram
If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.
Direction: TB (choices flow down through perturbation to conclusion stability)
Subgraphs: "ANALYTIC CHOICES", "PERTURBATIONS TESTED", "CONCLUSION STABILITY"
Node Styling:
stateNodeclass: analytic choice nodeshandlerclass: perturbation type nodesoutputclass: stable conclusion nodesgapclass: load-bearing untested choice nodesdetectorclass: sensitivity threshold nodes
Step 5: Write Output
Write the analysis to: temp/exp-lens-sensitivity-robustness/exp_diag_sensitivity_robustness_{YYYY-MM-DD_HHMMSS}.md
Output Template
# Sensitivity & Robustness Analysis: {Experiment Name}
**Lens:** Sensitivity & Robustness (Robustness)
**Question:** Which assumptions are load-bearing?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Sensitivity Matrix
| Analytic Choice | Remove | Change Value | Stress Test | Overall Classification |
|----------------|--------|-------------|-------------|------------------------|
| {choice name} | Stable / Fragile / Untested | Stable / Fragile / Untested | Stable / Fragile / Untested | Load-bearing / Minor / Untested |
## Load-Bearing Assumptions
| Assumption | Evidence Type | Impact if Changed | Tested? |
|------------|--------------|-------------------|---------|
| {assumption} | Ablation / Sweep / Literature / None | High / Medium / Low | Yes / No |
## Ablation Coverage Assessment
| Factor | Ablated? | Result | Interpretation |
|--------|----------|--------|----------------|
| {factor name} | Yes / No | {delta metric if yes} | Conclusion holds / Fragile / Unknown |
## Robustness Profile
| Dimension | Status | Notes |
|-----------|--------|-------|
| Preprocessing choices | Robust / Fragile / Untested | {detail} |
| Hyperparameter choices | Robust / Fragile / Untested | {detail} |
| Metric choices | Robust / Fragile / Untested | {detail} |
| Distribution shift | Robust / Fragile / Untested | {detail} |
## Perturbation Diagram (Optional)
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart TB
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Choices ["ANALYTIC CHOICES"]
CHOICE1["Preprocessing Choice<br/>━━━━━━━━━━<br/>{e.g., normalization}"]
CHOICE2["Model Choice<br/>━━━━━━━━━━<br/>{e.g., architecture}"]
LOAD_BEARING["Load-Bearing Choice<br/>━━━━━━━━━━<br/>{untested critical choice}"]
end
subgraph Perturbations ["PERTURBATIONS TESTED"]
PERTURB1["Remove<br/>━━━━━━━━━━<br/>Ablation"]
PERTURB2["Change Value<br/>━━━━━━━━━━<br/>Sensitivity sweep"]
THRESHOLD["Sensitivity Threshold<br/>━━━━━━━━━━<br/>{delta that changes conclusion}"]
end
subgraph Stability ["CONCLUSION STABILITY"]
STABLE["Stable Conclusion<br/>━━━━━━━━━━<br/>{holds under perturbation}"]
end
CHOICE1 --> PERTURB1
CHOICE2 --> PERTURB2
PERTURB1 --> THRESHOLD
PERTURB2 --> THRESHOLD
THRESHOLD --> STABLE
LOAD_BEARING -.->|"untested risk"| THRESHOLD
class CHOICE1,CHOICE2 stateNode;
class LOAD_BEARING gap;
class PERTURB1,PERTURB2 handler;
class THRESHOLD detector;
class STABLE output;Color Legend:
| Color | Category | Description |
| Teal | Analytic Choices | Decision points in the pipeline |
| Yellow | Load-Bearing Untested | Critical choices with no perturbation evidence |
| Orange | Perturbations | Types of tests applied |
| Red | Sensitivity Thresholds | Points where conclusion may change |
| Dark Teal | Stable Conclusions | Results robust to perturbation |
Recommendations
- {Most urgent ablation to run — highest impact untested choice}
- {Preprocessing sensitivity test needed}
- {Distribution shift or domain generalization test needed}
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-estimand-clarity` - For clarifying which conclusions are being stress-tested
- `/exp-lens-iterative-learning` - For tracking robustness improvements across experiment iterationsDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

