exp-lens-iterative-learning
Skill1 starsChanged 8 months agoMITView on GitHub
Create Iterative Learning experimental design diagram showing factor space exploration, adaptive allocation, and next-experiment recommendations. Decision-Theoretic lens answering "How does this maximize learning per cost?
In this repository59 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- arch-lens-c4-containerSkill
- arch-lens-concurrencySkill
- arch-lens-data-lineageSkill
- arch-lens-deploymentSkill
- arch-lens-developmentSkill
- arch-lens-error-resilienceSkill
- arch-lens-module-dependencySkill
- arch-lens-operationalSkill
- arch-lens-process-flowSkill
- arch-lens-repository-accessSkill
- arch-lens-scenariosSkill
- arch-lens-securitySkill
- arch-lens-state-lifecycleSkill
- audit-archSkill
- audit-bugsSkill
- audit-cohesionSkill
- audit-defense-standardsSkill
- audit-frictionSkill
- audit-implSkill
- audit-testsSkill
- design-guardsSkill
- dry-walkthroughSkill
- elaborate-phaseSkill
- exp-lens-benchmark-representativenessSkill
- exp-lens-causal-assumptionsSkill
- exp-lens-comparator-constructionSkill
- exp-lens-error-budgetSkill
- exp-lens-estimand-claritySkill
- exp-lens-exploratory-confirmatorySkill
- exp-lens-fair-comparisonSkill
- exp-lens-governance-riskSkill
- exp-lens-measurement-validitySkill
- exp-lens-pipeline-integritySkill
- exp-lens-randomization-blockingSkill
- exp-lens-reproducibility-artifactsSkill
- exp-lens-sensitivity-robustnessSkill
- exp-lens-severity-testingSkill
- exp-lens-unit-interferenceSkill
- exp-lens-validity-threatsSkill
- exp-lens-variance-stabilitySkill
- id-slopSkill
- implement-worktreeSkill
- investigateSkill
- make-arch-diagSkill
- make-experiment-diagSkill
- make-groupsSkill
- make-planSkill
- make-reqSkill
- make-scenariosSkill
- mermaidSkill
- plan-arcSkill
- rectifySkill
- review-approachSkill
- update-architectureSkill
- update-reqsSkill
- update-specsSkill
- verify-diagSkill
- verify-proposed-diagSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 3 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 30
Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …
- Very specific: line 38
BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY
- Very specific: line 50
Look for: hyperparameter, config, param, factor, level, range, grid, sweep
- Very specific: line 54
Look for: interaction, ablation, factorial, combination, cross, joint
- Very specific: line 62
Look for: bayesian_opt, optuna, hyperband, successive, early_stop, adaptive, bandit
- Very specific: line 111
Write the diagram to: temp/exp-lens-iterative-learning/exp_diag_iterative_learning_{YYYY-MM-DD_HHMMSS}.md
- Distinctive: line 175
STOP["{Stopping Rule}<br/>━━━━━━━━━━<br/>{threshold or condition}"] end
- Distinctive: line 235
Before creating the diagram, verify:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Iterative Learning Experimental Design Lens
Philosophical Mode: Decision-Theoretic Primary Question: "How does this maximize learning per cost?" Focus: Factor Selection, Interaction Probing, Adaptive Allocation, Stopping Rules, Next-Experiment Planning
When to Use
- Planning a sequence of experiments
- Optimizing hyperparameter search
- Ablation study design
- User invokes
/exp-lens-iterative-learningor/make-experiment-diag iterative
Critical Constraints
NEVER:
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Recommend one-factor-at-a-time exploration when interactions are plausible
ALWAYS:
- Evaluate exploration efficiency against the key uncertainty being reduced
- Identify high-value unexplored regions of the factor space
- Assess whether the stopping rule is principled or arbitrary
- Surface interaction structure that one-factor-at-a-time designs would miss
- BEFORE creating any diagram, LOAD the
/mermaidskill using the Skill tool - this is MANDATORY
Analysis Workflow
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Factor Space
- Find all experimental factors and their ranges
- Look for:
hyperparameter,config,param,factor,level,range,grid,sweep
Interaction Structure
- Find evidence of factor interactions
- Look for:
interaction,ablation,factorial,combination,cross,joint
Cost & Resource Model
- Find compute/time/cost constraints
- Look for:
budget,cost,time,gpu,compute,resource,quota,limit
Sequential Decision Logic
- Find adaptive or sequential experiment logic
- Look for:
bayesian_opt,optuna,hyperband,successive,early_stop,adaptive,bandit
Learning Objectives
- Find what uncertainty the experiments aim to reduce
- Look for:
objective,goal,hypothesis,question,target,uncertainty,explore
Step 2: Map the Design Space
Map the design space (factors x levels), identify which regions have been explored, which interactions are probed, and what the next high-value experiment would be. Assess whether the experimental sequence is efficient (reducing key uncertainty) or wasteful (one-factor-at-a-time in interacting systems).
Step 3: CRITICAL — Analyze Learning Efficiency
For every factor and experiment round:
- Information gain: What uncertainty does this experiment reduce? What does it leave unresolved?
- Interaction risk: Are there factors that likely interact? Does the design probe those interactions?
- Cost-efficiency: What is the cost per unit of learning? Is there a cheaper design with equivalent learning value?
- Exploration-exploitation: Is the strategy exploring new regions or exploiting known good regions prematurely?
- Stopping rule: Is the stopping criterion based on a principled uncertainty threshold, or is it arbitrary (fixed budget, time limit)?
Distinguish clearly:
- Full factorial: Explores all combinations — exhaustive but expensive, catches all interactions
- Fractional factorial: Efficient but aliases some interaction effects
- One-factor-at-a-time: Simple but misses interactions — inefficient in correlated spaces
- Adaptive/Bayesian: High efficiency when the response surface is smooth — risks local optima
Step 4: Create the Diagram
Use flowchart with:
Direction: LR (learning flows from exploration to refinement)
Subgraphs:
FACTOR SPACEEXPLORATION STRATEGYRESULTS SO FARNEXT EXPERIMENTSSTOPPING CRITERIA
Node Styling:
stateNodeclass: factors/parametershandlerclass: exploration strategiesphaseclass: experiment roundsoutputclass: results/learningscliclass: objectivesgapclass: unexplored high-value regionsdetectorclass: stopping criteria
Step 5: Write Output
Write the diagram to: temp/exp-lens-iterative-learning/exp_diag_iterative_learning_{YYYY-MM-DD_HHMMSS}.md
Output Template
# Iterative Learning Diagram: {System Name}
**Lens:** Iterative Learning (Decision-Theoretic)
**Question:** How does this maximize learning per cost?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Factor Space
| Factor | Range | Levels | Explored? | Interaction Risk |
|--------|-------|--------|-----------|-----------------|
| {Factor 1} | {min}–{max} | {n} | ✓ Full | Low |
| {Factor 2} | {min}–{max} | {n} | ✓ Partial | High |
| {Factor 3} | {min}–{max} | {n} | ✗ None | Unknown |
## Iterative Learning Design-Space Diagram
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Factors ["FACTOR SPACE"]
direction TB
F1["{Factor 1}<br/>━━━━━━━━━━<br/>{range}, {levels} levels"]
F2["{Factor 2}<br/>━━━━━━━━━━<br/>{range}, {levels} levels"]
F3["{Factor 3}<br/>━━━━━━━━━━<br/>{range}, {levels} levels"]
end
subgraph Strategy ["EXPLORATION STRATEGY"]
direction TB
STRAT["{Strategy Type}<br/>━━━━━━━━━━<br/>{e.g., Bayesian/Grid/OFAT}"]
end
subgraph Rounds ["RESULTS SO FAR"]
direction TB
R1["{Round 1}<br/>━━━━━━━━━━<br/>{n} runs, {finding}"]
R2["{Round 2}<br/>━━━━━━━━━━<br/>{n} runs, {finding}"]
end
subgraph Next ["NEXT EXPERIMENTS"]
direction TB
NX1["{Next Experiment 1}<br/>━━━━━━━━━━<br/>{rationale}"]
NX2["{Next Experiment 2}<br/>━━━━━━━━━━<br/>{rationale}"]
end
subgraph Stopping ["STOPPING CRITERIA"]
direction TB
STOP["{Stopping Rule}<br/>━━━━━━━━━━<br/>{threshold or condition}"]
end
OBJ["{Learning Objective}<br/>━━━━━━━━━━<br/>{key uncertainty to reduce}"]
UNEXPLORED["{High-Value Unexplored Region}<br/>━━━━━━━━━━<br/>{what is missing and why it matters}"]
OBJ --> F1
OBJ --> F2
OBJ --> F3
F1 --> STRAT
F2 --> STRAT
F3 --> STRAT
STRAT --> R1
R1 --> R2
R2 --> NX1
R2 --> NX2
NX1 --> STOP
NX2 --> STOP
UNEXPLORED -.->|missed| STRAT
%% CLASS ASSIGNMENTS %%
class F1,F2,F3 stateNode;
class STRAT handler;
class R1,R2 phase;
class NX1,NX2 output;
class OBJ cli;
class UNEXPLORED gap;
class STOP detector;Color Legend:
| Color | Category | Description |
| Dark Teal | Factors | Experimental factors and parameters |
| Orange | Strategy | Exploration strategy used |
| Purple | Rounds | Completed experiment rounds |
| Teal | Next | Recommended next experiments |
| Dark Blue | Objective | Learning objective |
| Yellow/Amber | Gaps | Unexplored high-value regions |
| Red | Stopping | Stopping criteria |
Efficiency Assessment
| Round | Cost | Key Learning | Uncertainty Reduced | Efficiency Rating |
| {Round 1} | {cost} | {finding} | {what resolved} | High/Med/Low |
| {Round 2} | {cost} | {finding} | {what resolved} | High/Med/Low |
Recommended Next Steps
| Priority | Experiment | Factors | Expected Learning | Cost Estimate |
| 1 | {description} | {factors to vary} | {uncertainty reduced} | {cost} |
| 2 | {description} | {factors to vary} | {uncertainty reduced} | {cost} |
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-sensitivity-robustness` - For factor sensitivity analysis
- `/exp-lens-error-budget` - For cost and resource constraint analysisDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

