exp-lens-benchmark-representativeness
Skill1 starsChanged 8 months agoMITView on GitHub
Create Benchmark Representativeness experimental design diagram showing coverage matrix, generalization gaps, and untested regions. Generalizability lens answering "Does this generalize beyond the test bed?
In this repository59 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- arch-lens-c4-containerSkill
- arch-lens-concurrencySkill
- arch-lens-data-lineageSkill
- arch-lens-deploymentSkill
- arch-lens-developmentSkill
- arch-lens-error-resilienceSkill
- arch-lens-module-dependencySkill
- arch-lens-operationalSkill
- arch-lens-process-flowSkill
- arch-lens-repository-accessSkill
- arch-lens-scenariosSkill
- arch-lens-securitySkill
- arch-lens-state-lifecycleSkill
- audit-archSkill
- audit-bugsSkill
- audit-cohesionSkill
- audit-defense-standardsSkill
- audit-frictionSkill
- audit-implSkill
- audit-testsSkill
- design-guardsSkill
- dry-walkthroughSkill
- elaborate-phaseSkill
- exp-lens-causal-assumptionsSkill
- exp-lens-comparator-constructionSkill
- exp-lens-error-budgetSkill
- exp-lens-estimand-claritySkill
- exp-lens-exploratory-confirmatorySkill
- exp-lens-fair-comparisonSkill
- exp-lens-governance-riskSkill
- exp-lens-iterative-learningSkill
- exp-lens-measurement-validitySkill
- exp-lens-pipeline-integritySkill
- exp-lens-randomization-blockingSkill
- exp-lens-reproducibility-artifactsSkill
- exp-lens-sensitivity-robustnessSkill
- exp-lens-severity-testingSkill
- exp-lens-unit-interferenceSkill
- exp-lens-validity-threatsSkill
- exp-lens-variance-stabilitySkill
- id-slopSkill
- implement-worktreeSkill
- investigateSkill
- make-arch-diagSkill
- make-experiment-diagSkill
- make-groupsSkill
- make-planSkill
- make-reqSkill
- make-scenariosSkill
- mermaidSkill
- plan-arcSkill
- rectifySkill
- review-approachSkill
- update-architectureSkill
- update-reqsSkill
- update-specsSkill
- verify-diagSkill
- verify-proposed-diagSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 3 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 30
Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …
- Distinctive: line 35
Document the relationship between benchmark selection and generalization claims
- Very specific: line 37
BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY
- Very specific: line 49
Look for: benchmark, dataset, test_suite, eval, corpus, split, GLUE, ImageNet
- Very specific: line 53
Look for: task, scenario, domain, category, difficulty, subset
- Very specific: line 61
Look for: generalize, real-world, production, deploy, robust, transfer, domain
- Very specific: line 107
Write the diagram to: temp/exp-lens-benchmark-representativeness/exp_diag_benchmark_representativeness_{YYYY-MM-DD_HHMMSS}.md
- Distinctive: line 228
Before creating the diagram, verify:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Benchmark Representativeness Experimental Design Lens
Philosophical Mode: Generalizability Primary Question: "Does this generalize beyond the test bed?" Focus: Task Distribution, Scenario Coverage, Missing Regions, Dataset Selection, Generalization Claims
When to Use
- Evaluating claims that extend beyond specific benchmarks
- Checking coverage of evaluation suite
- Assessing dataset diversity
- User invokes
/exp-lens-benchmark-representativenessor/make-experiment-diag benchmark
Critical Constraints
NEVER:
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
ALWAYS:
- Focus on GENERALIZATION GAP between benchmark coverage and claimed scope
- Show which regions of the target space are untested
- Document the relationship between benchmark selection and generalization claims
- Include a coverage matrix mapping scenarios to metrics
- BEFORE creating any diagram, LOAD the
/mermaidskill using the Skill tool - this is MANDATORY
Analysis Workflow
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Benchmark & Dataset Inventory
- Find all datasets, benchmarks, test suites used
- Look for:
benchmark,dataset,test_suite,eval,corpus,split,GLUE,ImageNet
Task & Scenario Coverage
- Find what scenarios, conditions, and domains are tested
- Look for:
task,scenario,domain,category,difficulty,subset
Metric Coverage
- Find all evaluation metrics used
- Look for:
metric,accuracy,f1,bleu,rouge,latency,cost,fairness
Claimed Generalization Scope
- Find claims about generality in docs, papers, READMEs
- Look for:
generalize,real-world,production,deploy,robust,transfer,domain
Distribution Characteristics
- Find data distribution analysis, class balance, domain stats
- Look for:
distribution,balance,skew,size,demographics,diversity
Step 2: Build the Coverage Matrix
Build the coverage matrix: rows = scenarios/domains tested, columns = metrics measured. Identify which cells are populated and which are gaps. Compare the coverage to the stated generalization claims.
Step 3: CRITICAL — Analyze Generalization Gap
For every generalization claim:
- Target population: What is the full population the claim extends to?
- Benchmark representation: What subset of that population is represented in the benchmark?
- Untested regions: What regions of the space are untested?
- Coverage ratio: Is the coverage sufficient to support the claim?
Distinguish clearly:
- Strong claims (e.g., "production-ready"): require broad, diverse coverage
- Scoped claims (e.g., "best on GLUE"): only require benchmark-specific coverage
- Implicit claims: claims made in framing but not stated explicitly
Step 4: Create the Diagram
Use flowchart with:
Direction: TB (claims flow from benchmarks up to generalization)
Subgraphs:
BENCHMARKS USEDSCENARIOS TESTEDMETRICS MEASUREDGENERALIZATION CLAIMSUNTESTED REGIONS
Node Styling:
stateNodeclass: benchmarks/datasetshandlerclass: tested scenariosoutputclass: measured metricscliclass: generalization claimsgapclass: untested regions/missing coveragedetectorclass: validation of generalization
Step 5: Write Output
Write the diagram to: temp/exp-lens-benchmark-representativeness/exp_diag_benchmark_representativeness_{YYYY-MM-DD_HHMMSS}.md
Output Template
# Benchmark Representativeness Diagram: {System Name}
**Lens:** Benchmark Representativeness (Generalizability)
**Question:** Does this generalize beyond the test bed?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Coverage Matrix
| Scenario / Domain | {Metric A} | {Metric B} | {Metric C} | Coverage |
|-------------------|-----------|-----------|-----------|----------|
| {Scenario 1} | ✓ | ✓ | ✗ | Partial |
| {Scenario 2} | ✗ | ✗ | ✗ | None |
| {Scenario 3} | ✓ | ✓ | ✓ | Full |
## Benchmark Representativeness Diagram
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart TB
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Benchmarks ["BENCHMARKS USED"]
direction TB
B1["{Benchmark 1}<br/>━━━━━━━━━━<br/>{size}, {domain}"]
B2["{Benchmark 2}<br/>━━━━━━━━━━<br/>{size}, {domain}"]
end
subgraph Scenarios ["SCENARIOS TESTED"]
direction TB
S1["{Scenario 1}<br/>━━━━━━━━━━<br/>{conditions}"]
S2["{Scenario 2}<br/>━━━━━━━━━━<br/>{conditions}"]
end
subgraph Metrics ["METRICS MEASURED"]
direction TB
M1["{Metric A}<br/>━━━━━━━━━━<br/>{what it captures}"]
M2["{Metric B}<br/>━━━━━━━━━━<br/>{what it captures}"]
end
subgraph Claims ["GENERALIZATION CLAIMS"]
direction TB
C1["{Claim 1}<br/>━━━━━━━━━━<br/>{source of claim}"]
C2["{Claim 2}<br/>━━━━━━━━━━<br/>{source of claim}"]
end
subgraph Gaps ["UNTESTED REGIONS"]
direction TB
G1["{Gap 1}<br/>━━━━━━━━━━<br/>{why it matters}"]
G2["{Gap 2}<br/>━━━━━━━━━━<br/>{why it matters}"]
end
VALIDATE["{Generalization Validity Check}<br/>━━━━━━━━━━<br/>Coverage vs. Claim scope"]
B1 --> S1
B2 --> S2
S1 --> M1
S2 --> M2
M1 --> C1
M2 --> C2
C1 --> VALIDATE
C2 --> VALIDATE
G1 -.->|missing| VALIDATE
G2 -.->|missing| VALIDATE
%% CLASS ASSIGNMENTS %%
class B1,B2 stateNode;
class S1,S2 handler;
class M1,M2 output;
class C1,C2 cli;
class G1,G2 gap;
class VALIDATE detector;Color Legend:
| Color | Category | Description |
| Dark Teal | Benchmarks | Datasets and test suites used |
| Orange | Scenarios | Tested scenarios and conditions |
| Teal | Metrics | Measured evaluation metrics |
| Dark Blue | Claims | Generalization claims made |
| Yellow/Amber | Gaps | Untested regions of target space |
| Red | Validation | Generalization validity check |
Generalization Gap Analysis
| Claim | Evidence (Benchmarks) | Gap (Untested) | Risk |
| {Claim 1} | {What covers it} | {What is missing} | High/Med/Low |
| {Claim 2} | {What covers it} | {What is missing} | High/Med/Low |
Representativeness Assessment
| Dimension | Current Coverage | Required for Claim | Verdict |
| Domain diversity | {count} domains | {needed} | ✓/✗ |
| Task variety | {count} tasks | {needed} | ✓/✗ |
| Scale range | {min}–{max} | {needed} | ✓/✗ |
| Distribution shift | {tested?} | {needed} | ✓/✗ |
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-measurement-validity` - For metric quality analysis
- `/exp-lens-validity-threats` - For systematic threat inventoryDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

