agentleFS
TrecekTrecek/useful-claude-skills/.claude/skills/exp-lens-benchmark-representativeness/SKILL.md

exp-lens-benchmark-representativeness

Skill1 starsChanged 8 months agoMITView on GitHub

Create Benchmark Representativeness experimental design diagram showing coverage matrix, generalization gaps, and untested regions. Generalizability lens answering "Does this generalize beyond the test bed?

The file

We split this skill into the rules it gives an agent

8 rules. About 0 are common, 3 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …

    line 30
  2. Distinctive:

    Document the relationship between benchmark selection and generalization claims

    line 35
  3. Very specific:

    BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

    line 37
  4. Very specific:

    Look for: benchmark, dataset, test_suite, eval, corpus, split, GLUE, ImageNet

    line 49
  5. Very specific:

    Look for: task, scenario, domain, category, difficulty, subset

    line 53
  6. Very specific:

    Look for: generalize, real-world, production, deploy, robust, transfer, domain

    line 61
  7. Very specific:

    Write the diagram to: temp/exp-lens-benchmark-representativeness/exp_diag_benchmark_representativeness_{YYYY-MM-DD_HHMMSS}.md

    line 107
  8. Distinctive:

    Before creating the diagram, verify:

    line 228

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Benchmark Representativeness Experimental Design Lens

Philosophical Mode: Generalizability Primary Question: "Does this generalize beyond the test bed?" Focus: Task Distribution, Scenario Coverage, Missing Regions, Dataset Selection, Generalization Claims

When to Use

  • Evaluating claims that extend beyond specific benchmarks
  • Checking coverage of evaluation suite
  • Assessing dataset diversity
  • User invokes /exp-lens-benchmark-representativeness or /make-experiment-diag benchmark

Critical Constraints

NEVER:

  • Modify any source code files
  • Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves

ALWAYS:

  • Focus on GENERALIZATION GAP between benchmark coverage and claimed scope
  • Show which regions of the target space are untested
  • Document the relationship between benchmark selection and generalization claims
  • Include a coverage matrix mapping scenarios to metrics
  • BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

Analysis Workflow

Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

Benchmark & Dataset Inventory

  • Find all datasets, benchmarks, test suites used
  • Look for: benchmark, dataset, test_suite, eval, corpus, split, GLUE, ImageNet

Task & Scenario Coverage

  • Find what scenarios, conditions, and domains are tested
  • Look for: task, scenario, domain, category, difficulty, subset

Metric Coverage

  • Find all evaluation metrics used
  • Look for: metric, accuracy, f1, bleu, rouge, latency, cost, fairness

Claimed Generalization Scope

  • Find claims about generality in docs, papers, READMEs
  • Look for: generalize, real-world, production, deploy, robust, transfer, domain

Distribution Characteristics

  • Find data distribution analysis, class balance, domain stats
  • Look for: distribution, balance, skew, size, demographics, diversity

Step 2: Build the Coverage Matrix

Build the coverage matrix: rows = scenarios/domains tested, columns = metrics measured. Identify which cells are populated and which are gaps. Compare the coverage to the stated generalization claims.

Step 3: CRITICAL — Analyze Generalization Gap

For every generalization claim:

  • Target population: What is the full population the claim extends to?
  • Benchmark representation: What subset of that population is represented in the benchmark?
  • Untested regions: What regions of the space are untested?
  • Coverage ratio: Is the coverage sufficient to support the claim?

Distinguish clearly:

  • Strong claims (e.g., "production-ready"): require broad, diverse coverage
  • Scoped claims (e.g., "best on GLUE"): only require benchmark-specific coverage
  • Implicit claims: claims made in framing but not stated explicitly

Step 4: Create the Diagram

Use flowchart with:

Direction: TB (claims flow from benchmarks up to generalization)

Subgraphs:

  • BENCHMARKS USED
  • SCENARIOS TESTED
  • METRICS MEASURED
  • GENERALIZATION CLAIMS
  • UNTESTED REGIONS

Node Styling:

  • stateNode class: benchmarks/datasets
  • handler class: tested scenarios
  • output class: measured metrics
  • cli class: generalization claims
  • gap class: untested regions/missing coverage
  • detector class: validation of generalization

Step 5: Write Output

Write the diagram to: temp/exp-lens-benchmark-representativeness/exp_diag_benchmark_representativeness_{YYYY-MM-DD_HHMMSS}.md


Output Template

# Benchmark Representativeness Diagram: {System Name}

**Lens:** Benchmark Representativeness (Generalizability)
**Question:** Does this generalize beyond the test bed?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Coverage Matrix

| Scenario / Domain | {Metric A} | {Metric B} | {Metric C} | Coverage |
|-------------------|-----------|-----------|-----------|----------|
| {Scenario 1}      | ✓         | ✓         | ✗         | Partial  |
| {Scenario 2}      | ✗         | ✗         | ✗         | None     |
| {Scenario 3}      | ✓         | ✓         | ✓         | Full     |

## Benchmark Representativeness Diagram

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart TB
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    subgraph Benchmarks ["BENCHMARKS USED"]
        direction TB
        B1["{Benchmark 1}<br/>━━━━━━━━━━<br/>{size}, {domain}"]
        B2["{Benchmark 2}<br/>━━━━━━━━━━<br/>{size}, {domain}"]
    end

    subgraph Scenarios ["SCENARIOS TESTED"]
        direction TB
        S1["{Scenario 1}<br/>━━━━━━━━━━<br/>{conditions}"]
        S2["{Scenario 2}<br/>━━━━━━━━━━<br/>{conditions}"]
    end

    subgraph Metrics ["METRICS MEASURED"]
        direction TB
        M1["{Metric A}<br/>━━━━━━━━━━<br/>{what it captures}"]
        M2["{Metric B}<br/>━━━━━━━━━━<br/>{what it captures}"]
    end

    subgraph Claims ["GENERALIZATION CLAIMS"]
        direction TB
        C1["{Claim 1}<br/>━━━━━━━━━━<br/>{source of claim}"]
        C2["{Claim 2}<br/>━━━━━━━━━━<br/>{source of claim}"]
    end

    subgraph Gaps ["UNTESTED REGIONS"]
        direction TB
        G1["{Gap 1}<br/>━━━━━━━━━━<br/>{why it matters}"]
        G2["{Gap 2}<br/>━━━━━━━━━━<br/>{why it matters}"]
    end

    VALIDATE["{Generalization Validity Check}<br/>━━━━━━━━━━<br/>Coverage vs. Claim scope"]

    B1 --> S1
    B2 --> S2
    S1 --> M1
    S2 --> M2
    M1 --> C1
    M2 --> C2
    C1 --> VALIDATE
    C2 --> VALIDATE
    G1 -.->|missing| VALIDATE
    G2 -.->|missing| VALIDATE

    %% CLASS ASSIGNMENTS %%
    class B1,B2 stateNode;
    class S1,S2 handler;
    class M1,M2 output;
    class C1,C2 cli;
    class G1,G2 gap;
    class VALIDATE detector;

Color Legend:

ColorCategoryDescription
Dark TealBenchmarksDatasets and test suites used
OrangeScenariosTested scenarios and conditions
TealMetricsMeasured evaluation metrics
Dark BlueClaimsGeneralization claims made
Yellow/AmberGapsUntested regions of target space
RedValidationGeneralization validity check

Generalization Gap Analysis

ClaimEvidence (Benchmarks)Gap (Untested)Risk
{Claim 1}{What covers it}{What is missing}High/Med/Low
{Claim 2}{What covers it}{What is missing}High/Med/Low

Representativeness Assessment

DimensionCurrent CoverageRequired for ClaimVerdict
Domain diversity{count} domains{needed}✓/✗
Task variety{count} tasks{needed}✓/✗
Scale range{min}–{max}{needed}✓/✗
Distribution shift{tested?}{needed}✓/✗

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-measurement-validity` - For metric quality analysis
- `/exp-lens-validity-threats` - For systematic threat inventory

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.