exp-lens-measurement-validity
Skill1 starsChanged 8 months agoMITView on GitHub
Analyze measurement validity for experimental design — auditing metric-construct alignment, proxy validity, reliability, sensitivity, and consequential validity. Argumentative lens answering "Do measurements justify the interpretation?
In this repository59 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- arch-lens-c4-containerSkill
- arch-lens-concurrencySkill
- arch-lens-data-lineageSkill
- arch-lens-deploymentSkill
- arch-lens-developmentSkill
- arch-lens-error-resilienceSkill
- arch-lens-module-dependencySkill
- arch-lens-operationalSkill
- arch-lens-process-flowSkill
- arch-lens-repository-accessSkill
- arch-lens-scenariosSkill
- arch-lens-securitySkill
- arch-lens-state-lifecycleSkill
- audit-archSkill
- audit-bugsSkill
- audit-cohesionSkill
- audit-defense-standardsSkill
- audit-frictionSkill
- audit-implSkill
- audit-testsSkill
- design-guardsSkill
- dry-walkthroughSkill
- elaborate-phaseSkill
- exp-lens-benchmark-representativenessSkill
- exp-lens-causal-assumptionsSkill
- exp-lens-comparator-constructionSkill
- exp-lens-error-budgetSkill
- exp-lens-estimand-claritySkill
- exp-lens-exploratory-confirmatorySkill
- exp-lens-fair-comparisonSkill
- exp-lens-governance-riskSkill
- exp-lens-iterative-learningSkill
- exp-lens-pipeline-integritySkill
- exp-lens-randomization-blockingSkill
- exp-lens-reproducibility-artifactsSkill
- exp-lens-sensitivity-robustnessSkill
- exp-lens-severity-testingSkill
- exp-lens-unit-interferenceSkill
- exp-lens-validity-threatsSkill
- exp-lens-variance-stabilitySkill
- id-slopSkill
- implement-worktreeSkill
- investigateSkill
- make-arch-diagSkill
- make-experiment-diagSkill
- make-groupsSkill
- make-planSkill
- make-reqSkill
- make-scenariosSkill
- mermaidSkill
- plan-arcSkill
- rectifySkill
- review-approachSkill
- update-architectureSkill
- update-reqsSkill
- update-specsSkill
- verify-diagSkill
- verify-proposed-diagSkill
The file
We split this skill into the rules it gives an agent
11 rules. About 0 are common, 9 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈9
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 30
Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …
- Distinctive: line 33
Treat every reported metric as a claim requiring a validity argument
- Very specific: line 37
BEFORE creating any optional diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY
- Distinctive: line 49
Look for: metric, score, accuracy, f1, precision, recall, bleu, rouge, loss, error_rate, latency
- Distinctive: line 53
Look for: better, worse, improves, indicates, measures, reflects, captures, proxy
- Distinctive: line 57
Look for: average, macro, micro, weighted, threshold, cutoff, aggregate
- Distinctive: line 61
Look for: also measured, alternative, we chose, instead of, limitation
- Distinctive: line 65
Look for: limitation, caveat, imperfect, proxy, approximate, does not capture
- Distinctive: line 89
If including the optional diagram:
- Very specific: line 108
Write the output to: temp/exp-lens-measurement-validity/exp_diag_measurement_validity_{YYYY-MM-DD_HHMMSS}.md
- Distinctive: line 223
Before creating any optional diagram, verify:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Measurement Validity Experimental Design Lens
Philosophical Mode: Psychometric Primary Question: "Do measurements justify the interpretation?" Focus: Metric-Construct Alignment, Proxy Validity, Reliability, Sensitivity, Consequential Validity
When to Use
- Metrics may not measure what they claim to; proxy metrics used instead of true objectives
- Evaluation scores treated as "truth" without validation
- Metric choice is contested or under-specified
- User invokes
/exp-lens-measurement-validityor/make-experiment-diag measurement
Critical Constraints
NEVER:
- Modify any source code or experimental artifacts
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
ALWAYS:
- Treat every reported metric as a claim requiring a validity argument
- Enumerate known failure modes: gaming, saturation, proxy collapse, and aggregation artifacts
- Assess reliability (stability under reruns) and sensitivity (ability to distinguish meaningful differences) for each metric
- Identify where metric-construct alignment is weak or unsupported by evidence
- BEFORE creating any optional diagram, LOAD the
/mermaidskill using the Skill tool - this is MANDATORY
Analysis Workflow
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Metric Definitions
- Find all metrics computed and reported
- Look for: metric, score, accuracy, f1, precision, recall, bleu, rouge, loss, error_rate, latency
Intended Interpretations
- Find what conclusions are drawn from each metric
- Look for: better, worse, improves, indicates, measures, reflects, captures, proxy
Metric Computation Details
- Find exactly how each metric is computed (aggregation, weighting, edge cases)
- Look for: average, macro, micro, weighted, threshold, cutoff, aggregate
Alternative Metrics Considered
- Find whether alternative metrics were evaluated and why they were rejected
- Look for: also measured, alternative, we chose, instead of, limitation
Construct-Metric Gap
- Find where the metric diverges from the construct it claims to measure
- Look for: limitation, caveat, imperfect, proxy, approximate, does not capture
Step 2: Build Validity Arguments
For each reported metric, construct a validity argument:
- What construct does this metric claim to measure?
- What evidence supports this claim?
- What are the known failure modes (gaming, saturation, proxy collapse)?
- Is the metric reliable (stable under reruns)?
- Is it sensitive (can it distinguish meaningful differences)?
Step 3: Analyze Metric-Construct Alignment
CRITICAL — for every metric-to-claim link:
- Is there a logical argument connecting the number to the concept?
- Could a system score high on this metric while being poor on the intended construct?
- What would gaming look like?
- Does the aggregation method (macro vs micro, mean vs median) preserve the intended construct?
- Are there known saturation regimes where the metric stops being informative?
Step 4: Optional Metric-Construct Mapping Diagram
This lens does NOT produce a primary mermaid diagram. The output is a structured validity argument. An optional simplified metric mapping diagram may be included if it clarifies the metric-construct relationship.
If including the optional diagram:
Direction: LR (constructs on left, metrics on right)
Minimal diagram: Construct nodes on the left, Metric nodes on the right, with edge labels indicating strength of alignment
Node Styling:
cliclass: Constructs and intended properties being claimedoutputclass: Measured metrics (what is actually computed)gapclass: Weak or missing alignments, proxy collapseshandlerclass: Proxy relationships and intermediate mappings
Connection Types:
- Solid arrows for strong, well-evidenced alignment
- Dashed arrows for proxy or contested alignment
- Edge labels naming the type of relationship or its weakness
Step 5: Write Output
Write the output to: temp/exp-lens-measurement-validity/exp_diag_measurement_validity_{YYYY-MM-DD_HHMMSS}.md
Output Template
# Measurement Validity Analysis: {Experiment Name}
**Lens:** Measurement Validity (Psychometric)
**Question:** Do measurements justify the interpretation?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Metric Inventory
| Metric | Construct Claimed | Computation | Reliability | Sensitivity |
|--------|-------------------|-------------|-------------|-------------|
| {metric name} | {what it claims to measure} | {aggregation/formula} | {stable / unstable / unknown} | {high / low / saturated} |
## Validity Arguments
### {Metric Name}
**Construct claimed:** {The property this metric is presented as measuring}
**Evidence for alignment:**
- {Supporting argument or citation}
**Evidence against alignment / known failure modes:**
- {Failure mode 1: e.g., gameable by surface pattern matching}
- {Failure mode 2: e.g., proxy collapses when distribution shifts}
**Reliability assessment:** {Stable under reruns? Sensitive to seed?}
**Sensitivity assessment:** {Can it distinguish meaningful differences in the relevant range?}
**Verdict:** {Strong / Partial / Weak / Unsupported}
---
## Proxy Collapse Risks
| Metric | Proxy For | Collapse Condition | Consequence |
|--------|-----------|--------------------|-------------|
| {metric} | {true construct} | {when proxy diverges from construct} | {what is falsely concluded} |
## Gaming Vulnerabilities
| Metric | Gaming Strategy | Detection Method |
|--------|----------------|-----------------|
| {metric} | {how to maximize score without improving construct} | {how to detect gaming} |
## Optional: Metric-Construct Mapping Diagram
{Include only if it clarifies alignment; omit if argument tables are sufficient}
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Constructs ["Intended Constructs"]
C1["Construct A<br/>━━━━━━━━━━<br/>The property claimed"]
C2["Construct B<br/>━━━━━━━━━━<br/>Another property"]
end
subgraph Metrics ["Measured Metrics"]
M1["Metric X<br/>━━━━━━━━━━<br/>Computation method"]
M2["Metric Y<br/>━━━━━━━━━━<br/>Computation method"]
end
subgraph Gaps ["Weak / Missing Alignments"]
G1["Proxy Collapse Risk<br/>━━━━━━━━━━<br/>Condition for divergence"]
end
%% ALIGNMENTS %%
C1 -->|"strong alignment"| M1
C2 -->|"proxy (weak)"| M2
M2 -.->|"diverges under"| G1
%% CLASS ASSIGNMENTS %%
class C1,C2 cli;
class M1,M2 output;
class G1 gap;Color Legend:
| Color | Category | Description |
| Dark Blue | Construct | Intended properties being claimed |
| Dark Teal | Metric | What is actually computed and reported |
| Yellow | Gap | Weak alignment, proxy collapse, or missing evidence |
| Orange | Proxy | Intermediate proxy relationships |
Summary Verdict
| Metric | Verdict | Primary Concern |
| {metric} | {Strong / Partial / Weak / Unsupported} | {One-line summary of the key validity concern} |
---
## Pre-Diagram Checklist
Before creating any optional diagram, verify:
- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating any optional diagram
- `/exp-lens-estimand-clarity` - For auditing the upstream claim the metric is meant to support
- `/exp-lens-benchmark-representativeness` - For auditing whether the evaluation set generalizesDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

