exp-lens-estimand-clarity
Skill1 starsChanged 8 months agoMITView on GitHub
Create Estimand Clarity experimental design analysis decomposing the implicit estimand from code vs. explicit claims from prose. Evidential lens answering "What exactly is the claim?
In this repository59 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- arch-lens-c4-containerSkill
- arch-lens-concurrencySkill
- arch-lens-data-lineageSkill
- arch-lens-deploymentSkill
- arch-lens-developmentSkill
- arch-lens-error-resilienceSkill
- arch-lens-module-dependencySkill
- arch-lens-operationalSkill
- arch-lens-process-flowSkill
- arch-lens-repository-accessSkill
- arch-lens-scenariosSkill
- arch-lens-securitySkill
- arch-lens-state-lifecycleSkill
- audit-archSkill
- audit-bugsSkill
- audit-cohesionSkill
- audit-defense-standardsSkill
- audit-frictionSkill
- audit-implSkill
- audit-testsSkill
- design-guardsSkill
- dry-walkthroughSkill
- elaborate-phaseSkill
- exp-lens-benchmark-representativenessSkill
- exp-lens-causal-assumptionsSkill
- exp-lens-comparator-constructionSkill
- exp-lens-error-budgetSkill
- exp-lens-exploratory-confirmatorySkill
- exp-lens-fair-comparisonSkill
- exp-lens-governance-riskSkill
- exp-lens-iterative-learningSkill
- exp-lens-measurement-validitySkill
- exp-lens-pipeline-integritySkill
- exp-lens-randomization-blockingSkill
- exp-lens-reproducibility-artifactsSkill
- exp-lens-sensitivity-robustnessSkill
- exp-lens-severity-testingSkill
- exp-lens-unit-interferenceSkill
- exp-lens-validity-threatsSkill
- exp-lens-variance-stabilitySkill
- id-slopSkill
- implement-worktreeSkill
- investigateSkill
- make-arch-diagSkill
- make-experiment-diagSkill
- make-groupsSkill
- make-planSkill
- make-reqSkill
- make-scenariosSkill
- mermaidSkill
- plan-arcSkill
- rectifySkill
- review-approachSkill
- update-architectureSkill
- update-reqsSkill
- update-specsSkill
- verify-diagSkill
- verify-proposed-diagSkill
The file
We split this skill into the rules it gives an agent
11 rules. About 0 are common, 9 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈9
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 30
Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …
- Distinctive: line 36
Document how complications (missing data, failures, exclusions) are handled
- Very specific: line 37
BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY
- Distinctive: line 49
Look for: hypothesis, claim, goal, objective, question, we show, we demonstrate, improves, outperforms
- Distinctive: line 53
Look for: treatment, intervention, method, approach, condition, configuration, ablation
- Distinctive: line 57
Look for: outcome, metric, measure, endpoint, target, response, dependent
- Distinctive: line 61
Look for: dataset, population, sample, domain, task, benchmark, scenario, setting
- Distinctive: line 65
Look for: missing, exclude, timeout, fail, drop, impute, censor, incomplete
- Distinctive: line 88
If a diagram adds value, create a simplified flowchart.
- Very specific: line 103
Write the analysis to: temp/exp-lens-estimand-clarity/exp_diag_estimand_clarity_{YYYY-MM-DD_HHMMSS}.md
- Distinctive: line 194
Before creating the diagram, verify:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Estimand Clarity Experimental Design Lens
Philosophical Mode: Evidential Primary Question: "What exactly is the claim?" Focus: Effect Definition, Target Population, Outcome Specification, Comparator, Aggregation Level, Complication Handling
When to Use
- Experiment has unclear or shifting hypotheses
- Multiple stakeholders interpret results differently
- Claims mix causal and predictive language
- User invokes
/exp-lens-estimand-clarityor/make-experiment-diag estimand
Critical Constraints
NEVER:
- Modify any source code or experiment files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
ALWAYS:
- Decompose every stated claim into formal contrast notation (Treatment A vs Treatment B on Outcome Y in Population Z)
- Flag every mismatch between prose claims and code implementation
- Identify the aggregation level (unit, group, time) explicitly
- Document how complications (missing data, failures, exclusions) are handled
- BEFORE creating any diagram, LOAD the
/mermaidskill using the Skill tool - this is MANDATORY
Analysis Workflow
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Stated Claims & Hypotheses
- Find hypothesis statements, research questions, README claims
- Look for: hypothesis, claim, goal, objective, question, we show, we demonstrate, improves, outperforms
Treatment Definition
- Find what intervention or manipulation is applied
- Look for: treatment, intervention, method, approach, condition, configuration, ablation
Outcome Definition
- Find what is measured as the result
- Look for: outcome, metric, measure, endpoint, target, response, dependent
Population & Scope
- Find what units, datasets, or contexts the claim covers
- Look for: dataset, population, sample, domain, task, benchmark, scenario, setting
Complication Handling
- Find how missing data, failures, timeouts, or exclusions are handled
- Look for: missing, exclude, timeout, fail, drop, impute, censor, incomplete
Step 2: Extract the Implicit Estimand
Answer each question from the code (not the docs):
- What is the treatment?
- What is the comparator?
- What is the outcome?
- What is the population?
- What is the time horizon?
- How are complications handled?
Step 3: Compare Claims to Implementation
Compare the explicit claims (from docs/papers) to the implicit estimand (from code). Flag mismatches between what the prose asserts and what the implementation actually measures.
CRITICAL — Analyze Claim Precision: For every stated claim:
- Can you write it as a formal contrast (Treatment A vs Treatment B on Outcome Y in Population Z)? If not, what is ambiguous?
- Does the code measure what the prose claims?
Step 4: Create the Optional Claim-Flow Diagram
If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.
Direction: TB (claim flows from intervention through measurement to conclusion)
Small diagram: 4-6 nodes showing Treatment → Mechanism → Outcome → Claim
Node Styling:
cliclass: treatment/intervention nodeshandlerclass: mechanism/pipeline nodesoutputclass: measured outcome nodesphaseclass: stated claim nodesgapclass: ambiguity or mismatch between claim and measurement
Step 5: Write Output
Write the analysis to: temp/exp-lens-estimand-clarity/exp_diag_estimand_clarity_{YYYY-MM-DD_HHMMSS}.md
Output Template
# Estimand Clarity Analysis: {Experiment Name}
**Lens:** Estimand Clarity (Evidential)
**Question:** What exactly is the claim?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Estimand Decomposition
| Component | Stated | Implemented | Match? |
|-----------|--------|-------------|--------|
| Treatment | {from prose} | {from code} | Yes / No / Partial |
| Comparator | {from prose} | {from code} | Yes / No / Partial |
| Outcome | {from prose} | {from code} | Yes / No / Partial |
| Population | {from prose} | {from code} | Yes / No / Partial |
| Time Horizon | {from prose} | {from code} | Yes / No / Partial |
| Complication Handling | {from prose} | {from code} | Yes / No / Partial |
## Claim Precision Assessment
| Claim | Formal Contrast | Ambiguities |
|-------|----------------|-------------|
| "{stated claim}" | Treatment A vs B on Y in Z | {list ambiguities} |
## Claim-Flow Diagram (Optional)
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart TB
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
TREATMENT["Treatment<br/>━━━━━━━━━━<br/>{intervention name}"]
MECHANISM["Mechanism<br/>━━━━━━━━━━<br/>{pipeline step}"]
OUTCOME["Measured Outcome<br/>━━━━━━━━━━<br/>{metric}"]
CLAIM["Stated Claim<br/>━━━━━━━━━━<br/>{claim text}"]
MISMATCH["Mismatch<br/>━━━━━━━━━━<br/>{ambiguity description}"]
TREATMENT --> MECHANISM
MECHANISM --> OUTCOME
OUTCOME --> CLAIM
OUTCOME -.->|"diverges"| MISMATCH
class TREATMENT cli;
class MECHANISM handler;
class OUTCOME output;
class CLAIM phase;
class MISMATCH gap;Color Legend:
| Color | Category | Description |
| Dark Blue | Treatment | Intervention applied |
| Orange | Mechanism | Pipeline processing |
| Dark Teal | Outcome | Measured result |
| Purple | Claim | Stated conclusion |
| Yellow | Mismatch | Ambiguity or claim-code divergence |
Ambiguity Register
| # | Ambiguity | Location | Severity | Resolution Needed |
| 1 | {description} | {file/section} | High / Medium / Low | {what to clarify} |
Recommendations
- {Specific action to resolve most critical ambiguity}
- {Rewrite suggestion for vague claim}
- {Code change to align implementation with stated estimand}
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-causal-assumptions` - For causal structure of the stated claim
- `/exp-lens-measurement-validity` - For whether the outcome metric is validDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

