agentleFS
TrecekTrecek/useful-claude-skills/.claude/skills/exp-lens-estimand-clarity/SKILL.md

exp-lens-estimand-clarity

Skill1 starsChanged 8 months agoMITView on GitHub

Create Estimand Clarity experimental design analysis decomposing the implicit estimand from code vs. explicit claims from prose. Evidential lens answering "What exactly is the claim?

The file

We split this skill into the rules it gives an agent

11 rules. About 0 are common, 9 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈9

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …

    line 30
  2. Distinctive:

    Document how complications (missing data, failures, exclusions) are handled

    line 36
  3. Very specific:

    BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

    line 37
  4. Distinctive:

    Look for: hypothesis, claim, goal, objective, question, we show, we demonstrate, improves, outperforms

    line 49
  5. Distinctive:

    Look for: treatment, intervention, method, approach, condition, configuration, ablation

    line 53
  6. Distinctive:

    Look for: outcome, metric, measure, endpoint, target, response, dependent

    line 57
  7. Distinctive:

    Look for: dataset, population, sample, domain, task, benchmark, scenario, setting

    line 61
  8. Distinctive:

    Look for: missing, exclude, timeout, fail, drop, impute, censor, incomplete

    line 65
  9. Distinctive:

    If a diagram adds value, create a simplified flowchart.

    line 88
  10. Very specific:

    Write the analysis to: temp/exp-lens-estimand-clarity/exp_diag_estimand_clarity_{YYYY-MM-DD_HHMMSS}.md

    line 103
  11. Distinctive:

    Before creating the diagram, verify:

    line 194

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Estimand Clarity Experimental Design Lens

Philosophical Mode: Evidential Primary Question: "What exactly is the claim?" Focus: Effect Definition, Target Population, Outcome Specification, Comparator, Aggregation Level, Complication Handling

When to Use

  • Experiment has unclear or shifting hypotheses
  • Multiple stakeholders interpret results differently
  • Claims mix causal and predictive language
  • User invokes /exp-lens-estimand-clarity or /make-experiment-diag estimand

Critical Constraints

NEVER:

  • Modify any source code or experiment files
  • Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves

ALWAYS:

  • Decompose every stated claim into formal contrast notation (Treatment A vs Treatment B on Outcome Y in Population Z)
  • Flag every mismatch between prose claims and code implementation
  • Identify the aggregation level (unit, group, time) explicitly
  • Document how complications (missing data, failures, exclusions) are handled
  • BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

Analysis Workflow

Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

Stated Claims & Hypotheses

  • Find hypothesis statements, research questions, README claims
  • Look for: hypothesis, claim, goal, objective, question, we show, we demonstrate, improves, outperforms

Treatment Definition

  • Find what intervention or manipulation is applied
  • Look for: treatment, intervention, method, approach, condition, configuration, ablation

Outcome Definition

  • Find what is measured as the result
  • Look for: outcome, metric, measure, endpoint, target, response, dependent

Population & Scope

  • Find what units, datasets, or contexts the claim covers
  • Look for: dataset, population, sample, domain, task, benchmark, scenario, setting

Complication Handling

  • Find how missing data, failures, timeouts, or exclusions are handled
  • Look for: missing, exclude, timeout, fail, drop, impute, censor, incomplete

Step 2: Extract the Implicit Estimand

Answer each question from the code (not the docs):

  1. What is the treatment?
  2. What is the comparator?
  3. What is the outcome?
  4. What is the population?
  5. What is the time horizon?
  6. How are complications handled?

Step 3: Compare Claims to Implementation

Compare the explicit claims (from docs/papers) to the implicit estimand (from code). Flag mismatches between what the prose asserts and what the implementation actually measures.

CRITICAL — Analyze Claim Precision: For every stated claim:

  • Can you write it as a formal contrast (Treatment A vs Treatment B on Outcome Y in Population Z)? If not, what is ambiguous?
  • Does the code measure what the prose claims?

Step 4: Create the Optional Claim-Flow Diagram

If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.

Direction: TB (claim flows from intervention through measurement to conclusion)

Small diagram: 4-6 nodes showing Treatment → Mechanism → Outcome → Claim

Node Styling:

  • cli class: treatment/intervention nodes
  • handler class: mechanism/pipeline nodes
  • output class: measured outcome nodes
  • phase class: stated claim nodes
  • gap class: ambiguity or mismatch between claim and measurement

Step 5: Write Output

Write the analysis to: temp/exp-lens-estimand-clarity/exp_diag_estimand_clarity_{YYYY-MM-DD_HHMMSS}.md


Output Template

# Estimand Clarity Analysis: {Experiment Name}

**Lens:** Estimand Clarity (Evidential)
**Question:** What exactly is the claim?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Estimand Decomposition

| Component | Stated | Implemented | Match? |
|-----------|--------|-------------|--------|
| Treatment | {from prose} | {from code} | Yes / No / Partial |
| Comparator | {from prose} | {from code} | Yes / No / Partial |
| Outcome | {from prose} | {from code} | Yes / No / Partial |
| Population | {from prose} | {from code} | Yes / No / Partial |
| Time Horizon | {from prose} | {from code} | Yes / No / Partial |
| Complication Handling | {from prose} | {from code} | Yes / No / Partial |

## Claim Precision Assessment

| Claim | Formal Contrast | Ambiguities |
|-------|----------------|-------------|
| "{stated claim}" | Treatment A vs B on Y in Z | {list ambiguities} |

## Claim-Flow Diagram (Optional)

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart TB
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    TREATMENT["Treatment<br/>━━━━━━━━━━<br/>{intervention name}"]
    MECHANISM["Mechanism<br/>━━━━━━━━━━<br/>{pipeline step}"]
    OUTCOME["Measured Outcome<br/>━━━━━━━━━━<br/>{metric}"]
    CLAIM["Stated Claim<br/>━━━━━━━━━━<br/>{claim text}"]
    MISMATCH["Mismatch<br/>━━━━━━━━━━<br/>{ambiguity description}"]

    TREATMENT --> MECHANISM
    MECHANISM --> OUTCOME
    OUTCOME --> CLAIM
    OUTCOME -.->|"diverges"| MISMATCH

    class TREATMENT cli;
    class MECHANISM handler;
    class OUTCOME output;
    class CLAIM phase;
    class MISMATCH gap;

Color Legend:

ColorCategoryDescription
Dark BlueTreatmentIntervention applied
OrangeMechanismPipeline processing
Dark TealOutcomeMeasured result
PurpleClaimStated conclusion
YellowMismatchAmbiguity or claim-code divergence

Ambiguity Register

#AmbiguityLocationSeverityResolution Needed
1{description}{file/section}High / Medium / Low{what to clarify}

Recommendations

  1. {Specific action to resolve most critical ambiguity}
  2. {Rewrite suggestion for vague claim}
  3. {Code change to align implementation with stated estimand}

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-causal-assumptions` - For causal structure of the stated claim
- `/exp-lens-measurement-validity` - For whether the outcome metric is valid

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.