exp-lens-validity-threats
Skill1 starsChanged 8 months agoMITView on GitHub
Create a validity threat matrix identifying alternative explanations and design mitigations. Adversarial lens answering "What alternative explanations survive?
In this repository59 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- arch-lens-c4-containerSkill
- arch-lens-concurrencySkill
- arch-lens-data-lineageSkill
- arch-lens-deploymentSkill
- arch-lens-developmentSkill
- arch-lens-error-resilienceSkill
- arch-lens-module-dependencySkill
- arch-lens-operationalSkill
- arch-lens-process-flowSkill
- arch-lens-repository-accessSkill
- arch-lens-scenariosSkill
- arch-lens-securitySkill
- arch-lens-state-lifecycleSkill
- audit-archSkill
- audit-bugsSkill
- audit-cohesionSkill
- audit-defense-standardsSkill
- audit-frictionSkill
- audit-implSkill
- audit-testsSkill
- design-guardsSkill
- dry-walkthroughSkill
- elaborate-phaseSkill
- exp-lens-benchmark-representativenessSkill
- exp-lens-causal-assumptionsSkill
- exp-lens-comparator-constructionSkill
- exp-lens-error-budgetSkill
- exp-lens-estimand-claritySkill
- exp-lens-exploratory-confirmatorySkill
- exp-lens-fair-comparisonSkill
- exp-lens-governance-riskSkill
- exp-lens-iterative-learningSkill
- exp-lens-measurement-validitySkill
- exp-lens-pipeline-integritySkill
- exp-lens-randomization-blockingSkill
- exp-lens-reproducibility-artifactsSkill
- exp-lens-sensitivity-robustnessSkill
- exp-lens-severity-testingSkill
- exp-lens-unit-interferenceSkill
- exp-lens-variance-stabilitySkill
- id-slopSkill
- implement-worktreeSkill
- investigateSkill
- make-arch-diagSkill
- make-experiment-diagSkill
- make-groupsSkill
- make-planSkill
- make-reqSkill
- make-scenariosSkill
- mermaidSkill
- plan-arcSkill
- rectifySkill
- review-approachSkill
- update-architectureSkill
- update-reqsSkill
- update-specsSkill
- verify-diagSkill
- verify-proposed-diagSkill
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 8 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈8
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 30
Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …
- Very specific: line 37
BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY
- Distinctive: line 49
Look for: version, update, release, change, drift, before, after, time, date
- Distinctive: line 53
Look for: log, metric, instrument, measurement, format, schema, API, breaking
- Distinctive: line 57
Look for: filter, exclude, select, subsample, eligible, criteria, dropout, attrition
- Distinctive: line 61
Look for: also, simultaneously, together, combined, alongside, additionally, migration
- Distinctive: line 65
Look for: access, share, copy, learn, adopt, contaminate, crossover, spillover
- Distinctive: line 85
Use the mermaid skill conventions to create a threat-flow diagram with:
- Very specific: line 103
Write the diagram to: temp/exp-lens-validity-threats/exp_diag_validity_threats_{YYYY-MM-DD_HHMMSS}.md
- Distinctive: line 201
Before creating the diagram, verify:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Validity Threats Experimental Design Lens
Philosophical Mode: Adversarial Primary Question: "What alternative explanations survive?" Focus: History Effects, Instrumentation Changes, Selection Effects, Co-interventions, Treatment Diffusion
When to Use
- Quasi-experimental designs without full randomization
- System experiments with environmental changes
- Longitudinal comparisons
- User invokes
/exp-lens-validity-threatsor/make-experiment-diag validity
Critical Constraints
NEVER:
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
ALWAYS:
- Apply Campbell & Stanley's full threat taxonomy — do not skip threats just because they seem unlikely
- For every observed difference, enumerate at least 3 alternative explanations
- Assess mitigation strength honestly — "partially mitigated" is better than false confidence
- Distinguish threats that are ruled out by design from those that remain plausible
- BEFORE creating any diagram, LOAD the
/mermaidskill using the Skill tool - this is MANDATORY
Analysis Workflow
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Temporal Changes (History)
- Find whether the environment changed between conditions
- Look for: version, update, release, change, drift, before, after, time, date
Instrumentation Changes
- Find whether measurement tools or methods changed
- Look for: log, metric, instrument, measurement, format, schema, API, breaking
Selection & Filtering
- Find whether groups are compared after differential filtering
- Look for: filter, exclude, select, subsample, eligible, criteria, dropout, attrition
Co-interventions
- Find whether multiple changes occurred simultaneously
- Look for: also, simultaneously, together, combined, alongside, additionally, migration
Treatment Diffusion
- Find whether control units could access treatment
- Look for: access, share, copy, learn, adopt, contaminate, crossover, spillover
Step 2: Apply Threat Taxonomy
Apply Campbell & Stanley's threat taxonomy to the specific experiment. For each threat category:
- Is it plausible in this context?
- What design feature mitigates it?
- How strong is the mitigation?
Build the threat matrix.
Step 3: Analyze Alternative Explanations
CRITICAL — Analyze Alternative Explanations: For every observed difference between conditions:
- List at least 3 alternative explanations besides the intended causal claim
- For each: What evidence rules it out? What evidence remains consistent with it?
Step 4: Create the Diagram
Use the mermaid skill conventions to create a threat-flow diagram with:
Direction: TB (threats flow from sources through mitigations to residual risk)
Subgraphs:
- THREAT SOURCES
- DESIGN MITIGATIONS
- RESIDUAL THREATS
Node Styling:
gapclass: Active unmitigated threatsdetectorclass: Design mitigationsstateNodeclass: Threat sourceshandlerclass: Partially mitigated threatsoutputclass: Successfully mitigated threats
Step 5: Write Output
Write the diagram to: temp/exp-lens-validity-threats/exp_diag_validity_threats_{YYYY-MM-DD_HHMMSS}.md
Output Template
# Validity Threat Analysis: {Experiment Name}
**Lens:** Validity Threats (Adversarial)
**Question:** What alternative explanations survive?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Threat Matrix
| Threat | Category | Plausibility | Mitigation | Residual Risk |
|--------|----------|-------------|------------|---------------|
| {threat} | {Campbell-Stanley category} | {Low/Medium/High} | {design feature} | {Low/Medium/High} |
## Alternative Explanations
| Observed Difference | Alternative Explanation | Ruling-Out Evidence | Consistent Evidence |
|--------------------|------------------------|---------------------|---------------------|
| {difference} | {explanation} | {evidence} | {evidence} |
## Threat-Flow Diagram
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
graph TB
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Sources ["THREAT SOURCES"]
direction TB
HIST["History Effect<br/>━━━━━━━━━━<br/>Environmental change<br/>between conditions"]
INST["Instrumentation<br/>━━━━━━━━━━<br/>Measurement tool<br/>change"]
SEL["Selection Effect<br/>━━━━━━━━━━<br/>Differential filtering<br/>of units"]
end
subgraph Mitigations ["DESIGN MITIGATIONS"]
direction TB
RAND["Randomization<br/>━━━━━━━━━━<br/>Controls selection<br/>and history"]
CTRL["Control Group<br/>━━━━━━━━━━<br/>Absorbs environmental<br/>changes"]
end
subgraph Residual ["RESIDUAL THREATS"]
direction TB
DIFF["Treatment Diffusion<br/>━━━━━━━━━━<br/>Control access<br/>to treatment"]
COINT["Co-interventions<br/>━━━━━━━━━━<br/>Simultaneous<br/>changes"]
end
HIST --> CTRL
INST --> CTRL
SEL --> RAND
RAND --> CTRL
CTRL --> DIFF
CTRL --> COINT
%% CLASS ASSIGNMENTS %%
class HIST,INST,SEL stateNode;
class RAND,CTRL detector;
class DIFF,COINT gap;Campbell-Stanley Checklist
| Threat | Applicable? | Assessment |
| History | {Yes/No/Partial} | {assessment} |
| Maturation | {Yes/No/Partial} | {assessment} |
| Testing | {Yes/No/Partial} | {assessment} |
| Instrumentation | {Yes/No/Partial} | {assessment} |
| Statistical Regression | {Yes/No/Partial} | {assessment} |
| Selection | {Yes/No/Partial} | {assessment} |
| Experimental Mortality | {Yes/No/Partial} | {assessment} |
| Selection-Maturation Interaction | {Yes/No/Partial} | {assessment} |
| Diffusion of Treatments | {Yes/No/Partial} | {assessment} |
| Co-interventions | {Yes/No/Partial} | {assessment} |
Key Findings
- {Description of most critical surviving alternative explanations}
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-causal-assumptions` - For DAG-based causal structure analysis
- `/exp-lens-severity-testing` - For statistical severity and power analysisDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

