choosing-causal-designs
Skill1 starsChanged 44 days agoMITView on GitHub
Selects a causal design when assignment already happened and randomisation was never possible: interrupted time series for one unit and one switch date, difference-in-differences for treated and untreated units seen before and after, synthetic control for one treated unit and many donors, matching on observables, instrumental variables, and the exit that says there is no valid comparison group. Use when measuring the impact of something already rolled out, a launch, a migration, a pricing change, or a campaign that reached everyone at once.
In this repository9 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- building-modelsSkill
- defining-metricsSkill
- designing-experimentsSkill
- forming-hypothesesSkill
- reading-experimentsSkill
- routing-questionsSkill
- writing-reportsSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 15
If the change has not fully shipped, a holdout or staggered rollout is still possible and strictly better.
- Distinctive: line 85
The calibration habit: when a causal-design estimate later gets tested properly (a holdout on the next iteration), compare.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Choosing causal designs
Assignment already happened; the design's job is to reconstruct the comparison that randomisation would have provided. Pick by how assignment happened, not by which method is fashionable, and hand back an effect size plus the assumptions it rests on, stated in the report, because here the assumptions are doing the work the coin flip would have done.
First, one check worth thirty seconds: is randomisation really unavailable? If the change has not fully shipped, a holdout or staggered rollout is still possible and strictly better. This skill is for the past tense.
The selection table
Work down; take the first row whose conditions hold.
| Assignment looked like | Design | Needs |
| One unit (or everyone), one sharp product switch on a known date, no untreated twin | ITS – reference/its.md | A long, stable pre-period; no simultaneous shock; a discrete switch, not a diffuse push |
| Some units got it, some did not, both observed before and after | DiD – reference/did.md | Parallel pre-trends |
| One treated unit, many untreated candidates to blend | Synthetic control – reference/synthetic-control.md | A donor pool the treatment did not touch |
| Individuals selected themselves in, on things you can observe | Matching – reference/matching-iv.md | Selection on observables (a strong claim; say it) |
| Something nudged uptake without touching the outcome directly | IV – reference/matching-iv.md | A real instrument (rare; most candidates fail) |
| None of the above | No comparison group – reference/no-comparison-group.md | Honesty |
The last row is a first-class exit, not a failure. Saying "there is no defensible number here" out loud beats shipping a number that dissolves under the first follow-up question, and the reference file says what to offer instead.
One row needs guarding against a tempting misread: a campaign or marketing push that ran everywhere at once is not an ITS candidate, even though "everyone, one date" appears to match. ITS needs a discrete switch in the product itself; a media flight is a diffuse, ramping shock whose effect shape is unknown, it usually arrives with seasonality and other pushes, and the pre-trend projection cannot separate any of that. That case takes the last row: bounds at most, a referral to whoever owns mix modeling or panel data, and a holdout designed into the next flight.
The second row has its own misread: two units, one treated, is still DiD. An Android switch with iOS untouched has a comparison group, and the twin is the counterfactual, not a falsification check bolted onto an ITS. With two clusters the standard errors are weak and the report says so in words; that is a reason to state the inference honestly, never a reason to drop to a design that throws away the only comparison there is.
What every design here must state
- The counterfactual, in one sentence. What would have happened without the change, and which data stands in for it. If the sentence cannot be written, the design is not chosen yet.
- The identifying assumption, in the requester's language. "Madrid would have moved like Barcelona" (DiD), "the trend would have continued" (ITS), "adopters and matched non-adopters differ only on what we matched" (matching). This is the sentence the report hangs on.
- The falsification checks run and passed. Pre-trends for DiD, placebo dates for ITS, placebo units for synthetic control, balance tables for matching. A design whose checks were not run is an assertion, not an estimate.
- What would change the answer. The known threat the design cannot exclude: the simultaneous marketing push, the seasonality no control captures, the unobservable that matching cannot see.
Discipline shared across all five
- Pick the design before computing the estimate, from the assignment story alone. Running three designs and reporting the one with the nicest number is the observational version of peeking.
- Effects here deserve wider error bars than their standard errors say. Standard errors price sampling noise, not assumption risk. Say which of the two dominates.
- Modeling is the machinery, not the design. DiD is a regression; synthetic control is a weighted counterfactual model. The design decides what comparison is valid; the model just computes it. Adding controls to a regression is not, by itself, a design.
- File the result through
writing-reportswithdesignset honestly in the prior-store record (its,did,synthetic_control,matching,iv). Observational effects mixing into the experimental prior is exactly the kind of thing the field records to prevent. - The calibration habit: when a causal-design estimate later gets tested properly (a holdout on the next iteration), compare. A team that never checks its observational calls against experimental ones never learns which of its designs to trust.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

