robustness-battery
Skill17 starsChanged 3 months agoMITView on GitHub
Use when preparing a paper for submission or responding to referees and you need a full battery of robustness checks for a headline result, or when the user asks "what happens if…" about a single spec change. Enumerates reviewer-anticipated variants across estimator, sample, controls, weights, and clustering, runs them in Stata, and produces a comparison table + sign/significance stability summary; a what-if mode runs one change in a scratch copy and shows the result in chat.
In this repository14 other files it gives its agents.
- codex-bridgeSkill
- codex-review-planSkill
- overleafSkill
- plan-debateSkill
- power-firstSkill
- referee-panelSkill
- refresh-pipelineSkill
- review-codex-planSkill
- spec-curve-sweepSkill
- stata-preflightSkill
- sync-overleafSkill
- tableSkill
- verify-claimsSkill
- verify-numbers-pipelineSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 3 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 25
When the user asks "what happens if we add bank FE / cluster by X / drop 2020 / …":
- Very specific: line 29
Run it (batch Stata via a space-free wrapper.
- Very specific: line 49
When the user asks for a named set of columns "in one pass," run the main spec plus those columns together, in a single pass, and return …
- Distinctive: line 53
Never silently drop a failing spec: If a column errors (r(504), collinearity, time-invariant regressor, empty sample), keep the column in …
- Very specific: line 67
Run independent variants in parallel (e.g.
- Distinctive: line 79
Stop only when every tracked coefficient has ≥ 5 variants AND every variant either:
- Very specific: line 86
If a variant runs > 30 minutes, run it in the background and wait with a monitor loop rather than bare sleep polling.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Robustness Battery
Standardized referee-anticipating robustness sweep for any headline coefficient. All estimation
runs in Stata (reghdfe/ivreghdfe/ppmlhdfe/robreg, from SSC) — never Python.
Before anything: read the project's rules
The project's CLAUDE.md and memory override the defaults below. Check them for:
- the FE ladder (e.g. cumulative: (1) industry×year, (2) +rating, (3) +borrower, (4) +bank), and any FE the project rules out (e.g. no firm FE);
- the full control set — use it in every variant unless the variant is about controls;
- the default clustering (e.g. two-way firm + officer instead of firm);
- heterogeneity: pooled interactions, not sample splits, unless the user asks for splits.
Then state the exact main spec back to the user — LHS, RHS, FE set, sample filter, clustering — before running anything.
What-if mode (one change at a time)
When the user asks "what happens if we add bank FE / cluster by X / drop 2020 / …":
- Copy the main do-file to a scratch variant. Never edit the main do-file in this mode.
- Apply only the one change the user named. Don't add or drop anything else.
- Run it (batch Stata via a space-free wrapper; see
stata-preflight) and check the log. - Show the result in chat: a small table with baseline vs variant — coef, SE, stars, N, clusters — plus the exact estimation command. One or two sentences on what moved.
- Adopt the change into the main spec only when the user says so.
Full battery: variants to run (5 families × ≥1 each)
- Estimator — OLS (
reghdfe), Huber/M and MM (robreg), Poisson (ppmlhdfe, if the outcome is non-negative), stacked DiD (if event-time). - Sample — drop top/bottom 1% of outcome, drop largest 1% by size, balanced panel only, alt time window (e.g., excluding COVID years), drop singleton FE groups.
- Controls / FE — each rung of the project's FE ladder; the full control set throughout.
- Weights — unweighted, WLS by inverse propensity, WLS by firm size.
- Clustering — the project default, plus the obvious alternatives (one-way vs two-way,
wild-cluster bootstrap via
boottestwhen clusters are few).
Aim for ≥ 5 variants per headline coefficient.
Fast-grid mode (when the user names the columns)
When the user asks for a named set of columns "in one pass," run the main spec plus those
columns together, in a single pass, and return one comparison table with coef, SE,
and N per column. A paper's standing grid lives in that project's memory, not here.
- Never silently drop a failing spec. If a column errors (r(504), collinearity, time-invariant regressor, empty sample), keep the column in the table and write the failure reason in its cell. Report the Stata error code verbatim.
- One pass means one batch: don't hand back column 1 and wait for a follow-up asking for column 2.
Execution
For each variant, write a small Stata do-file that:
- Loads the same data the main spec uses.
- Applies only the variant's modification.
- Estimates the model.
- Appends one row to a results CSV with columns:
family, variant, coef, se, t, p, n, n_clusters, pass_sign, pass_sig.
Run independent variants in parallel (e.g. xargs -P 4 over batch Stata wrappers). Python
may read the CSV to draw the forest plot; it never estimates anything.
Output
- The full raw results CSV, in the project's results folder.
- An appendix
.textable grouped by family, following the project's table style. - A coefficient stability forest plot, headline overlaid as a vertical line.
- Short summary (≤ 8 lines): how many variants pass sign, how many pass significance, and any flagged spec where the headline breaks (name the variant).
Stopping rule
Stop only when every tracked coefficient has ≥ 5 variants AND every variant either:
- Passes both sign and significance, OR
- Has been explicitly noted in the summary with a sentence on why it differs.
Notes
- Link the full grid by path rather than pasting it; show the comparison table in chat when the user asks.
- If a variant runs > 30 minutes, run it in the background and wait with a monitor loop rather than bare
sleeppolling.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

