agentleFS
zirui-songzirui-song/claude-skills/skills/robustness-battery/SKILL.md

robustness-battery

Skill17 starsChanged 3 months agoMITView on GitHub

Use when preparing a paper for submission or responding to referees and you need a full battery of robustness checks for a headline result, or when the user asks "what happens if…" about a single spec change. Enumerates reviewer-anticipated variants across estimator, sample, controls, weights, and clustering, runs them in Stata, and produces a comparison table + sign/significance stability summary; a what-if mode runs one change in a scratch copy and shows the result in chat.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 3 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    When the user asks "what happens if we add bank FE / cluster by X / drop 2020 / …":

    line 25
  2. Very specific:

    Run it (batch Stata via a space-free wrapper.

    line 29
  3. Very specific:

    When the user asks for a named set of columns "in one pass," run the main spec plus those columns together, in a single pass, and return …

    line 49
  4. Distinctive:

    Never silently drop a failing spec: If a column errors (r(504), collinearity, time-invariant regressor, empty sample), keep the column in …

    line 53
  5. Very specific:

    Run independent variants in parallel (e.g.

    line 67
  6. Distinctive:

    Stop only when every tracked coefficient has ≥ 5 variants AND every variant either:

    line 79
  7. Very specific:

    If a variant runs > 30 minutes, run it in the background and wait with a monitor loop rather than bare sleep polling.

    line 86

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Robustness Battery

Standardized referee-anticipating robustness sweep for any headline coefficient. All estimation runs in Stata (reghdfe/ivreghdfe/ppmlhdfe/robreg, from SSC) — never Python.

Before anything: read the project's rules

The project's CLAUDE.md and memory override the defaults below. Check them for:

  • the FE ladder (e.g. cumulative: (1) industry×year, (2) +rating, (3) +borrower, (4) +bank), and any FE the project rules out (e.g. no firm FE);
  • the full control set — use it in every variant unless the variant is about controls;
  • the default clustering (e.g. two-way firm + officer instead of firm);
  • heterogeneity: pooled interactions, not sample splits, unless the user asks for splits.

Then state the exact main spec back to the user — LHS, RHS, FE set, sample filter, clustering — before running anything.

What-if mode (one change at a time)

When the user asks "what happens if we add bank FE / cluster by X / drop 2020 / …":

  1. Copy the main do-file to a scratch variant. Never edit the main do-file in this mode.
  2. Apply only the one change the user named. Don't add or drop anything else.
  3. Run it (batch Stata via a space-free wrapper; see stata-preflight) and check the log.
  4. Show the result in chat: a small table with baseline vs variant — coef, SE, stars, N, clusters — plus the exact estimation command. One or two sentences on what moved.
  5. Adopt the change into the main spec only when the user says so.

Full battery: variants to run (5 families × ≥1 each)

  1. Estimator — OLS (reghdfe), Huber/M and MM (robreg), Poisson (ppmlhdfe, if the outcome is non-negative), stacked DiD (if event-time).
  2. Sample — drop top/bottom 1% of outcome, drop largest 1% by size, balanced panel only, alt time window (e.g., excluding COVID years), drop singleton FE groups.
  3. Controls / FE — each rung of the project's FE ladder; the full control set throughout.
  4. Weights — unweighted, WLS by inverse propensity, WLS by firm size.
  5. Clustering — the project default, plus the obvious alternatives (one-way vs two-way, wild-cluster bootstrap via boottest when clusters are few).

Aim for ≥ 5 variants per headline coefficient.

Fast-grid mode (when the user names the columns)

When the user asks for a named set of columns "in one pass," run the main spec plus those columns together, in a single pass, and return one comparison table with coef, SE, and N per column. A paper's standing grid lives in that project's memory, not here.

  • Never silently drop a failing spec. If a column errors (r(504), collinearity, time-invariant regressor, empty sample), keep the column in the table and write the failure reason in its cell. Report the Stata error code verbatim.
  • One pass means one batch: don't hand back column 1 and wait for a follow-up asking for column 2.

Execution

For each variant, write a small Stata do-file that:

  • Loads the same data the main spec uses.
  • Applies only the variant's modification.
  • Estimates the model.
  • Appends one row to a results CSV with columns: family, variant, coef, se, t, p, n, n_clusters, pass_sign, pass_sig.

Run independent variants in parallel (e.g. xargs -P 4 over batch Stata wrappers). Python may read the CSV to draw the forest plot; it never estimates anything.

Output

  1. The full raw results CSV, in the project's results folder.
  2. An appendix .tex table grouped by family, following the project's table style.
  3. A coefficient stability forest plot, headline overlaid as a vertical line.
  4. Short summary (≤ 8 lines): how many variants pass sign, how many pass significance, and any flagged spec where the headline breaks (name the variant).

Stopping rule

Stop only when every tracked coefficient has ≥ 5 variants AND every variant either:

  • Passes both sign and significance, OR
  • Has been explicitly noted in the summary with a sentence on why it differs.

Notes

  • Link the full grid by path rather than pasting it; show the comparison table in chat when the user asks.
  • If a variant runs > 30 minutes, run it in the background and wait with a monitor loop rather than bare sleep polling.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.