audit-reproducibility

---
name: audit-reproducibility
description: Enforce the replication-protocol.md rule by cross-checking numeric claims in a manuscript against the actual R / Stata / Python outputs. Report PASS/FAIL per claim against tolerance thresholds. Use before submission and before releasing a replication package.
argument-hint: "[manuscript path] [outputs-dir] (outputs-dir defaults to output/)"
allowed-tools: ["Read", "Grep", "Glob", "Write", "Bash", "Agent", "Task", "Monitor"]
effort: high
---

# Audit Reproducibility

Compare numeric claims in a manuscript (point estimates, standard errors, p-values, counts) against the actual outputs produced by the analysis pipeline. Report PASS / FAIL per claim against the tolerance thresholds defined in [`.claude/rules/replication-protocol.md`](../../rules/replication-protocol.md).

**Core principle:** If the paper says `ATT = -1.632 (0.584)` and the code produces `-1.628 (0.591)`, we verify — **numerically** — that the difference is within the documented tolerance. No more "looks close enough" eyeballing.

**Two directions, not one.** *Vertically*, each claim is checked against the output that produced it. *Horizontally*, it is checked against every **other** artifact that displays the same number — the supplement table, the slide deck, the poster. The vertical check passes contentedly while a deck quotes last month's value; only the horizontal one catches that. Declared displays live in the passport's `appears_in` list — see [`replication-protocol.md` → The horizontal check](../../rules/replication-protocol.md#the-horizontal-check).

## When to use

- **Before submission.** Catches the "I updated the analysis but forgot to update Table 2" bug.
- **Before presenting or teaching from the same numbers.** Catches the deck, the poster, or the supplement that was never regenerated after the last rerun.
- **Before releasing a replication package.** Verifies the code actually reproduces the paper.
- **After a major revision.** Ensures the paper still matches the latest code.
- **Quality-gate in `/commit`.** Pair with a pre-commit invocation on manuscript + analysis changes.

## Inputs

- `$0` — path to the manuscript (`.tex`, `.qmd`, `.md`, `.pdf`). Required.
- `$1` — path to the outputs directory. Defaults to `output/`, where every language's pipeline writes (R, Stata via [`/stata-replication`](../stata-replication/SKILL.md), Python). Recognised alternatives: `_targets/objects/` (R `targets` workflows), any directory the user-specified outputs live in. If `output/` does not exist but a pre-v2.6 `scripts/<lang>/_outputs/` does, use that and say so in the report.

## Workflow

### Phase 0: Pre-flight

1. Read [`replication-protocol.md`](../../rules/replication-protocol.md) for the tolerance thresholds currently in effect.
2. Verify the outputs directory exists and is non-empty. If empty or stale (older than the manuscript), prompt the user to re-run their pipeline (e.g., `Rscript scripts/R/00_run_all.R`) before auditing.
3. Ensure a `sessionInfo.txt` or equivalent environment capture exists in the outputs dir.

### Phase 1: Extract claims from the manuscript

Parse the manuscript for numeric claims. Patterns to match:

- **Point-estimate + SE**: `ATT = -1.632 (0.584)`, `$\beta = 0.342$ (0.091)`, `hat{\tau} = 1.28**` with starred significance
- **Table cells**: `& -1.632$^{***}$ & 0.584 &` in LaTeX table environments
- **Counts**: `our sample of 2,847 firms`, `$N = 2{,}847$`
- **Summary stats**: `mean = 0.423`, `SD = 0.087`
- **P-values**: `p < 0.01`, `$p = 0.003$`

Record each claim as a tuple:

```
{
  claim_id: "Table2_col3_ATT",
  location: "Table 2, Column 3, row 'Treatment'",
  kind: "point_estimate" | "standard_error" | "p_value" | "count" | "percentage",
  reported_value: -1.632,
  uncertainty: 0.584,              # only for point estimates
  significance_stars: 3,            # 0-3 or None
  raw_context: "the ATT estimate of -1.632 (0.584) indicates..."
}
```

Write the extracted claims to `quality_reports/reproducibility_claims_[manuscript-name].json` so the user can review the extraction before audit.

### Phase 2: Extract results from outputs

Scan `$1` for corresponding values. Priority order:

1. **`.rds` files** — `readRDS(path)$coef[["treatment"]]` style lookups. Can use `Rscript -e "saveRDS(summary(readRDS(...)), '/tmp/audit.rds')"` to extract.
2. **`.tex` tables** — parse LaTeX table cells directly; match on column headers + row labels.
3. **`.csv` summary files** — pandas/readr parse, key-value lookup.
4. **`.out` / `.log` files** (Stata, regress output) — regex extraction.
5. **`.json`** — direct key lookup.

Record each extracted result:

```
{
  source: "output/results.rds",
  lookup_key: "fit_main$coefficients['treated']",
  value: -1.628,
  uncertainty: 0.591,
  p_value: 0.005
}
```

### Phase 2b: Collect every declared display (passport mode)

For each passport claim, read its `appears_in` list and pull the value **as displayed** at each entry:

```
{
  claim_id: "C3",
  displays: [
    { path: "manuscript.tex", locator: "Table 1, Col 2", display_precision: 3, shown: 0.342 },
    { path: "Slides/Lecture04_Results.tex", locator: "frame 'Main result'", display_precision: 2, shown: 0.34 }
  ]
}
```

Rules for this pass:

- The primary `location:` is one of the displays, not a separate thing — if `appears_in` repeats it, that is one display, not two.
- **A declared display that cannot be resolved — missing file, or no value at the `locator` — is recorded as `shown: NOT_FOUND`.** It resolves to FAIL in Phase 4c. Never skip it: a locator that has quietly stopped matching is exactly where a stale number hides.
- A claim with **no** `appears_in` list is horizontally *unchecked*, not horizontally clean. Report it that way (see Phase 5) rather than silently passing it.

### Phase 3: Match claims to results

Use fuzzy heuristics when exact labels don't match:

- Name similarity (`"treatment effect"` ~ `"ATT"` ~ `"treated"`)
- Magnitude similarity (if two candidates have values within 10% of the reported, prefer the one with closer SE)
- Context hints from the claim's `raw_context` field (table number, row label, description)

For every claim, produce a match candidate with a confidence score. Claims below 0.7 confidence get flagged as "UNMATCHED — manual review needed" rather than silently passing.

### Phase 4: Tolerance check

For each matched claim, apply the thresholds from `replication-protocol.md`:

| Kind | Tolerance | Example |
|---|---|---|
| Integers (N, counts) | Exact | 2,847 must equal 2,847 |
| Point estimates | `abs(reported - computed)` < 0.01 | -1.632 vs -1.628 → diff = 0.004 → PASS |
| Standard errors | `abs(reported - computed)` < 0.05 | 0.584 vs 0.591 → diff = 0.007 → PASS |
| P-values | Same significance level | p<0.01 and p<0.01 → PASS; p<0.01 and p=0.03 → FAIL |
| Percentages | ±0.1pp | 42.3% vs 42.35% → PASS |

Respect any **tolerance overrides** the user has written into their `replication-protocol.md` fork (they may loosen for MC noise or tighten for administrative data).

### Phase 4b: Disposition — PASS / FAIL / EXPLAINED / UNMATCHED

A tolerance check resolves to one of four dispositions:

- **PASS** — within tolerance.
- **FAIL** — outside tolerance, with no defensible alternative recorded. **Blocks** (exit 1).
- **EXPLAINED** — outside tolerance, **but** the author has recorded a *concrete, named alternative specification* that accounts for the gap (see the downgrade rule). Surfaced in the report and carried into a response-to-referees; does **not** block.
- **UNMATCHED** — no computed counterpart found (Phase 3 confidence < 0.7). Never auto-downgradable.

**A mismatch is not automatically a failure.** In applied work the most common out-of-tolerance result is a *defensible alternative spec*, not a bug — `reghdfe` vs `feols` clustering df, a different bandwidth-selection rule, a different MC seed/reps, or display rounding. The skill's job is to *stage the disagreement* for a human auditor, not to pronounce the code right and the paper wrong. (The df-adjustment note in "Stata-specific notes" below is the canonical example of a named alternative.)

**The manuscript is not the oracle.** When the computed value disagrees with the manuscript, do not presume the code is correct and the paper stale — nor the reverse. A refactor may have broken a previously-correct table (the *on-disk output* is the buggy one), or the paper may carry an old number. The computed value is a **challenger**, not ground truth. Report a mismatch as "one of {paper, code} must change — isolate which," never "revert the code to match the paper." This prevents the trap of reverting a genuine bug-fix just to make the paper 'reproduce.'

#### Downgrade rule: FAIL → EXPLAINED

A FAIL may be downgraded to EXPLAINED **only** when a *specific named alternative* is recorded for that exact claim — in the passport entry's `notes:` field (passport mode) or the audit report's author-note column (default mode). Example of a valid note:

> "reghdfe vs feols clustering-df adjustment; under the reghdfe small-sample correction the published value is −1.19, within rounding of the script's −1.187. CODE-CORRECTED pending."

The author is the **auditor**: the skill stages the two-sided comparison (reported value *and* computed value, both shown); the human writes the one-line named alternative; the skill records it and thereafter respects it. Tag the resolution `PAPER-CORRECTED`, `CODE-CORRECTED`, or `DEFENSIBLE-ALTERNATIVE`.

**Hard floor — never downgradable to EXPLAINED:**
- A blank note, "unclear", "looks fine", or any note that does not *name a concrete alternative spec*.
- An **UNMATCHED** claim (no computed counterpart to compare against).
- A flat numerical contradiction with no alternative offered.

(Citation/existence claims are out of scope here — `/verify-claims` owns those, and applies the same named-alternative softening on its side.)

#### Repeated EXPLAINED is a signal (two-strikes)

Reuse the two-strikes rule from `review-paper --adversarial` and [`summary-parity.md`](../../rules/summary-parity.md): if the **same** claim is downgraded to EXPLAINED in **two consecutive audits** without ever being corrected to PASS (the author keeps invoking the alternative but never updates paper or code), stop treating it as quietly resolved. Surface it prominently in Phase 5 — *"this contested number has been EXPLAINED twice but never corrected"* — so a standing disagreement can't hide behind a recorded note indefinitely. In passport mode, detect this by comparing the current `status`/`notes` against the prior audit's.

### Phase 4c: Horizontal check — the displays must agree with each other

Phase 4 compared the claim to the code. This phase compares the claim's displays to **each other**, pairwise over the `displays` collected in Phase 2b:

1. **Round both sides to the coarser precision.** Two displays agree when they are equal after both are rounded to the **smaller** of their two `display_precision` values. `0.34` in a deck against `0.342` in the paper agrees at 2 decimals; `0.29` against `0.342` does not.
2. **No tolerance applies.** The `tolerance:` block governs the vertical comparison, where two *measurements* are compared. Two displays of one number are two *copies* — after the rounding in step 1 they either match or they do not.
3. **Dispositions, in the vocabulary of Phase 4b:**
   - **PASS** — every pair agrees.
   - **FAIL** — any pair disagrees, or any display is `NOT_FOUND`. **Never downgradable to EXPLAINED**: a named alternative spec explains why code and paper differ and has nothing to say about why two copies of one number differ. If a display genuinely shows a *different* specification, it is a different claim and belongs in its own passport entry.
   - **UNCHECKED** — the claim declares no `appears_in` list. Reported, non-blocking; the fix is to declare the displays.
4. **Report which side moved, not which side is right.** Name the artifact whose value differs from the rest, and when it was last touched; the author decides whether the deck is stale or the paper is. Do not hand-edit one display to match its sibling — re-derive both from the output the claim points at ([`replication-protocol.md` → Anti-patterns](../../rules/replication-protocol.md#anti-patterns)).

### Phase 5: Report

Write `quality_reports/reproducibility_audit_[manuscript-name].md`:

```markdown
# Reproducibility Audit: [Manuscript Title]

**Date:** [YYYY-MM-DD]
**Manuscript:** [path]
**Outputs directory:** [path]
**Tolerance source:** .claude/rules/replication-protocol.md

## Summary

| Status | Count |
|---|---|
| PASS (vertical and horizontal) | N |
| FAIL (diff > tolerance, no named alternative) | M |
| EXPLAINED (out of tolerance, named alternative recorded) | E |
| UNMATCHED (manual review) | K |
| HORIZONTAL DRIFT (declared displays disagree, or a display not found) | H |
| HORIZONTALLY UNCHECKED (claim declares no `appears_in`) | U |
| **Overall verdict** | **PASS / FAIL** (FAIL iff M > 0 or H > 0; EXPLAINED does not fail the audit) |

## PASS (all within tolerance)
| Claim | Reported | Computed | Diff | Tolerance |
|---|---|---|---|---|
| Table2_col3_ATT | -1.632 (0.584) | -1.628 (0.591) | 0.004 / 0.007 | 0.01 / 0.05 |

## FAIL (outside tolerance — BLOCKER)
| Claim | Reported | Computed | Diff | Tolerance | Location in paper | Author note (name a concrete alternative to downgrade → EXPLAINED) |
|---|---|---|---|---|---|---|

## EXPLAINED (out of tolerance; defensible named alternative recorded — non-blocking, carry into response-to-referees)
| Claim | Reported | Computed | Named alternative (why the gap is defensible) | Resolution |
|---|---|---|---|---|
| Table3_col2_ATT | -1.187 | -1.19 | reghdfe vs feols clustering-df adjustment | DEFENSIBLE-ALTERNATIVE |

## UNMATCHED (manual review)
| Claim | Raw context | Candidate sources |
|---|---|---|

## HORIZONTAL DRIFT (one number, two values — BLOCKER)
| Claim | Display A | Display B | Compared at | Values | Which side moved |
|---|---|---|---|---|---|
| C3 | manuscript.tex — Table 1, Col 2 | Slides/Lecture04_Results.tex — frame 'Main result' | 2 decimals | 0.34 vs 0.29 | deck older than the claim's output_file |

## HORIZONTALLY UNCHECKED (no `appears_in` declared)
| Claim | Primary display | Other artifacts to declare |
|---|---|---|

## Environment
[sessionInfo excerpt]

## Next steps
1. Resolve each FAIL row — either correct the manuscript, rerun the analysis, or (if the gap is a defensible alternative spec) record a concrete named alternative to downgrade it to EXPLAINED.
2. Resolve each HORIZONTAL DRIFT row — regenerate the lagging display from the claim's output, never by retyping the other display's value. A `NOT_FOUND` display is fixed by correcting the `locator`, not by dropping the entry.
3. Review UNMATCHED rows — add explicit lookup keys or widen the search scope.
4. Declare `appears_in` for the HORIZONTALLY UNCHECKED rows — every artifact a reader sees the number in.
5. Review EXPLAINED rows before submission — each should map to a sentence in the response-to-referees.
6. After zero FAILs and zero HORIZONTAL DRIFT (EXPLAINED rows allowed), the paper is replication-ready.
```

### Phase 5b: Append to the replication log

The report above and the passport are rewritten on every run, so neither keeps a history. The log does. After every run — default and passport mode alike — **append** one block to `quality_reports/replication-log.md`. It is committed, so a co-author or data editor can see what was checked, against which commit, and how each number was obtained, without reading the code.

- **Append only.** Write with `>>`; never edit or reorder a past block. A correction is a new run, not an edit. The repo-hygiene gate fails a commit that edits or removes a committed line — at the pre-commit hook against the last commit, and in CI against the branch the work merges into. It proves no entry was edited, not that every run was logged. A disclosure redaction is the one exception: commit it with `ALLOW_LOG_REWRITE=1` and the reason in the commit message.
- **Stamp the revision.** The short commit hash, plus `-dirty` when anything outside `quality_reports/` is uncommitted (untracked files included) — so a verdict on uncommitted code says so, and the audit's own report files do not trigger it. Outside git, or before the first commit, the stamp is `no-commit`.
- **"How computed" is the exact accessor or command Phase 2 used** — enough for someone else to recompute that one number.
- **Restricted data:** the log carries the same numbers as the report, so it is committed only after disclosure clearance, like any output ([`confidential-data.md`](../../rules/confidential-data.md)).

```bash
LOG=quality_reports/replication-log.md
[ -f "$LOG" ] || printf '%s\n' "# Replication log" "" \
  "Append-only record of /audit-reproducibility runs: what was checked, against which commit, and how each number was computed. Never edit a past entry; a correction is a new run." "" > "$LOG"
# -dirty = anything uncommitted outside quality_reports/, untracked files included (the audit writes its own files in quality_reports/)
if REV=$(git rev-parse --short HEAD 2>/dev/null); then
  [ -n "$(git status --porcelain -- . ':(exclude)quality_reports' 2>/dev/null)" ] && REV="$REV-dirty"
else
  REV="no-commit"   # not a git repository, or nothing committed yet
fi
printf '## %s — %s @ %s\n\n' "$(date +%F)" "<manuscript path>" "$REV" >> "$LOG"
# Quoted heredoc: nothing below is expanded, so an accessor's `$` or backtick is written as-is.
cat >> "$LOG" <<'EOF'
Outputs: `<outputs dir>` · Verdict: **<PASS|FAIL>** (<M> FAIL, <H> horizontal drift, <E> EXPLAINED, <K> unmatched)

| Claim | Location | Reported | Computed | How computed | Tolerance | Verdict |
|---|---|---|---|---|---|---|
| Table2_col3_ATT | main.tex: Table 2, col 3 | -1.632 | -1.628 | `readRDS("output/results.rds")$coef[["treatment"]]` | 0.01 | PASS |

EOF
```

One row per audited claim, in the order of the report.

## Exit behavior

- **All PASS (or PASS + EXPLAINED):** exit 0, summary printed.
- **Any HORIZONTAL DRIFT:** exit 1, on the same footing as a FAIL — two artifacts showing different values for one number is a defect whichever of them is right. HORIZONTALLY UNCHECKED rows warn (exit 0) and are listed.
- **Any FAIL:** exit 1, summary printed to stderr. This makes the skill usable as a `/commit` pre-commit gate — see `replication-protocol.md` for the enforcement pattern. **EXPLAINED rows do NOT count as FAIL and never trigger exit 1** — they are surfaced, not blocking. The gate keeps its full teeth for genuine FAILs (no named alternative) and for fabricated/UNMATCHED claims.
- **UNMATCHED > 0 (with 0 FAIL):** exit 0 with warning — user must manually review.

## Source-language coverage

The skill compares manuscript claims against outputs in three source-language ecosystems. All three write to the same `output/` directory:

| Source | Default outputs dir | Read-output via | Common claim sources |
|---|---|---|---|
| **R** (default) | `output/` | `readRDS()`, `arrow::read_parquet()`, `vroom::vroom()` | `.rds` / `.parquet` / `.csv` / `tinytable` `.tex` |
| **Stata** (v1.9.0) | `output/` | `haven::read_dta()` from R, or `pyreadstat.read_dta()` from Python | `.dta` / `esttab` `.tex` / `.smcl` log values |
| **Python** | `output/` (or `_targets/`) | `pandas.read_parquet`, `pickle.load` | `.parquet` / `.pickle` / `.csv` |

**Stata-specific notes (v1.9.0):**

- `.dta` outputs are read via `haven::read_dta()` (R), `pyreadstat.read_dta()` (Python), or by parsing the corresponding `esttab` `.tex` if the table-cell value is what the manuscript cites.
- Manuscript cell `\input{output/tab_main.tex}` is the strongest provenance signal — the cell value comes mechanically from the .do file. Match the location in the `.tex` to the regression call in `03_analyze.do`.
- Clustering df adjustments can differ between `reghdfe` and base `reg, cluster()`. If a SE mismatches at the 2nd decimal, the tolerance in `replication-protocol.md` covers it; if it mismatches at the 1st decimal, investigate the df adjustment.

## Passport-mode (v1.9.0)

When `quality_reports/passports/<paper-slug>.yaml` exists, the skill operates in **passport mode**: instead of emitting a one-shot report, it **reads, updates, and rewrites** the passport file in place.

- For each `claims:` entry in the passport, perform the same numeric audit as the default mode (extract reported value from manuscript at `location`, locate computed value at `source_file:source_line` / `output_file:output_field`, compare against `tolerance:`), **then** the horizontal sweep of Phases 2b and 4c over the entry's `appears_in` list.
- After each claim is audited, update `status` in place:
  - PASS → claim within tolerance **and** every declared display agrees at the coarser precision.
  - FAIL → claim outside tolerance **and** the entry's `notes` does not name a concrete alternative; **or** two declared displays disagree; **or** an `appears_in` entry could not be located in its file. Record the discrepancy in `notes` — reported vs computed for a vertical FAIL, the two display values and their paths for a horizontal one. Blocks (exit 1).
  - EXPLAINED → claim outside tolerance **but** the entry's `notes` already records a *specific named alternative spec* (not blank, not "unclear"). The skill reads `notes` on its next run and resolves the same out-of-tolerance claim to EXPLAINED instead of FAIL — surfaced, non-blocking. The hard floor still applies: an UNMATCHED claim or a note without a named alternative stays FAIL.
  - STALE → if `source_file`, `output_file`, or any `appears_in` path has a modification time later than `last_verified_on`, mark STALE and re-run the audit logic (after the rerun, status becomes PASS / FAIL / EXPLAINED — STALE is transient).
- Update `last_verified_on` and `last_verified_by: "/audit-reproducibility"` per claim.
- Update `paper.last_audit` at the top level.

If a claim in the manuscript is detected that has no matching passport entry, emit an UNVERIFIED warning — the author should add it (passport scope is author-curated, not auto-populated, to avoid bad inferences).

Passport mode does NOT delete passport entries. If a claim disappears from the manuscript, the passport entry remains with a STALE status — the author decides whether to delete (claim retracted) or update the entry's `location` (claim moved).

Nor does it add `appears_in` entries on its own. A display the sweep happens to notice — the same value in a deck or a supplement that the passport never declared — is reported as a suggestion in the HORIZONTALLY UNCHECKED table; the author declares it. Same reasoning as the no-auto-populate rule above: an inferred display that is actually a different quantity would fail the horizontal check forever.

See [`.claude/rules/replication-protocol.md`](../../rules/replication-protocol.md) "Claims Provenance: `passport.yaml`" for the full schema and integration points (`/commit`, `/review-paper`).

## Cross-references

- [`.claude/rules/replication-protocol.md`](../../rules/replication-protocol.md) — the tolerance contract + passport schema.
- [`templates/passport-template.yaml`](../../../templates/passport-template.yaml) — starter file to copy for a new paper; the `appears_in` list is where displays are declared.
- [`.claude/hooks/claim-reconcile.py`](../../hooks/claim-reconcile.py) — the event-driven nudge: writing a tracked script, output, or declared display names the claims to re-audit. It counts declarations; this skill does the comparing.
- [`.claude/skills/review-r/SKILL.md`](../review-r/SKILL.md) — catches code-style issues; this skill catches NUMERICAL reproducibility.
- [`.claude/skills/diagnose/SKILL.md`](../diagnose/SKILL.md) — when a claim resolves to **FAIL** and you need to localize *which* pipeline step produced the out-of-tolerance value, hand off to `/diagnose` (single-claim root-cause: reproduce → minimise → bisect).
- [`.claude/skills/review-paper/SKILL.md`](../review-paper/SKILL.md) — content review; pair with this skill for a full pre-submission audit.
- [`.claude/skills/replication-package/SKILL.md`](../replication-package/SKILL.md) — gates on this skill before assembling the AEA DCAS deposit.
- [`.claude/skills/capture-environment/SKILL.md`](../capture-environment/SKILL.md) · [`.claude/skills/disclosure-check/SKILL.md`](../disclosure-check/SKILL.md) — environment capture + restricted-data screening downstream.

## What this skill does NOT do

- **Re-run your analysis.** The skill compares CURRENT outputs against manuscript claims. If the outputs are stale, re-run your pipeline first (the pre-flight phase will warn).
- **Catch wrong specifications.** A regression that compiles cleanly and produces a reproducible `-1.632` is reproducible. Whether `-1.632` is the RIGHT estimand is a `review-paper` / domain-reviewer question.
- **Check external package versions.** The `sessionInfo.txt` capture lets a reviewer see the env; pinning versions is on the user (via `renv.lock` or a `DESCRIPTION` file).

## Long batch reruns: use the Monitor tool (Apr 2026)

When `/audit-reproducibility` is asked to verify *all* numeric claims in a paper, the safest approach is to re-run the full pipeline (`00_run_all.R` or equivalent) and compare the regenerated outputs to the manuscript values. For pipelines that take more than a couple of minutes, background-launch the rerun and use Anthropic's **Monitor tool** (Apr 2026 Week 15) to stream stdout. The audit can react to errors mid-stream rather than waiting for the entire pipeline to finish before noticing a failed step.