falsify
Skill15 starsChanged 8 days agoMITView on GitHub
Red-team protocol for trying to refute a research claim (theorem, speed-up, security level, novelty) before anyone else does. Use at the P6 falsification gate, before any number or theorem goes into the paper, and whenever a sub-agent reports a surprisingly good result. Produces a verdict of survived, weakened or refuted, with evidence, in CLAIMS.md.
8 other projects use a version of this.
Adopted by
Most recently taken up by
- richardochristjia/skillstheir own version, first seen Nov 1, 2026
- angeliefu396-oss/falsifytheir own version, first seen Nov 1, 2026
- SWHee/skillstheir own version, first seen Oct 13, 2026
- 263311487-ux/falsifytheir own version, first seen Oct 8, 2026
- calm-trader/skillstheir own version, first seen Sep 25, 2026
- Spoon94/skill-vaulttheir own version, first seen Aug 5, 2026
Other versions8 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 8
- angeliefu396-oss/falsify★ 24same name
- Spoon94/skill-vault★ 5same name
- calm-trader/skills★ 2same name
- richardochristjia/skills★ 2same name
- 263311487-ux/falsify★ 0same name
- ccst-llc/skills★ 0same name
- dandye/falsify-skill★ 0same name
- SWHee/skills★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository29 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- abstract-craftSkill
- anonymize-checkSkill
- artifact-packSkill
- baseline-pinSkill
- bench-protocolSkill
- bib-verifySkill
- camera-readySkill
- eprint-searchSkill
- experiments-writingSkill
- figure-harvestSkill
- idea-mining-loopSkill
- lean-bridgeSkill
- lit-matrixSkill
- log-to-evidenceSkill
- mock-reviewSkill
- paper-figuresSkill
- paper-playbookSkill
- param-estimationSkill
- phase-gateSkill
- polish-writingSkill
- reading-notesSkill
- rebuttalSkill
- related-work-writingSkill
- sage-checkSkill
- technique-overviewSkill
- textbook-indexSkill
- venue-calibrationSkill
The file
We split this skill into the rules it gives an agent
6 rules. 0 are common, 5 are distinctive, and 1 is very specific to this project.
0
Common
Most of the 7 edited versions we read keep these.
5
Distinctive
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 18
If it is wrong, say exactly where.
- Distinctive: line 19
Every attempt must actually run something: a computation, a script or a search.
- Distinctive: line 21
If no counterexample is found, say "could not refute" and list what was tried.
- Distinctive: line 52
Do not use the paper's table or a previous summary.
- Very specific: line 69
Literature pre-emption: Search for the core identity or algorithm under other names.
- Distinctive: line 105
When fanning out, give each sub-agent:
“Common” is measured: kept by at least 6 in 10 of the 7 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
falsify: try to break the claim before a reviewer does
Principle
The job is to refute the claim, not to confirm it. An agent handed a framework tends to collect supporting evidence and amplify a wrong premise. In practice, most refuted claims in multi-agent rounds turn out to be the orchestrator's own hypotheses. So:
- The proposer first writes the claim as a falsifiable statement. It names exact quantifiers and parameters, gives a kill switch (the observation that would refute it), and describes the smallest experiment that could refute it.
- The falsifier's prompt says: "Your primary task is to verify this, not to redo it. If it is wrong, say exactly where; that is worth more than a confirmation."
- Every attempt must actually run something: a computation, a script or a search. Report real outputs. Never invent them.
- If no counterexample is found, say "could not refute" and list what was tried. Do not hide behind vague wording.
Input
A row in CLAIMS.md (ID | claim | proposer | status | falsifier evidence | EVIDENCE refs)
with status open, plus the sources it cites (THEORY.md section, EVIDENCE rows, logs,
code).
Attack order (cheap and decisive first)
Work through all six. Stop early only on a clean refutation.
- Definitions. Are all objects well-defined for every parameter allowed? Does a symbol mean the same thing in the abstract, the theorem and the experiment? Typical failures: two different definitions of the same quantity (for example, a padded length and a polynomial degree written with one symbol); an "element of order r" that is not guaranteed to exist; a cost measure that counts different operations in the two arms.
- Quantifiers. Is it "for all" or "there exists"? For all parameters, or only for those satisfying an unstated side condition? Does the proof use the hypothesis in the direction stated? Is the claimed injectivity or existence only checked on the examples? Is a necessary condition presented as sufficient? Write the negation and try to satisfy it.
- Edge parameters. Try the smallest values (p = 2, 3; n = 1; r = 1, 2), boundary
values (p ≡ 3 mod 4 when the claim needs p ≡ 1; B at the admissibility threshold),
degenerate cases (a trivial group, a zero polynomial, an empty set) and the
largest parameter used in the experiments. Do a small exhaustive search wherever
feasible, for example with
skills/sage-check. One finite counterexample settles the question. - Numeric recomputation. Recompute every number in the claim from the raw logs.
Do not use the paper's table or a previous summary. Check:
- that the rows of a table come from the same configuration. Mixing the fastest-time column from one setting with the fewest-operations column from another is a classic failure;
- that "best vs best" is really best vs best, and the ratio uses the stated protocol;
- arithmetic inside tables (sums, degrees, ratios, rounding:
roundvsceilcan differ by exactly 1); - that the same number is identical everywhere it appears: abstract, introduction, evaluation, appendix;
- that failure modes are attributed correctly. An uncaught exception with a core dump is not a "graceful noise-budget exhaustion" until the error message says so.
- Hidden baseline unfairness. Different parameters (the library's real preset vs
the one printed in the paper), threads, compiler flags, machine, day or load; a
stale build; a weaker baseline than the strongest available; a cited number never
reproduced locally; offline work moved out of the timed region; a
single-configuration win reported as general. Cross-check with
skills/baseline-pinandskills/bench-protocol. - Literature pre-emption. Search for the core identity or algorithm under other
names. Check: the target paper's own citations; ePrint from the last 24 months;
follow-ups citing the baseline; the textbook treatment (ask
math-librarian). A reformulation of known work ("X is exactly Y viewed as a module over Z") downgrades novelty even when everything is correct. Record the ePrint or DOI. Never cite from memory; if a reference cannot be verified, write "needs verification".
Verdict taxonomy (write into the CLAIMS.md status column)
| verdict | meaning | required evidence |
survived | every attack in the order above was attempted and none succeeded | list of attempts, with commands and outputs or the searches run |
weakened | true only under an extra hypothesis, over a smaller parameter range, with a smaller constant, or with less novelty than claimed | the precise restated claim, plus the counterexample or recomputation that forced it |
refuted | a counterexample, a recomputation contradicting the number, or prior art containing the result | a minimal reproducible counterexample (script plus output) or the exact prior-art citation |
A weakened verdict must include the replacement claim text. The proposer either
adopts it or appeals to pi-orchestrator. Keep refuted rows in the table with
their reason, so no one repeats the dead end.
Output format
| C7 | For every prime p ≡ 1 (mod 4) and every B ≤ B_max(p), map φ_B is injective | theorist | weakened | sage-check exhaustive p<200: first failure p=13, B=2 (script: checks/c7.py, output: results/c7.txt). Holds when 2(|A|+1)B < p (sufficient, not necessary). Replacement: "... if 2(|A|+1)B < p" | E14 |Also append to the shared errata section (STATE.md → "Errata and rolling updates")
when a verdict changes something other agents rely on. State which source now wins.
Veto
falsifier holds a veto. No claim with status open or refuted may appear in
paper/. A weakened claim may appear only in its restated form. writer checks
CLAIMS.md before each section, and submission-rebuttal re-checks before submission.
Review-brief pattern (for parallel falsifier sub-agents)
When fanning out, give each sub-agent:
- one group of claims (arithmetic and units; performance numbers; artifacts and links; argument completeness; novelty);
- the verdict labels required on the first line (
REFUTED/CONFIRMED/UNVERIFIABLE); - a "do NOT do" list naming what the other sub-agents cover;
- an instruction to read the other sub-agents' outputs and explicitly adjudicate any conflicts.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

