Falsify First

Before you argue for a claim, try to make it false. A claim that nobody attacked is not evidence.

Rules

• Search for the counterexample before you defend the claim. A failed candidate shows the shape of the correct one.
• When the claim breaks, find the precondition that carried the weight. A patch that makes it pass hides the cause.
• Split a precondition that does two jobs, and attack each half alone. A double-duty condition is the most frequent hidden cause.
• Measure the baseline of a claim about a difference first. "X is present where it should not be" is false when X is already present by another route.
• Exclude the instrument from its own population. A probe that counts its own definition agrees with itself and proves nothing.
• State what measured a number and where it ran. A swapped instrument or a wrong tree gives a wrong number with no symptom.
• Never report a manual search as a proof. Write "attacked this way, did not break" and name the way.
• Derive a high-risk number by two paths that share no calculation step. When they disagree, investigate the instrument first.
• When two causes predict the same result, change the probe until they differ. One fact against a cause does not prove another.
• For a claim of complete coverage, name the list that the code reads. A general mechanism does not prove which items it covers.
• Before you retire a source system, account for every original file. A check of what the import carried cannot see a file that the import never read.

Runbook

1. State the claim so it can be false. Name the observation that refutes it. If you cannot name one, you have a preference. Sharpen it into a claim first.
2. Pick the cheapest oracle that can say no. From cheapest to most expensive:
   • run it;
   • a property test over generated inputs;
   • a set of adversarial inputs that you build by hand;
   • a second implementation;
   • a type checker or a contract checker.
3. Search, do not sample for confirmation. Bias the inputs toward the boundary, the degenerate case, the empty case and the case the author finds inconvenient.
4. When it breaks, read the counterexample. The precondition that carried the weight is usually visible in it.
5. Stop on a survivor and record the attack. "Survived N generated cases and these three adversaries" is evidence. "I could not find a counterexample" is not.

Build the adversary from the failure

When an analysis finds a mechanism by which the claim can fail, make that mechanism the next test case. Write it down while you have it. A generic fixture set is weak.

When no cheap oracle exists, name the input that would falsify the claim and look for it by hand. Say in the deliverable that the search was manual and what it covered.

Measurements and third-party projects

• Before a number, a difference or a coverage figure enters a decision, read measurement: baselines, excluding the instrument, pinning the tree, and the limits of this method.
• Before a document recommends a third-party project, read third-party.