Verify Outputs

A figure that renders is not a figure that is right, and neither is a figure that can be read. This skill exists because those are three separate properties and only the first one is self-evident.

Every defect that motivated it was invisible in the source and obvious in the image.

The three checks — all required, in this order

1. Accuracy — recompute, do not read. Verify a table by recomputing it from the source, not by looking at it. Every count in a caption, every N in a table, every percentage in a sentence is a claim, and each one is checkable. If you wrote the number and the note describing it, the note is a hypothesis you wrote down, not evidence.

2. Sanity — predict, then look. Say what the output should show before you open it. Does the geography match known geography, the sign match theory, the magnitude match the units? An output that renders cleanly and says something impossible is the most dangerous kind, because nothing complains.

3. Readability — open it and look at it. Can the thing it exists to show actually be seen? Labels legible and non-overlapping, colour scale the right way round, axis range letting the signal show — and does the caption match what the image actually draws? A caption that asserts what the figure cannot show is a defect of the same severity as a wrong number, because the reader believes the caption.

Run the screen

python -X utf8 verify_outputs.py --figures FIGDIR [FIGDIR ...] --tables TABLEDIR [...]

Exit code 1 on findings, so it can gate a commit. It reads rendered pixels, because that is where the defects live. Then review by eye everything it flags, plus a sample.

Sample across mechanisms, not at random

Defects live in shared policy code — a colormap rule, a norm rule, a scale rule — so each one hits every artifact that policy touches, 4–30 at a time. Pick one artifact per colour policy, per normalisation type, per data family. Reviewing 14 figures chosen this way found six distinct defects; random sampling of 14 would likely have found one or two.

Say what you eyeballed versus what you screened

"I reviewed them" when you screened them is a false claim about your own work. State both numbers.

The recurring failure modes

Nine mechanisms, each observed in production:

1. The conditional that never fires. if "flag" not in matplotlib.colormaps: register(...) — flag is a builtin, so the branch never ran and 30 maps silently drew with the wrong colormap, inverted. No error; the code reads correctly. If you write if X: do_important_thing, prove X is ever true. (Taken matplotlib names include flag, spring, winter, ocean, terrain, gray.)
2. "This run's list" treated as "everything that exists." A stale-artifact sweep deleted 317 good files, and a notes table was truncated 321 rows → 5, both because a partial run wrote or deleted as if it were a full one. Anything that deletes or overwrites from a computed set must know whether the set is complete or partial. Write the partial-run case first; it is the one that loses data.
3. Masked values rendering as missing. LogNorm masks non-positive values and a colormap's bad-colour is transparent by default, so zeros drew blank — identical to no-data — across 16 maps and 39,829 values, under titles reading "0 no data". Set cmap.set_bad(...) explicitly whenever a norm can mask.
4. A guard written from a guess about the failure. A completeness check rejected any file containing the substring "404"; the files were pipe-delimited counts, so |4046| matched and 15 of 77 provinces were thrown away. Produce the failure mode and look at it before writing the guard against it.
5. Comparing against the wrong baseline. Two measurements came from different rasters on different grids; the 5.7% difference was the baseline, not the treatment. Caught only by a null case — the subpopulation where the treatment does nothing must measure nothing. Build that control every time.
6. Invariants from intuition instead of the domain. "No place is closer to a port than to the coast" failed for 3,091 units — Bangkok Port is 18 km up a river, as are Hamburg and Antwerp. When an invariant fails at scale, suspect the invariant first. Prefer invariants on the small curated set (8 ports) over the large derived one (7,586 units).
7. A convention applied outside its domain. A fixed 0–1 colour scale is right for variables spanning that range and wrong for ones topping out at 0.45 — where it wasted half the palette and hid the study's main gradient. A convention needs a stated domain.
8. A headline statistic that is a count of threshold crossings. "9 of 18 changed sign" was accurate and knife-edge: one case sat at +0.0017. Report magnitude of movement alongside the count, and how many cases sit near the threshold.
9. Fixing a symptom before understanding the mechanism. A first fix moved the metric the wrong way; the cause was a 3×3 kernel, not the thing being adjusted. When a fix worsens the metric, that is information — enumerate candidate mechanisms and measure to discriminate before changing anything.

Two asymmetries

Loud failure is cheap; silent success is expensive. The bad guard failed loudly and cost twenty minutes. The unregistered colormap succeeded silently and shipped thirty wrong figures. Be most suspicious of the code that never complains.

A checker you have never seen fail is an untested checker. Run new validations against known-bad input deliberately and confirm they fail. A check that only ever passes is worthless.