Verify before judging

Every artifact carries context, and context rots. The question is never "is this correct?" — it is "is this correct here, now?"

Precise arithmetic on a broken instrument produces a confident wrong answer, and it produces it faster than sloppy arithmetic would.

Four checks, in order

Run them before the number becomes evidence. Order matters — a later check is meaningless if an earlier one fails.

1. Does the instrument measure what its name says?

Read the definition. Do not infer meaning from the field name.

Real examples, all from one week:

• oldest_age_h — I read "how long the oldest item has waited." It was MIN(source_post_timestamp) — the age of the content, not the queue wait. A cluster created today containing a month-old paper reported 984 hours.
• dup_vector_misses — I read "vectors that are missing." It counted near-identical vectors landing in different clusters. Not absence. The opposite: a real under-merge signal. A comment three lines up said so.
• published_today — a rolling 24-hour window, not a calendar day. Comparing two point-in-time readings measures window slide, not throughput.

Also check: is the value you're reading derived from another value you're also citing? Two numbers where one is f(other) are one piece of evidence, not two. (over_headroom = max(0, eligible − headroom) is not independent of eligible.)

2. Is the code actually running?

Deployed ≠ wired ≠ executing. Distinguish:

• Merged but not deployed
• Deployed but behind a flag that's off
• Flag on but the call site is gated by a different condition
• Running but writing to a key nobody reads

The last one is the most common and hardest to see. A counter that's written and never consumed looks identical to one that works, until you grep for readers and find zero.

Absence of a signal is not absence of the behavior. If you can't see it, first ask whether you could see it.

3. Is the environment healthy?

A regression in a metric during a deploy, a lock migration, or a cold start is usually the deploy, not the code. Check for transient conditions before diagnosing. Look at whether the failure recovers on its own before writing a fix.

4. Is the reading current, and does it postdate the event?

Never judge an event with a snapshot taken before it. Never re-cite a number from memory or from an earlier investigation — pull it fresh. Numbers written down an hour ago are a different claim than the same numbers now, and you will not remember which.

Ratios need samples

A ratio gives you direction. A sample gives you the verdict.

Before acting on a rate, look at the actual items in the numerator. Repeatedly, the sample overturned the conclusion the ratio implied:

• A 0.5 "miss rate" that was one event counted 27 times because the deduplication step couldn't fold non-Latin titles.
• A "50% drop rate" at an ingest gate that turned out to be correct behavior — the feed was general-interest and half the items genuinely weren't in scope.

Ask specifically: does this ratio count events or items? One large event generating many rows will dominate a rate and look like a broad failure.

Read the labels the code wrote for itself

Mature codebases annotate their own limits — in docstrings, in a limits field on the payload, in a comment above the constant. In every case where I got a metric wrong, the correct interpretation was written down within a few lines of the value.

The label was honest. I didn't read it.

Grep the module for limit, caveat, known, approximate, not, and read the comment block above the function before trusting its output.

Related

• falsifier-design — the same skepticism applied to safety metrics
• blast-radius — what to check before changing something