Evidence layers

Most wrong audits are not wrong about the facts. They are right about one layer and reported as if they were about another.

A single mature system usually holds the same entity four times over:

The rest of this skill is the discipline for not mixing them.

1. Declare the layer before you measure anything

Write the layer down before running the first command, not after. Either:

• every figure in the report carries its own layer, or
• the report opens with one line — "All figures are production unless marked" — and every exception is marked.

A report with no layer line is not a report yet.

2. A claim about users must be measured against production

Source files are evidence of intent. They are not evidence of what is served.

The gap between them is where a real estate lives: things built and never published, things published then superseded, things archived but still on disk, things hand-edited in production that no file explains. Every one of those produces a confident, wrong finding if you read the wrong layer.

Scan production. If you cannot, say the finding is unverified against production, in the finding itself, not in a footnote.

3. "Not found" is not "does not exist"

Before reporting anything absent, prove you looked in the right place:

• Name the exact paths, tables or queries you searched.
• Search by id, not by name. Names are edited; ids are not.
• Check the archive before calling something missing — a retired row looks identical to an absent one from most angles.
• Check whether it moved. Directory reorganisations are the single commonest source of "we have no X".

A finding that says X is missing must be able to answer where did you look? in one line. If it can't, downgrade it to X not found in <place>, which is a different and much weaker claim.

4. Check whether the thing is retired before calling it broken

Archived, deprecated and feature-flagged-off entities are full of exactly the defects an audit is hunting: stale fields, broken references, missing translations, dead links. Reporting them wastes the reader's attention and erodes trust in the findings that matter.

For every defect found, one question first: is anything actually serving this? Trace it to a route, a query, a menu entry, a feed. If nothing reaches it, the finding is "retired content still stored", severity low, not "broken feature", severity high.

5. Sample size is part of the claim

"The design is inconsistent" from one file read is not a finding, it is a hypothesis. Qualitative claims about a population need either a census or a stated sample:

• Census — you enumerated all of them. Say how many.
• Sample — say how many you looked at, out of how many, and how you picked them.

Never generalise from the first item you opened. It is the first item because it sorted first, and things that sort first are disproportionately old.

6. Prefer the existing read-only tooling

Mature repos accumulate audit and inventory scripts. Use them before writing a new scan:

• They already handle the layer selection, so you inherit the right default.
• Their output is comparable with the last report, which turns a snapshot into a trend.
• A new scan written for one report has never been reviewed and usually disagrees with the old one for uninteresting reasons.

Find them first (scripts/, tools/, bin/, the CI config). Write a new one only when you can say what the existing ones cannot answer.

7. Report format

Rank by severity, and make each finding carry its own evidence:

SEV  FINDING                          LAYER       EVIDENCE
---  -------------------------------  ----------  --------------------------
1    <what is wrong, in one line>     production  <ids, counts, query used>
     Impact: <who is affected, how>
     Fix:    <the smallest change that resolves it>

Separate confirmed from suspected into two lists. A reader who cannot tell which is which has to re-verify everything, which is the same as having no report.

Checklist

• Layer declared before measuring
• User-facing claims measured against production
• Every "missing" says where it looked
• Every defect traced to something that serves it
• Every population claim carries a census or a sample size
• Existing audit tooling used or its absence explained
• Confirmed and suspected findings kept apart