Run the evaluation

Your eval harness (<your-eval-command>) is the one committed way to get an accuracy number. Do not hand-roll a measurement loop in a scratchpad — ad-hoc harnesses produce numbers that are not comparable across sessions, and can be wrong by a large factor without anyone noticing (a several-fold mirage has actually happened from an ad-hoc script).

Copy this checklist and track your progress:

• 1. Reuse (reuse-before-rerun): same split, config and commit already run? Go to Step 4
• 2. Run the shipped config on <dev-split>; held-out only on the user's explicit words
• 3. Run the gates-off config when the failures need diagnosing
• 4. Read headline, emit rate and precision-on-emitted together
• 5. Compare with the goals/bounds file; cite the run id with every number

If the record lacks the commit SHA, dirty flag or split hash, fix the harness and return to Step 2.

Wire this up

Substitute for your project:

• <your-eval-command> --split <split> — the committed eval entrypoint (a script that runs your model/pipeline over a named split and writes a run record)
• the split table below — replace names/sizes with your project's actual splits
• the metrics list — replace with your project's headline + supporting metrics
• <runs-directory> — wherever run records persist

The command

<your-project-python> <your-eval-command> --split <dev-split>

A held-out split should refuse to run without an explicit "I mean it" flag or similar guard. Do not pass that flag on your own initiative. It exists so at least one number at the end of the project has never been optimised against; spending it early on routine iteration cannot be undone. Optional guard: the opt-in guard-holdout hook (hooks/guard-holdout.py in this plugin; setup in hooks/README.md) stops a held-out run until the user has agreed.

Read multiple numbers, not one

A single accuracy figure cannot tell "we fixed a wrong output" from "we stopped producing one". Read at least a headline correctness metric plus an emit/coverage rate and a precision-on-emitted rate together, e.g.:

headline_correct_rate     38.2%   191/500   <- HEADLINE
emit_rate                 39.8%   199/500
precision_on_emitted      96.0%

Concretely, in a naming engine: turning off validity gates moved emissions 199 → 441 while the headline round-trip-exact rate stayed at exactly 191, and precision on what was emitted fell 96.0% → 43.3%. Loosening a gate bought raw coverage, not correctness — a single rate would have hidden that. If your emit/coverage rate rose and your precision fell, the change bought coverage with less honesty; say so explicitly rather than reporting the coverage number alone.

Never quote a cheap proxy metric as the result — string/token similarity (BLEU, token accuracy, edit distance) against a reference label, or any metric computed against noisy/self-inconsistent reference data. If the harness computes them, print them labelled clearly as secondary/diagnostic, never as the headline. Measured case: on one split, a proxy metric fell between two runs whose true correctness metric (reference-free round-trip validity) was identical — the proxy tracked emission volume, not correctness, because the reference labels it compared against were only ~82% self-consistent.

Prefer a reference-free validation whenever one exists. For structure↔name work that means: round-trip the output through a parser/canonicalizer (e.g. parse the generated name back to a structure, canonicalize both the input and the round-tripped structure — RDKit → canonical InChIKey — and compare) rather than comparing strings to a reference name. The same pattern generalizes elsewhere: round-trip a predicted SMILES through a canonicalizer and compare it to the input; check a predicted reaction's mass balance; verify a docked pose against a physical/geometric constraint. Reference-free checks catch errors that string-similarity to a noisy reference label cannot.

Two configurations, and why you need both

<your-eval-command> --split <dev-split>                 # shipped: the headline
<your-eval-command> --split <dev-split> --gates off     # diagnostic: the causes

If your pipeline has validity/self-consistency gates that turn a bad output into an abstention, run both: the shipped config is the honest number to report, but it collapses every cause into one "abstained" bucket. The gates-off (or otherwise unfiltered) run surfaces the underlying failure reasons (e.g. parse_fail vs constitution_mismatch) so you can cluster and diagnose them separately. If the headline metric is identical in both configs, this costs nothing extra in headline terms — cluster the gates-off run, report the shipped one.

Before you run

• You may run alongside other measurement or gate work if nothing writes the same output artifact. Budget a shared resource (license, GPU, validator) with a lock that is released even when a worker is killed, not with pgrep, which matches itself.
• Check what your run's logs look like at scale before relying on grep-based filtering. Verbose per-row warnings can make a large run's output unreadable; filter them from what you read, keeping the full log on disk (e.g. 2>&1 | tee <run>.log | grep -v "WARNING:"), because refusal-census parses those warnings.

Output

Every run should persist to <runs-directory>/<timestamp-or-id>.json (or similar) with, at minimum: the commit SHA, whether the working tree was dirty, the split's content hash, and the resolved configuration (which gates/flags were active). Cite the run id whenever you quote a number. Without that provenance a number is not interpretable — a config knob can move the emit/coverage rate by tens of points, and a record that keeps only a timestamp can't tell two configurations apart after the fact.

Thresholds

Check a result against your project's committed goals/bounds file (a goals.json or equivalent — see the eval-loop skill for how to structure one) rather than against memory. That file should hold the current baselines and which bound is the headline.