Design Resumable Model Evaluation

Make interruption cheap and evidence comparable.

Freeze an evaluation revision

Create a unique revision for every combination of corpus membership, model, prompt/instructions, schema, decoder settings, validator, run count, and thresholds. Hash this configuration into the state file. Never seed a new revision from incompatible results.

Write the case manifest before inference. Select cases from stable metadata, not previous outcomes.

Persist atomically

Use an append-only result stream or atomically replace a small state file after every case/run. Store:

• case and run IDs;
• frozen configuration hash;
• raw and normalized output;
• parse, identifier, semantic, execution, and safety gate results;
• abstention and generation-failure status;
• start/end timestamps and latency;
• interruption and retry metadata.

On resume, validate the configuration hash, scan completed keys, and execute only missing (case, run) pairs. Deduplicate safely if a crash occurred after writing output but before updating summary state.

Preserve benchmark semantics

Evaluate first-attempt output with exactly one generation when that is the product question. Do not call production remediation, repair malformed JSON, or deterministically choose a fallback. Exercise remediation in a separate test and metric series.

Keep required run counts and thresholds unchanged. A resumable evaluator improves execution efficiency, not acceptance criteria.

Stop only on safe math

For a minimum rate t over N required trials, fail early only when even winning every remaining trial cannot reach the required successes:

successes + remaining < ceil(t * N)

For maximum-failure gates, stop only once the allowed failure count is irreversibly exceeded. Apply the same reasoning separately to each gate and subgroup. Do not early-pass unless all required trials are complete.

Report

Report completed and required runs, completed fixtures/cases, failures, abstentions, gate numerators and denominators, distribution/diversity gates, p50/p95 latency, interruptions, resumptions, and exact automated gate results. State which human, device, energy, usefulness, or product gates remain untested.