audit-benchmark-validity

Purpose

Treat benchmarks as scientific measurement instruments: audit construct validity, contamination, metric pathology, coverage, leaderboard dynamics, and protocol drift.

Input contract

required: [benchmark_spec, task_definition, metric_definition, evaluation_records]
optional: [leaderboard_history, protocol_versions, coverage_target]
constraints: [claims must be linked to benchmark evidence]

Execution protocol

Do not perform called SOP operations inline; each loaded SOP owns its contract and thresholds.

1. You MUST load skill inventory-reference-items to inventory benchmark components and versions. You MUST load skill decompose-evaluation-metric to decompose the evaluation metric. You MUST load skill assess-construct-validity to assess the benchmark construct. You MUST load skill extract-evaluation-protocol to reconstruct protocol versions.
2. Select validity, contamination, saturation, coverage, protocol-forensics, or evaluation-comparison mode.
3. You MUST load skill audit-data-contamination to audit contamination. You MUST load skill map-coverage-space to map data and task coverage. You MUST load skill analyze-leaderboard-dynamics to analyze saturation and temporal dynamics. You MUST load skill compare-evaluation-protocols to compare protocol and metric variants. You MUST load skill probe-benchmark-artifact to probe artifacts and shortcut paths.
4. You MUST load skill audit-reporting-quality to audit reporting quality and return the validity verdict with threats, evidence, and required repairs. If the audit exposes systematic coverage gaps, consider coverage-white-space-search. If the benchmark or metric may share assumptions with the tested system, consider audit-validator-independence.

Output contract

produces: [validity_verdict, threat_register, contamination_findings, coverage_map, protocol_drift_report, repair_actions]
delta_fields: [findings, evidence_updates, uncertainties, decisions, open_questions, recommended_jumps]

Thresholds and quality gates

• Every source HARD-GATE remains mandatory; no exit with an untested construct, contamination path, metric pathology, coverage claim, or protocol change.
• Resource and sampling gates use relative benchmark/target/evidence coverage rather than fixed benchmark, paper, or web counts. Declare the eligible universe, record numerator, denominator, batch increment, stopping reason, and source references.
• Saturation claims require an explicit stopping criterion and evidence that additional search/testing no longer changes the conclusion; report marginal information gain and saturation state.
• BetterBench-style criterion lists remain content checklists. Their item count is not converted into a percentage; the audit reports criterion coverage ratio over the declared applicable set and an independent-source ratio.
• Evaluation comparisons must state the controlled protocol difference and its expected impact.

Failure and counterexamples

Reject "valid" when benchmark artifact probes are absent, contamination is unknown but ignored, or leaderboard gains cannot be separated from protocol drift. A high score is not evidence of construct validity by itself.

Provenance map

7 architecture old entries: archaeology, audit, saturation, validity probing, coverage mapping, protocol forensics, evaluation comparison. Provider-specific retrieval compressed.

Legacy context checkpoint / Delta notes

Append benchmark version, construct claims, probes, contamination evidence, coverage gaps, protocol diffs, verdict, and repair decisions.

Preserved source criteria ledger

Context checkpoint / Delta notes

Append benchmark version, construct claims, probes, contamination evidence, coverage gaps, protocol diffs, verdict, and repair decisions.