evaluator-scoring-surface-audit
Skill4 starsChanged 21 days agoMITView on GitHub
Read a benchmark's evaluator, verifier and scoring source line by line for gaps between what is counted, what is checked and what is only assumed, then turn each gap into a priced lever, a tool, or a report to the organisers. Use when joining a benchmark, after any evaluator change, or when the leaderboard moves in a way your cost model does not explain.
In this repository79 other files it gives its agents.
- AGENTS.mdAGENTS.md
- accepted-proof-exact-retuneSkill
- arxiv-portable-researchSkill
- auto-autoresearch-pipelineSkill
- beachhead-ratchetSkill
- bounded-commuting-foldSkill
- burn-the-house-downSkill
- calibrated-static-cost-proxySkill
- circuit-edit-integration-safetySkill
- colab-laneSkill
- computer-algebra-oraclesSkill
- constraint-construction-searchSkill
- cross-model-review-gateSkill
- deepseek-pressure-testSkill
- ecdsafail-fastevalSkill
- ecdsafail-lowq-playbookSkill
- ecdsafail-oracle-loopSkill
- ecdsafail-pareto-ratchetSkill
- ecdsafail-scratch-dietSkill
- elicit-researchSkill
- exact-evaluator-replicaSkill
- exact-hold-passSkill
- exact-postpass-stackSkill
- exact-search-engineeringSkill
- fast-lean-proof-loopSkill
- fit-probe-and-spliceSkill
- gpu-affine-table-synthesisSkill
- greedyfrog-methodSkill
- hard-problem-contractSkill
- hold-demand-ledgerSkill
- hosted-carry-lendingSkill
- inert-op-screen-to-structural-ruleSkill
- invariant-chartSkill
- jev-second-perspectiveSkill
- lambda-exchange-deskSkill
- leader-rebase-stackSkill
- lean-arklib-proofsSkill
- lean-finite-certificate-splittingSkill
- lever-switches-and-break-controlsSkill
- mission-commandSkill
- nasqret-playbookSkill
- paged-walk-descentSkill
- peak-lifetime-ledgerSkill
- pinned-table-stackingSkill
- pip-disciplineSkill
- portable-source-assertionsSkill
- proximity-prize-campaignSkill
- proximity-prize-frontier-archaeologySkill
- quantum-affine-lookup-synthesisSkill
- quantum-borrowed-wire-counterexampleSkill
- quantum-dense-qromSkill
- quantum-dirty-borrowingSkill
- quantum-divstep-reconstructionSkill
- quantum-encoded-arithmeticSkill
- quantum-graded-divider-placementSkill
- quantum-guarded-permutation-auditSkill
- quantum-mbundo-lifetimesSkill
- quantum-record-miningSkill
- quantum-replay-arithmeticSkill
- quantum-residue-scan-ambiguitySkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 5 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 15
Use when joining a benchmark, after every evaluator or verifier commit, and when a rival's score moves in a way your cost model cannot …
- Distinctive: line 17
Do not use it to look for ways around the stated rules.
- Distinctive: line 50
List every source of randomness and what seeds it: When the artefact's own bytes seed its test inputs, a no-op edit re-rolls the sample.
- Distinctive: line 73
Every score term has a line reference.
- Distinctive: line 75
Every harness defect has a failing test before the fix and a passing one after.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Chliidruckts (evaluator-scoring-surface-audit)
The scoring surface is the set of quantities the score reads and the set of properties the verifier checks. A gap is a place where the two differ: something counted that can fall without touching a checked property, or a check whose bound differs from the scorer's. The audit lists both sets with line references, crosses them, and classifies every gap before anything is built.
Use it / leave it
- Use when joining a benchmark, after every evaluator or verifier commit, and when a rival's score moves in a way your cost model cannot explain.
- Do not use it to look for ways around the stated rules. A gap that contradicts the benchmark's written intent is reported to the organisers and not used. The source campaign flagged its own such observation in the submission note that used an executed-count lever.
- It does not replace reproducing the metric. Build or obtain an exact local evaluator first
(
exact-evaluator-replica).
Inputs
- Evaluator, verifier and scoring source at a pinned commit, including the official runner's wrapper script.
- The written rules.
- Two or three recent top submission notes.
Procedure
- List every quantity the score reads. For each: the exact line that increments it and the condition under
which it increments. Read the code, not the README.
Worked example: the charged-gate term is the average executed count over the sample, not the emitted count; and
the counter increments by executed samples even when the gate's controls are never both set.
Done when every term of the score formula has a file:line and an increment condition in
assets/surface-table.md. - List every check. Value, phase or side-effect cleanliness, scratch, size caps, allowed imports, time and memory limits. For each: its line, its bound, and its sampling (all inputs, a sample, or none). Done when every reason the verifier can reject is a row.
- Cross the two lists.
- For each counted quantity: can it be lowered without touching any checked property?
- For each check: is its bound the same as the scorer's bound for the same quantity?
- For each written rule: is there a line that enforces it? Worked example of a bound mismatch: a verifier capped a claim file at 128 bytes, which limited a claimed constant to 122 digits, while the scorer accepted values below 10^295. The fix raised the cap to 302 bytes and added a boundary test at 10^295 - 1 and 10^295. Done when each cell of the cross has "no gap" or a named gap.
- List every source of randomness and what seeds it. When the artefact's own bytes seed its test inputs, a no-op edit re-rolls the sample. That fact turns every sampled check into a search problem and must be in the table. Done when each random draw has a seed source and a statement of who can influence it.
- Classify each gap.
class meaning action exact lever value-preserving on all inputs price it, then build sampled lever value-preserving on the verifier's sample only needs a failure-rate budget and a disclosure; see inert-op-screen-to-structural-ruleharness defect the harness contradicts itself (bound mismatch, crash, wrong accounting) report with a patch and a test intent conflict legal by the code, against the written intent report, do not use Done when no gap is unclassified. - Price the top three with one ten-minute experiment each before building anything. An experiment is the smallest edit that moves the counted quantity, scored by the judge or the replica. Done when three gaps have a measured price or a measured zero.
- Write the audit as a table in the campaign repository, pinned to the evaluator commit. Re-run steps 1 to 3 on every evaluator commit and diff the tables. Done when the table is committed with the pin in its header.
Checks that say it worked
- Every score term has a line reference.
- At least one gap is confirmed by an experiment, or the audit states "none found" with both lists attached.
- Every harness defect has a failing test before the fix and a passing one after.
- Both evaluator paths were read: the local command and the official runner.
Failure modes
- Treating a sampled lever as exact. "Never fires on the sample" is not "never fires".
- Using an intent conflict silently. State the observation in the note and tell the organisers.
- Auditing the README instead of the code. The README says what was meant; the line says what is counted.
- Missing a second evaluator path. The local command and the official runner can differ in limits, flags or versions.
- A stale audit. The table is valid for one evaluator commit.
Transfer
Direct to every benchmark with public scoring code: circuit cost, prover time or cells, gas, proof size. The same four classes apply. On a benchmark with exact verification the "sampled lever" class is empty and every candidate lever needs a proof.
Evidence this pays
Citation key: N:<id8>:<line> is a public submission note of the source campaign (solver BitWonka, ecdsa.fail)
by id prefix; T: is a file in the public ecdsafail-tools repository; E: is a commit in that solver's evm.fast
fork. Numbers are quoted; none was re-measured for this skill.
- Executed against emitted count: conditional replay moved 1,452,820 emitted to 1,404,058 executed at 1,300 qubits (N:4b7d7272:8-14). The construction is credited to Codex (OpenAI) in that note (:12). The same note flags that scoring on emitted count may match the intended cost model better (:10).
- Charged but inert ops: one line of evaluator behaviour (T:README.md:15) is the basis of the inert-op drop, which the tool's README describes as a reimplementation of a lever already in the field.
- Stream layout: one line (T:fasteval/README.md:19) is what makes an exact parallel replica possible.
- Bound mismatch: E:7523d379 is a two-file harness fix with a boundary test and no score evidence.
The record shows the pattern paying under this account, with mixed authorship: three findings with a score or tool effect and one harness fix without score evidence. It does not show that one person ran it as a single procedure; the seven steps are a generalisation.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

