exact-evaluator-replica
Skill4 starsChanged 21 days agoMITView on GitHub
Build and validate a fast, exact local copy of a benchmark's official evaluator: same inputs, same counts, same score, proven by a differential fuzz test and a reproduced graded log. Use when the official judge is too slow, too memory-hungry or too opaque to sit inside the edit loop of an optimization campaign (circuits, provers, gas, proof size), when many candidates must be scored per day, or when a failure-rate estimate needs fresh samples the judge cannot give.
In this repository79 other files it gives its agents.
- AGENTS.mdAGENTS.md
- accepted-proof-exact-retuneSkill
- arxiv-portable-researchSkill
- auto-autoresearch-pipelineSkill
- beachhead-ratchetSkill
- bounded-commuting-foldSkill
- burn-the-house-downSkill
- calibrated-static-cost-proxySkill
- circuit-edit-integration-safetySkill
- colab-laneSkill
- computer-algebra-oraclesSkill
- constraint-construction-searchSkill
- cross-model-review-gateSkill
- deepseek-pressure-testSkill
- ecdsafail-fastevalSkill
- ecdsafail-lowq-playbookSkill
- ecdsafail-oracle-loopSkill
- ecdsafail-pareto-ratchetSkill
- ecdsafail-scratch-dietSkill
- elicit-researchSkill
- evaluator-scoring-surface-auditSkill
- exact-hold-passSkill
- exact-postpass-stackSkill
- exact-search-engineeringSkill
- fast-lean-proof-loopSkill
- fit-probe-and-spliceSkill
- gpu-affine-table-synthesisSkill
- greedyfrog-methodSkill
- hard-problem-contractSkill
- hold-demand-ledgerSkill
- hosted-carry-lendingSkill
- inert-op-screen-to-structural-ruleSkill
- invariant-chartSkill
- jev-second-perspectiveSkill
- lambda-exchange-deskSkill
- leader-rebase-stackSkill
- lean-arklib-proofsSkill
- lean-finite-certificate-splittingSkill
- lever-switches-and-break-controlsSkill
- mission-commandSkill
- nasqret-playbookSkill
- paged-walk-descentSkill
- peak-lifetime-ledgerSkill
- pinned-table-stackingSkill
- pip-disciplineSkill
- portable-source-assertionsSkill
- proximity-prize-campaignSkill
- proximity-prize-frontier-archaeologySkill
- quantum-affine-lookup-synthesisSkill
- quantum-borrowed-wire-counterexampleSkill
- quantum-dense-qromSkill
- quantum-dirty-borrowingSkill
- quantum-divstep-reconstructionSkill
- quantum-encoded-arithmeticSkill
- quantum-graded-divider-placementSkill
- quantum-guarded-permutation-auditSkill
- quantum-mbundo-lifetimesSkill
- quantum-record-miningSkill
- quantum-replay-arithmeticSkill
- quantum-residue-scan-ambiguitySkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 6 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 16
Use when a judge run takes more than a few minutes or more memory than the workstation has, when many candidates must be scored per day, or …
- Distinctive: line 18
Leave it when the judge already runs in seconds.
- Distinctive: line 19
Leave it when the judge's source is not public.
- Distinctive: line 20
Never use a replica number as the official one.
- Very specific: line 30
Write the determinism map: Read the evaluator end to end.
- Distinctive: line 37
Choose the unit of parallelism from the map: When offsets do not depend on execution, units are independent: one producer generates the …
- Distinctive: line 50
Add what the judge cannot give: totals even when samples fail, a per-sample failure count, the index of the first failing sample.
- Very specific: line 84
All are Python 3 standard library.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Paths such as scripts/..., references/... and assets/... are relative to this skill's folder.
Doppelgängerli (exact-evaluator-replica)
A replica is a second implementation of the official evaluator (the judge) that prints the same numbers for
the same artefact, faster or in less memory. It is exact or it is nothing: a replica that is "close" is a proxy and
belongs to calibrated-static-cost-proxy. Every other loop in a campaign runs on this tool, so it is built first.
Use it / leave it
- Use when a judge run takes more than a few minutes or more memory than the workstation has, when many candidates must be scored per day, or when a failure-rate estimate needs fresh samples.
- Leave it when the judge already runs in seconds.
- Leave it when the judge's source is not public. Then only a calibrated proxy is possible.
- Never use a replica number as the official one. The submission is scored by the official runner.
Inputs
- The evaluator's source at a pinned commit.
- At least one artefact with an official graded log (the passing path).
- A machine that can run the official evaluator on small artefacts.
Procedure
- Write the determinism map. Read the evaluator end to end. For every random draw record its source, its order,
and whether the amount drawn depends on what executed. Fill
assets/determinism-map.md. Worked example: in the source benchmark all randomness is one SHAKE256 stream seeded by a hash of the artefact; the first 64 x 9,024 bytes make the inputs; every measuring op then reads 8 bytes per 64-sample batch whether or not its condition lets it execute, so batch b's bytes sit at offset 649024 + 8M*b (M = measuring ops). Done when you can state, for any unit of work, the exact offset of its randomness. - Choose the unit of parallelism from the map. When offsets do not depend on execution, units are independent: one producer generates the stream in order and hands each unit its slice; workers simulate units. Bound the queue (depth 2 in the example) and recycle buffers, since the producer is the bottleneck and extra workers only add memory. When offsets depend on execution, stop: an exact parallel replica is not available. Report that finding and fall back to a serial replica with a compact representation, or to a proxy. Done when the unit and its offset formula are written in the map.
- Compact the representation. Pack each record to the minimum the semantics need (20 bytes against the judge's 56 in the example). Reserve index 0 of the condition table as an all-ones "no condition" word so the hot loop has no branch for it. Keep the judge's loader format, magic bytes and size limits unchanged. Done when the replica loads every artefact the judge loads and rejects every artefact the judge rejects.
- Print the judge's own lines. Same labels, same rounding, same score formula, so outputs compare with
diff. Add what the judge cannot give: totals even when samples fail, a per-sample failure count, the index of the first failing sample. Done when a passing artefact's replica output contains every judge line verbatim. - Write a random-artefact generator. It must cover every op kind, every condition form, nesting, and the
malformed-but-accepted cases (closing a block at depth 0, measuring ops inside conditions). Start from
scripts/gensynth.py: give it a JSON spec of the judge's op kinds and replaceencode()for a binary format. Done when--require-coverageexits 0. - Differential test. For at least three seeds, run judge and replica on the same random artefact and compare
every count and every average. Random artefacts fail every sample, so read the judge's averages from wherever it
records them on failure (a results file it appends to, in the example) and restore that file afterwards:
The label and the column are your judge's own. For the bundled toy they arepython3 scripts/difftest.py --gen '<gen> {out} {n} {seed}' --judge '<judge> {artefact}' \ --replica '<replica> {artefact}' --field 'tested samples' --field 'wrong outputs' \ --side-file results.tsv --side-col '3=avg executed cost' --n 200000 11 22 33'1=avg executed charged'. Done when it printsall seeds match. This covers semantics, accounting and stream alignment only. - Golden test on the passing path. Reproduce at least one officially graded log to the last printed digit. Random artefacts never reach the passing path, so step 6 cannot stand in for this. Done when one graded log matches digit for digit with the code you will actually use.
- Add a salt switch. An option that mixes a string into the seed gives a fresh, independent sample set for the same artefact. Print on every salted run that the score is not official. Failure rate = mean failing samples over several salts; standard error = sd / sqrt(runs). Done when two salts give different inputs and an unsalted run still matches the judge.
- Pin and re-gate. Record the evaluator commit and the list of files whose change invalidates the replica (evaluator, simulator, input derivation, dependency manifest). Re-run steps 6 and 7 when any of them changes. Done when the pin and the file list are written next to the replica.
Bundled scripts
All are Python 3 standard library. Each has --selftest.
| script | purpose |
scripts/difftest.py | Differential test driver: generator, judge and replica commands in, labelled fields compared, side file snapshotted and restored, non-zero exit on any mismatch. |
scripts/gensynth.py | Spec-driven random artefact generator with a coverage report. |
scripts/toy_bench.py | A miniature benchmark: a scalar judge reading randomness from a cursor, and a bit-sliced parallel replica reading it by offset. Read it once as the worked example of steps 1 to 8. Its --bug 1 and --bug 2 switches inject the two classic defects (stream misalignment, wrong charge condition). |
assets/determinism-map.md | Template for step 1. |
The toy uses a text format and 200 samples. For a real judge, write the replica in the judge's language against its
own crates or modules, and keep difftest.py and the generator as the gate.
Checks that say it worked
all seeds matchon at least three seeds.- One graded log reproduced digit for digit.
- Replica and judge agree on the first failing sample of a known-bad artefact.
- An injected defect in the replica makes the differential test fail (the gate has teeth).
- Memory and wall time written down next to the judge's (example: 1.24e9 ops in about 3 minutes and 33 GB, against about 69 GB for the judge).
Failure modes
- A passing-path bug hidden by random artefacts. They always fail. Never skip step 7.
- A dependency swap changes the stream. The source campaign's first version used a different SHAKE implementation behind a self-test against the reference one. Keep such a self-test whenever a primitive is swapped.
- Replica numbers reported as official. Label every figure with the tool, salted or unsalted, and the pin.
- The judge changes silently. Step 9 is the guard.
- An inherited golden test. In the source campaign only one of four golden rows was re-checked with the published code. After any refactor, re-run the golden test.
- A build of the artefact that still needs the judge's memory. The replica fixes evaluation only; check whether the generator can stream its output.
Transfer
- Gas races: a local meter differential-tested against chain receipts.
- Provers: a cell or memory counter tested against the official run.
- Proof-checking benchmarks: a local replay of the verifier's policy checks
(see
evaluator-scoring-surface-auditfor a bound mismatch found that way).
Evidence this pays
Citation key: N:<id8>:<line> is a public submission note of the source campaign (solver BitWonka, ecdsa.fail)
by id prefix; T: is a file in the public ecdsafail-tools repository. Numbers are quoted from those sources; none
was re-measured for this skill.
- Published 11 Oct 2026 (T: commit 9f92470). Every low-qubit row after it is scored with it (N:5fa63407:125, N:2d167363:12).
- The official run it replaced "thrashed in 72 GB... stopped after two hours" (N:ddd607c9:181).
- Four graded logs matched to the last digit (T:fasteval/README.md:68-75); one of the four with the published code (README:77).
- Five other solvers name the tool in their notes within five weeks. These are mentions, not verified independent use.
- Mechanism sources: T:fasteval/README.md:14-24 (stream layout), T:fasteval/fasteval.rs:63-66 (compact op), :414-470 (producer and workers), :154-157 (salt), T:fasteval/gensynth.rs:1-8, T:fasteval/validate.sh:23-58. The bundled scripts are independent generic implementations; no code is copied from that repository.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

