agentleFS
glslangglslang/windbg-mcp/.claude/skills/eval-bench/SKILL.md

eval-bench

Skill21 starsChanged 11 days agoMITView on GitHub

Run and read the local-model benchmark (`tools/local_model_eval.py`) against this server - the grid's fences, what the grader's matching rules are for, `--verify-key`, `--compare`/`--series`, and how a moved aggregate gets misread as a controlled result. Use when running the eval, adding or re-grading a task, or writing up a benchmark result.

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Every record carries server (the build that answered), model_digest (the weights behind a mutable ollama tag), suite, and harness_version …

    line 120
  2. Distinctive:

    Check which half of the bench a prose change can reach before predicting what it will do.

    line 289

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Benchmarking a model against this server

The grid, and what bites while running it

docs/local-model-eval.md is the result; this is what bites while running it again. The grid is four scripts — the ollama driver, the Claude Code driver, the Foundation Models driver, and the matrix runner that spawns one of the three per cell and grades the log afterwards.

The fm backend varies none of the grid's three axes, and that is its result rather than a gap. Apple's on-device model has one model with no tag to choose, a window fixed at whatever the OS enforces, and reasoning: false — so a plan naming another model, asking for a context, or setting think is refused at validation rather than run. Each of those would otherwise key a cell against a record that carries something else (num_ctx: null, one fixed model name), and a cell nothing can match re-runs on every invocation while the plan goes on claiming the axis. Only --tools crash fits the window at all, so the surface axis collapses to one cell too. Read its row on its own and never fold it into an aggregate with the ollama cells; docs/apple-foundation-models.md is the write-up and carries the measurements.

Record what the runtime served, not what you asked for. num_ctx on a request does not shrink an instance ollama already holds: with a 32,768 instance loaded, cells asking for 8,192 are served 32,768 and look perfectly healthy — a 17,300-token prompt "fitting" in 8k, which is the result the context axis exists to find and would have been fiction. /api/ps is the only place the truth appears. Every record carries served_context, the grader marks a cell where the two disagree with ?, and the runner evicts the model between windows. The first run of the grid recorded five such cells; they were dropped and re-run.

The grader's three matching rules each came from a real wrong verdict, and all three are in present():

  • A number matches only between hex boundaries — 0x22 is the device type ioctl_decode asks for and 0x22200B is the code in the question, so plain containment passed for any answer that repeated the question. One model scored correct while saying FILE_DEVICE_KEYBOARD.
  • Leading zeros are formatting — the tool prints 0x802 and a model writing 0x0802 agrees with it. That one marked a correct control answer wrong.
  • A separator between hex digits is formatting too. WinDbg writes fffff801`3c65bca8; Opus writes 0xfffff801_3c65bca8. Both name the address the key holds.

Two of those three were found by reading the control's answers, which is the argument for having a frontier row at all.

possible_on in the task file is a prediction, and predictions about this server are wrong in one direction: too pessimistic. Facts here are reachable by more than one route — open_dump's summary carries the bug check and the module count, crash_triage's frame 0 is the pc that registers reports — so a task the tool table says needs inspect may be answerable with crash alone. Verify against the dump before scoring a model wrong for finding the other route; the arm64_pc entry was corrected mid-run for exactly this.

Resume is per cell, so the log legitimately holds a task twice. An outstanding task re-runs its cell's whole list, and the grader keeps the last record per (cell, draw, task). Do not "fix" a duplicated task id by deleting rows.

One draw per cell is what the grid runs, and it cannot answer "X caused Y" — a rule docs/local-model-eval.md states in as many words and that has not stopped three write-ups (#209 once, #212 twice) from reading a moved aggregate as a controlled result (item 42). The trap is composition: a total that holds across two runs whose callers changed is a coincidence, not a rate. What answers a rate is draws: n on a cell group — the same cell n times, one thing varied — and the draw index is inside the grader's key, so repeats accumulate instead of the last one winning. Two consequences when editing that code. A record with no draw is draw 1, which is the only reason a run recorded before the change still grades to what it graded to, and it lives in draw_of rather than in four or 1s. And --matrix prints a distribution (3Y2n) only when a cell was repeated; a single draw prints the bare mark, so nothing already published is restated in a new notation.

And the seed does not replay a draw on this bench, so do not write that it does. Each draw asks for seed: <draw index> and records it, which is right where a seed reproduces a sample (the draws become repeatable, and arm A's draw 3 pairs with arm B's). It does not here: four identical requests to qwen3.8:27b-mlx under seed: 7 returned four different answers (ollama 0.32.15, MLX, measured 2026-08-24 — after the comment claiming otherwise had already been written). The column is what was asked for; the distribution over draws is the measurement.

The Claude Code row needs four fences or it measures something else. --strict-mcp-config (or it falls back to the editor's registered windbg-vm, whose credential is a different client and gets the whole 51-tool surface — the surface axis then measures nothing); --disallowedTools for the built-ins (or it answers a question about a sample dump by grepping this repository); ENABLE_TOOL_SEARCH=false (or MCP tool schemas are deferred and fetched with ToolSearch, so the surface costs that row almost nothing while every other row pays in full); and a neutral working directory, since Claude Code reads the project it is started in. Its prompt-token column is still not comparable with the ollama rows — its own system prompt is most of it — and the document says so rather than quoting it.

A model "inventing" tools on a narrowed surface is almost always this server advertising them, and there were two channels; both are narrowed now. The instructions string went per-client with item 40, and a tool's description with item 41 - the descriptions of tools a crash client is served used to name modules (open_dump), debug_batch (interrupt, end_session), backtrace (crash_triage) and go (interrupt), five references on an eleven-tool surface. Re-running the five min cells against item 40's fix (2026-08-24) moved unserved calls 17 -> 14, and 13 of the remainder named exactly those five, which is what item 41 then removed. So the metric is still unserved rather than "hallucinated", and it still measures the server before it measures the model. One call in 61 was a genuine invention, which is the floor.

It is now two columns, taught+wanted (item 43), split by the task's own possible_on: a reach off the surface on a task this surface can answer against one on a task it cannot. Summed they hide each other, which is the whole argument — re-graded, item 41's fix is 4+10 -> 0+6, an elimination of the half it was aimed at rather than a 57% improvement in a total. --grade --assert-no-taught exits non-zero on a taught call and wanted is deliberately not assertable. The split attributes need, not provenance: an opener's result taught modules on unloaded_driver until #217, and that task is one min cannot answer, so those calls are wanted. Lower bound on advertising, upper bound on need.

Fourth run, five draws of the five min cells (2026-08-25, against #217): modules went from 3 of 5 cell-draws to 0 of 25, and what remains is mostly invented (execute_command, run_command). The arms differ by exactly one server behaviour these tasks reach - the opener's summary; the other two paths #217 changed were measured as unexercised (0 user-mode refusals in 95 crash_triage calls, 0 post-commit failures) - so this is the cleanest causal read the bench has produced — with the weakness on the other side, which is one draw per cell. Two things only draws could say: ioctl_decode is answered from a frontier model's own knowledge 5/5 and from a local one's 1/5, and arm64_pc is 5n in every row, which is what sent someone to look at the task rather than the models. It went unanswered in all 35 runs in the logs still on disk, and has been answered right once ever — qwen, in the original grid, reasoning that frame 0 is the pc (item 44). The key is the literal pc, nt!KeBugCheck2+0x2e8; almost every model gives the bug check's parameter 1 instead — the address whose execution faulted. A task that fails everywhere is a task to read, not a model to blame.

A run records what it ran against, and that is what makes two of them comparable (item 46). Every record carries server (the build that answered), model_digest (the weights behind a mutable ollama tag), suite, and harness_version for the Claude rows, which can have no digest (nor can the fm rows: Apple's weights have no address to record, so they carry model_digest: null and name the OS build instead) - opus and sonnet are aliases resolved inside a client this bench does not own. --compare reads two logs with two rules that are not one rule: a changed question blocks a pairing (via stale_prompt, printed at the row), while a changed build, model or window is named above the table for a reader to weigh - conflating them is how a moved aggregate gets read as a controlled result. --series reduces logs to one row per run in docs/eval-runs.json. Six things bite. The server's version now carries its git revision (0.11.0+g1a2b3c4), stamped by build.rs, so anything asserting on it is a prefix check - and the smoke test additionally asserts the revision is there when built from a checkout, which is the assertion that catches a build.rs that stopped running. build.rs's watch list and its dirty check are one INPUTS const, because emitting any rerun-if-changed replaces Cargo's default of watching the whole package, and two lists would disagree about what a clean build is; -dirty.<digest> therefore means "the build inputs differ from that commit", not "the tree is dirty", and the digest is over the diff so two uncommitted iterations on one HEAD are two identities. Git paths go through rev-parse --git-path (and the branch ref through symbolic-ref, since --git-path takes a path relative to the git dir and not a revision): a git worktree checkout has a .git file, so a literal .git/HEAD is a watched path that does not exist, which Cargo reads as always-changed and recompiles the crate on every no-op build. And the two /api/ps facts are one call (runtime_identity): asking twice could catch different instances and pair one model's window with another's digest. The surface and the window are per-cell facts; the weights and the build are deliberately not - review asked for both and both were built and then removed, because reaching the state they guard needs a model re-pulled mid-run or a run spanning a rebuild, and neither happens here. tools/ is a developer script and the bar for defending it against states its own workflow cannot produce is lower than src/'s. A surface is compared by digest, not byte length - a same-length reword or an equal-sized allowlist swap moves neither the count nor the length, though a comparison spanning the rollout falls back to what both sides recorded and says unverifiable rather than moved - and unrecorded (nobody recorded it) is kept apart from unavailable (this row has no such answer), or every run with a Claude cell reads as a legacy log.

An identity field is a backend's statement, never the grader's inference (item 80). Each driver writes an identity block answering every name in local_model_eval.IDENTITY_FIELDS - weights, reasoning, harness - with None where the row has no answer, and identity() reads that one key. Adding a field means editing all three drivers; a name added to the tuple with no driver opinion fails tools/test_local_model_eval.py (python3 -m unittest discover -s tools -p 'test_*.py', which CI does not run), and so does a grader that defaults one instead of saying so. The temptation is to read the nulls the drivers already write, and it does not work: model_digest is null on two backends deliberately, but think is false on an fm row and absent on a Claude one - one value, one omission, neither null - and those two are precisely the cases the removed dispatch existed for. The one place it survives is legacy_identity(), for logs written before the block, and that is frozen: a record without the key predates it, so nothing new ever belongs there. Two things this cost that are worth knowing before reading an old write-up. Re-grading moved no score and three identity lines, all of them overclaims - harness <version> on a mixed run was the Claude rows' alone, reasoning unrecorded on a pre-axis log is unavailable, unrecorded, and the reasoning A/B's two arms differ in composition in a way only arm B's new harness unavailable line shows. And docs/eval-runs.json is two --series invocations concatenated, not one over eval-out/*.jsonl: the older three logs are graded against eval_tasks_v1.json and the newer three against eval_tasks.json, so a single run over the glob silently re-grades half of them against questions they were never asked.

The key is a snapshot, and --verify-key is what re-takes it (item 45). The six tasks are graded against facts read off the checked-in dumps with this server's own tools, so a fact that stops being what the server reports leaves the suite grading and every model scoring against nothing — a key that has rotted looks exactly like a model that got worse. Each task carries a verify binding of (tool, args) steps to the values expected back, and WINDBG_MCP_TOKEN=<full surface> python3 tools/local_model_eval.py --verify-key re-reads the lot. Five things bite when touching it. It is a command, not a CI gate, and that is the decision rather than an omission: the oracle is present(), so a Rust test would need a second copy of three rules each learned from a wrong verdict — run it after a dbgscope bump, a symbol-path change or a new sample. The binding grounds expect, it does not generate it: two of unloaded_driver's three groups are phrasings of a relation (matched: 0 is what "not loaded" means), so a run reports each group as value, relation or skipped. Which are relational is declared (states), never inferred — reading "no pinned value matched" as a relation let a group edited to a value the tools do not answer pass, which is a broken key reached through the mode meant to catch one — so a group grounds claims but nothing renders is a failure, as is a group no step claims, and a grounds group is checked alternative by alternative (each must render or be a spelling of one that does), since appending an alternative widens what the grader accepts without moving anything a whole-group check would see. The gate is asked per dump, not once of the host: docs/smoke-test.md records an engine failing differently per dump, so a host-wide gate stands the ARM64 step down over a missing x64 PDB — and a failed probe is a task failure, not a closed gate, because a closed gate stands steps down and passes; so are a modules answer with no module list and a target with no nt, as is a gated step with no target to probe. Nothing there reads a structured answer with a default - one probe helper answers a value or a reason, which is what five review rounds finding the same shape in six places bought - the last of them a renamed symbols on the nt record, which reads as None and is not a PDB-backed state, and a symbols that is no longer a string, and a task whose every grounding step stands down at a gate is INCOMPLETE and non-zero rather than OK - driver_blame has no ungated route where arm64_pc does. Pins compare types too (False == 0 and 227.0 == 227 are true in Python, and one of those would have hidden a schema change). A states group names the pins its relation rests on, so the exemption cannot outlive the fact underneath it. A pin can be too tight: the pc fact was first pinned as registers.32.value, a position in the ARM64 bank that an engine may reorder without the key having rotted, so a read path enters a list by name (registers.name=pc.value) as well as by index. And tools/eval_tasks_v1.json carries no binding on purpose — it is the wording published logs were graded against — so the mode refuses it by name rather than skipping it.

A scenario is graded with --scenario, and everything else refuses its log (item 39). A suite carrying facts (tools/eval_scenario.json) is run as one conversation per cell, and the grade is where the run broke rather than a score per step. What bites:

  • An expect alternative is matched against the normalised answer, and normalise() strips a _ between two hex digits. So DRIVER_POWER_STATE_FAILURE arrives as driver_power_statefailure (E_F), and the natural alternative would mark every correct answer wrong. That applies to the main suite's keys too. Scanned 2026-10-17, one alternative there is dead this way: arm64_pc's backticked fffff801`3c65bca8. It costs nothing only because the bare fffff8013c65bca8 sits in the same group. test_the_shipped_scenario_agrees_with_its_own_bindings caught it offline, by grounding each fact against its own pins with no server. Run it after editing a fact.
  • A fact is searched for from its own step onward. 0x9F_3_ACPI_IMAGE_pci.sys at step 1 contains acpi, step 7's answer.
  • A fact is credited by its value, not by the claim around it. On the first v2 run, qwen called Arg3 the IRP at step 2 and listed Arg4's address beside it as "the related device object". The step was credited, and the break was reported at step 3. Its two late facts were the same thing: nvlddmkm named as the largest image, and ACPI quoted inside that failure bucket. So read the answers before believing the broke at column or a late.
  • The conversation id, not the task, is the unit. Two attempts deduplicated by task splice into a transcript no model had. The runner mints the id and writes it into its kill notes, which is why claude_code_drive.py, with no transcript to carry, is refused for a scenario plan.
  • A later fact's binding may send what an earlier fact established: step 5 asks device_object about step 4's address. So --verify-key holds each binding to the prompts so far plus the earlier facts' alternatives, and a wrong earlier fact fails the later binding too. That is the chain showing up in the key's own check, not a second bug.
  • An emptied key is still a scenario. is_scenario reads whether facts is present, so "facts": [] is refused by scenario_key by name. Read as "not a scenario" it ran the steps as a task list, and a step has no expect, so every answer passed (Codex on #504).
  • A fact that is a rank is pinned through an order, not a position. modules.-size.1.name is the second-largest image and modules.+start.152.name the image after index 151 by start address, whatever order the server lists in: the segment re-sorts by one numeric field, reading a 0x string as a number. Pin the values on either side of the rank too, because a tie keeps the server's order and passes a name pin; and pin the last index (226), because matched counts the rows before limit cut them and only an index that exists says the sort saw them all.
  • The window a scenario fills is the model's own, and the MLX runtime does not enforce a smaller one. Both bench models are 262,144 (ollama show). A cell /api/ps reported at 32,768 evaluated a 79,352-token prompt and answered from text 49,000 tokens back, so on ollama 0.40.6 a window axis is the same condition run twice. Compare a transcript with the model's own window: none of the v2 runs got within half of it. "2.4 times the window" was written against the served figure, and the user caught it.

think takes a level as well as a switch, and only a level the model declares (2026-10-18). The user's call: reasoning off is no longer a realistic eval, so a run asks for each model's top level. /api/show's thinking.values lists them, and they differ per model: qwen3.8:27b-mlx has low, medium and xhigh, with true meaning medium and no high, and gemma4:31b-mlx has only true. ollama 0.40.6 accepted think: "high" for qwen without an error, so the driver refuses an undeclared level (refuse_an_undeclared_level) rather than recording a level nobody applied. A level is its own arm (arm_named): xhigh records as xhigh, so it cannot share a log with on (one_arm_per_log), and it reaches the driver through think_env, which has to round-trip with think_of. So a plan spells it as the driver records it: "XHigh" or "true" would key an arm no record carries, and one_arm_per_log refuses both. What a level buys is unmeasured, so do not write that it reasons more. On one counting question, three draws a level, qwen's reasoning text did not order by level (xhigh gave 1,326 once and 7,268 once; low gave 10,078 once).

Re-run against item 41 (2026-08-24): unserved calls 14 -> 6, with debug_batch going 10 -> 0 and every name this server was teaching now gone. What that re-run mostly taught is how easy it is to over-read at n=1, in a file that already says so. Two claims had to be pulled back after review, and both had passed my own reading first. The three modules calls did not move even though open_dump no longer names it - which shows the description is not necessary, and not that it caused none of the earlier three: the callers changed (nemotron/Opus/Sonnet, then qwen/gemma/Opus), so an aggregate holding at three is a coincidence of composition. And "unserved on answerable tasks went 4 -> 0" was a double count of our own - all four were gemma's debug_batch, so it is the first row again, not independent evidence. What survives is the shape: every survivor is a direct reach for a module listing on unloaded_driver, the one task whose answer lives in a tool a crash client is not served.

A description reaches every row; the instructions reach two. tools/list is read by all five rows, so item 41 moved every prompt (-223 tokens on each ollama row, -307 on each Claude one) where item 40 could not move a local one by a token. Check which channel a prose change travels on before predicting which columns it can touch.

And the ollama rows never read the instructions at all, which is what nearly made that fix look bigger than it is. tools/local_model_drive.py's handshake keeps the negotiated protocol version and discards the rest of the initialize result, so a local row's prompt is the one-sentence system prompt plus tools/list: narrowing that string cannot move those prompt-token columns by one token, and it did not. Claude Code injects a server's instructions into its own system prompt, so the two control rows are the only ones any instructions measurement is about. Check which half of the bench a prose change can reach before predicting what it will do.

The bench listener is the shipped per-client feature in anger. tools/bench_listener.ps1 serves full, lean and min from one foreground process, tokens arriving on stdin; its startup line naming the three clients and their surfaces is the check that the run is measuring what it thinks. A cell changes the bearer token and nothing else.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.