Working in this repository

CONTRIBUTING.md owns the process; this file is the short form a session reads first. Where the two disagree, CONTRIBUTING wins and this file is fixed.

What to do, in which order

The order of work lives outside the tree: the maintainer keeps a plan under docs/plans/ (git-ignored) and a memory note that holds the current ordered steps. Read those before choosing a task; do not infer the order from the git log. A session covers the tasks its plan assigns and stops at the boundary the plan names, even with context to spare; at the stop it commits, rewrites the memory note's current steps in place, hands over any push, and says whether a fresh session can continue from the files alone.

Commits

• One workstream per commit; a change that would be reverted separately is a separate commit. Subject is the outcome, body is the mechanism; no session narration, no attribution trailers.
• A backlog entry is committed by the session that writes it, in a commit of its own, as soon as it is written — the queue line in BACKLOG.md, or the move that files one under backlog/. The commit body carries the observation behind the entry, which the one-sentence entry cannot; and an uncommitted entry is a stray change every later session has to explain or step around.
• A skill, agent or brief edit that changes behaviour is tested before it lands (CONTRIBUTING, "Testing a wording change"): a baseline that fails, one wording, five fresh sessions per arm - on each of two models, twenty in all, where the rule is one of restraint, since that baseline is taken on the strongest model too - every grader of the case in both arms, the result written into the case's results/manual-run.md. A wording that costs a neighbouring grader is declined with its reason, not reworded a third time on the session's own judgement.
• The version bump goes in the commit that changes behaviour; a bumped manifest releases on push.

Backlog

• A new entry starts with a search, not with a line. grep -rin the mechanism — two or three words of what goes wrong, not the words an entry would have used — over all three files, BACKLOG.md and backlog/, because the queue is in the first and everything that has left it is in the other two, and a search reaching only the queue writes a new line for a mechanism already filed. A hit in the queue takes a sighting in that entry's parenthetical instead of a second line; a hit in backlog/parked.md moves that entry back to its tier with the new circumstance, already carrying two; a hit in backlog/declined.md is the answer to the line rather than a reason to write it. Only a session that hit the mechanism itself adds a sighting; an entry carrying no count stands at the one sighting that queued it, so the first sighting added makes two.
• Recurrence is the promotion filter: two sightings — other work or another place, never the same failure met twice in one pass — or one sighting whose failure was measured - a rate, a run, or a failure that landed and whose cost the entry names, never a near miss caught before it cost anything - is what makes an entry worth a wording round, and an entry that has reached neither by the time a round reads it is parked into backlog/parked.md, keeping its sentence and naming what a second sighting would be, which is a finding about the mechanism rather than a decline. BACKLOG.md's header carries the shape, CONTRIBUTING's "Where a lesson waits" the reasoning.

Review

Most changes here are verified by something stronger than a reading before they land: wording-tested contracts, grader patterns tested on captured runs and scored by the runner, fixtures built under a hostile global git config. Do not re-review those. Review the prose that loads into sessions — hooks/session-brief.md, skills/*/SKILL.md and the references they point at, agents/*.md, CONTRIBUTING.md, README.md, and CLAUDE.md itself, which loads automatically into every session here and is therefore the least reviewed and the most obeyed — as one doc-vs-code pass over its diff: each changed sentence against the file it cites and its neighbouring copies (brief core against skill, agent contract against owning skill, a reference against the SKILL.md clause that points at it, README paraphrase against skill, this file's short form against the CONTRIBUTING section it compresses). Run records and the backlog's entries carry no behaviour and are not reviewed — but the headers of BACKLOG.md and the two files under backlog/ are instruction prose a session obeys at intake, and the range below leaves all three out, so those headers are read in the same pass from a diff of their own (git diff prose-passed..HEAD -- BACKLOG.md backlog). Shell under scripts/ and evals/fixtures/, and any eval case never run in both arms, get an ordinary review.

The range is the watermark, not the last push: git diff prose-passed..HEAD -- hooks skills agents CLAUDE.md CONTRIBUTING.md README.md, where prose-passed is a lightweight local tag marking the commit through which this prose has actually been read. The pass is its own unit of work, never started inside a task; it is due at about 150 changed lines, at a push whatever the size, or after two task units that changed a path in that range or one of the backlog's three headers; and it closes by moving that tag to the head it read, which is its only durable output (CONTRIBUTING, "Before pushing", has the mechanics and why a push is the wrong anchor).

Findings files (review-findings-*.md) are point-in-time and git-ignored; delete one once everything in it is closed or carried into the plan.

Gates

The eight local gates are the steps of .github/workflows/guard.yml minus the skills-spec validator, which is not installed locally; run them at the head before handing over a push. The local shellcheck is newer than CI's, so a clean local lint does not prove CI's lint step.

Evals

• evals/results/ is git-ignored and holds the baseline JSON of each release; the runner keeps each run's findings file or reply as grader evidence in that JSON, which is what to hand-grade when a judge verdict looks wrong. The judge is unreliable on long files, on findings renumbered at a merge, and on a case whose graders are many tone rubrics over one document — there it over-fails both arms and unevenly, so the difference between them is no safer to read than the rate, and a row is hand-graded in both arms before any decision rests on it.
• scripts/eval-trend.py folds those kept runs into one row per case, grader and release, which is the whole key a wording round's two arms share: copy a round in as <round>@baseline.json and <round>@<wording>.json — the only thing that keeps a declined wording out of the release's own trend — and read a ! on a row as one version folded from two plugin checkouts, which the fold cannot tell apart.
• A new grader is mechanical — regex, file_exists, tool_used — unless its condition is irreducibly semantic; but a pattern anchored in a string the fixture seeded scores whether the run edited the draft rather than whether it did what the rule asks, and passes in both arms once that wording is softened. Anchor it in what the run must produce, and leave a rubric about what the run writes instead, and where, to a judge. Calibrate any grader on reps from both sides of its line - a set that all passes can show a good rep failed, never a bad one passed, and one that all fails the reverse, and the gap is widest on a pattern that must not match - using a smaller model's reps, a baseline arm's, or artifacts written just across it, and where no rep from the other side exists, say beside the rate which of its errors is untested.
• A git act inside the eval sandbox runs on the git the scaffold puts there: every case's fixture.sh calls evals/fixtures/stage-git.sh, which writes a wrapper into the run's own home and a .zshenv putting it first, and a new case is not finished until its scaffold carries that line, whether or not its graded act is a git command. The sandbox's own git is a stub that dies writing its cache, and the working binary loses the PATH search from any position, being a symbolic link into a tree the sandbox will not let the shell stat - so prepending its directory changes nothing and the failure reads as a PATH order problem it is not. Without the line the split is silent and lands as a grader rate: seven reps of ten found the absolute path in the round that measured it while three escalated the act as blocked, which fails the act's grader and any grader reading the list it belongs on - unusable for that round rather than failed - and ten of ten performed it on the re-take. Settle a question about a run's environment from what a run keeps: --keep-temp leaves the run root, whose config/settings.json is the sandbox policy and whose trace holds every command a rep ran, and a case file's execution.env takes EVAL_* keys only, everything else coming from the operator's shell or the scaffold. A case's allowed_tools is neither the tool set nor the grant: the runner runs each rep in dontAsk mode, where a call that would have to ask is refused, allows without asking every tool the invocation's --allow-tools grants (writes scoped to the workspace and the run's temp directory) plus the list's entries from its read-only set (Skill and Agent among it), and withholds every tool that is neither granted nor in that set, bar a few it never withholds (ToolSearch and, beside a granted Edit or Write, NotebookEdit among them), whatever the list says. Every kept run a run JSON pointed to when this was written has an init event listing tools its case never named, every refusal in those traces is of a tool the case lists, and on the mutation case, whose list leaves Skill out, the Skill tool's own check let every call to it through. So read which tools a run had from its init event and whether a call could run from that call's result, never from the case file, and not from the call's presence either, since the runner counts a refused call as a call.
• A file-targeted grader whose target is absent throws and is scored a fail, not_contains included; a behaviour that legitimately deletes a file cannot be graded through that file. focus and target take one literal existing file — no glob, no directory — so a grader reads only a path fixed before the run: the case prompt names it, or the grader reads the index that points at it. An llm grader with no focus is shown the run's final message alone, and a file_exists grader reads the run's own file changes rather than the tree (CONTRIBUTING, "Testing a wording change", lists what each grader type may see).
• A grader file's frontmatter, and a prompt.md's, ends at the first --- after its opening line, mid-line, quoted or in a comment. A cut that leaves invalid YAML or drops a key the body cannot stand in for, such as type, refuses the case at load for nothing; any other loads without a warning, the value shortened at the dashes and every later key at its default - a lost match reads contains, so a pattern cut to a bare anchor passes on every target and a not_contains grader whose pattern survived is inverted - and a grader that lost its pattern or criteria takes the rest of the file instead. Spell three dashes in a pattern -{3} inside a quoted value, which a missed run then leaves open and the case refused, or put the pattern alone in the body, which is read raw. A file whose first line is not --- has no frontmatter: a grader file is dropped without a warning, or, as the case's only grader, leaves it refused with graders: Required, and a prompt.md runs on the default for every key its frontmatter held, with that frontmatter in the prompt. Check a file free before a launch with the runner's expression, /^---\s*\n([\s\S]*?)---\s*\n?/, and Bun's YAML parse. A regex grader gets no flag it does not declare but the g a count:N match adds, so unless it declares m, ^ and $ anchor the whole target; a file target loses a leading byte-order mark first, and input_match takes no flags (CONTRIBUTING, "Testing a wording change").
• A tool_used grader's input_match is tested against the call's whole input JSON — Bash's prose description and any comment the command carries included — so a pattern on a bare identifier scores a call that changed something else as the act, and a heredoc or a Write carrying the whole file reads an untouched line as the act applied. Anchor on the transformation: the replaced text together with the line that replaces it, or a whole body written without the part the act removes. The tools it does not name are a blind spot its arms need not share — the rule under test chooses the tool, and a wording asking for every row one at a time moved six reps of ten onto Edit, where the Bash-anchored grader read 4/10 against a hand-read 10/10 — so keep a companion grader per tool, read those rates across the arms, and hand-read the draw of every rep one of them fails, which needs --keep-temp because the runner deletes each run's workspace. The runner also counts a refused call - it flags one a hook or the run's permission settings refused but reads the flag only for mock tallies - so a grader on an act either can stop (the skill gate's commit, PR create, memory write or dispatch, or any call the run's policy denies) reads the stopped attempt as the act: anchor it in what the act leaves behind, such as the log the commits landed in.
• A tool_used: Skill grader marked arm: with-only is the trigger's own indicator: scored in neither arm of a two-arm run and never evaluated in the arm without the plugin - under --ablation none, which is how a round's arms run, scored like any other - and the only thing that separates "the skill never loaded" from "its rules did not hold" - except on a replay fixture that seeds the load, where its 0/N reads only the absence of a second load by the rep: that a resume passes the seeded body is a probe's to show, and which body an arm held, a phrase only that arm's regenerated history.jsonl carries. A skill whose condition is a standing rule rather than a named task is never reached through that tool by being named - 59/59 on the suite's model on the cases whose prompt names work the skill owns against 0/13 on the two whose skill fires on how a message ends, though the brief named that skill and its trigger - and the brief reaching the model is a different event that does not imply it. A moment the session acts on does reach it: the brief's first-tool-call line loads tracking-open-asks first in a fresh conversation (5/5 under an earlier wording with the same imperative, 23/23 under the landed one, all on the suite's model) and loaded it in no resumed one (0/5, that earlier wording), and the skill gate stops a commit, a PR create, a memory write or a dispatch once until its skill loads - every kept rep it stopped loaded that skill with its first call after the stop, though the retry passes either way - while no case yet reaches its memory stop: it matches a run's own memory under config/projects/, where no kept rep wrote (they only tried to list or read it, every such call refused by the runner's deny rules, which refuse a Write or Edit there before any hook runs and a shell write only after the gate's stop), and it does not match the memory case's store/projects/. So a rule in the body of a standing-rule skill neither moment reaches on the case, or of tracking-open-asks in a resumed conversation whose seeded turns never loaded it, cannot be moved by any round the runner can buy: read the indicator (on a seeded replay fixture, the probe and the seed's phrase) before designing the arms, and either restate the rule to land where every rep does read or do not buy a runner round for it - and on a case either moment reaches, read a lit indicator as the moment firing, not as the prompt reaching the trigger.
• The landing sweep reaches every case whose rep changes a rule file git does not ignore (SKILL.md, CLAUDE.md, CLAUDE.local.md, AGENTS.md, GEMINI.md, CONTRIBUTING.md, README.md, a .md directly under agents/, under skills/<name>/references/ or under a references/ beside a SKILL.md), or a CLAUDE.local.md git ignores (inside an ignored directory, one the rep wrote with Edit or Write): in the arm with the plugin that turn is handed, at its stop, the lines elsewhere still carrying what its change replaced, where it finds any, and acts on them before it ends, so a grader on any file the hand-off can name reads the hook's work beside the wording's, and the owning skill's indicator does not show it: the hook loads no skill, and none of the 35 eval reps it spoke to loaded maintaining-project-memory. In the kept runs five cases' reps wrote one - the three sweep cases and the writer case by design, and the tracking case's README.md rename - and replayed at each rep's first stop the hook handed lines on the sweep cases alone, so the other two stay comparable across the landing only while a replay stays silent. The hand-off is a hook_additional_context attachment in the session transcript under the run's config/projects/, never in the stream-json trace: replay the hook on a case's reps before designing a round there, and where it speaks, read the hand-off from the transcript (--keep-temp) before a grader (CONTRIBUTING, "Testing a wording change").
• The runner's --case filter takes one * glob.
• The default judge fails conditions that a hand read and a stronger judge both pass, on a multi-condition rubric over a file of a few kilobytes — measured against hand labels on two kept findings files, where it agreed with five of eight cells and the stronger judge with eight of eight. The judge is chosen once per invocation and no grader file can name one (model, judge_model, judge-model and judgeModel are rejected at load), and that choice is settled rather than open: pass --judge-model sonnet, $2.00–3.50 more on a baseline upwards of $55 (evals/judge-calibration.md has the cells). One condition per grader and a hand grade from the run JSON's evidence stay worth using on their own merits, not as substitutes for it.
• A curtailed rep is unmeasured, never a fail, and is replaced rather than counted, and a batch's --max-cost-usd sits above its expected spend, since the ceiling is checked before each launch and the run that reaches it has its judged graders scored as fails - but why it curtailed is a finding, with two sources only its trace separates: a rule that asks for more work costs more turns, and an instrument defect costs every rep of both arms turns, so the arm asking for more meets the cap first and the crossing reads as the wording's alone. The rep this rule was drawn from crossed a case's 60-turn cap under a wording that prices the work it asks for and with at least eight of its sixty calls spent on a broken git; with the git repaired, that case's reps under the same wording peaked at 36. Read both arms' turn and duration spread before any grader - from each trace's result records, since the run JSON's turns keeps only the last, and a rep that dispatches a subagent can write two, so one that ran 65 turns reads as 6 - and what a curtailed rep spent its turns on: where the instrument took them, repair it and re-take; where the treated arm's own work sits near the cap, raise the case's limits in both arms and re-take. Likewise, a rubric's rate is not comparable across two readings where a wording changed how much the run writes about what that rubric judges: one held 4/5 on a baseline and 0/5 under a wording that made each rep write a per-row verdict list, on files that affirm those rows by name. Nor across a repair to the instrument, where the move can be the judge's alone: one held 10/10 before the sandbox's git was staged and 5/8 after, on the same model, rubric and judge, with a hand read passing both sides in full - and the ready-made cause, that a working git let the reps re-apply mutations and write verdict language beside the record, is refuted by the runs before the repair, which reached a working git by the binary's full path, re-applied the same row, and wrote the same language in files the judge passed; so across a repair, test a cause for the move against the side the repair had not reached before writing it down. Hand-grade such a row rather than reading its rate, on both sides of the change whichever way it moved, and say so in the case's record.
• A condition deciding which reps count is one test in both arms, and a rep either arm drops for it is replaced: where the arm without a component cannot show the condition, a replay of the component's code as the treated arm ran it stands in, and filtering one arm alone makes the arms' counted reps different draws. The replay is fed every input the live code reads, or names the one it lacks and has every rep that input could change read by hand; otherwise it parts from the live run where the missing input matters, as it does where an input reaches it in a form the live run did not see. The landing-sweep hook learns of a write outside its rule files only from the turn's successful Edit and Write calls in the session transcript, and a replay without one would have had a rep that brought every copy into line before its first stop, both false positives included - the double fail the case exists to catch - re-taken as an instrument defect. Agreement with the live component on the reps at hand clears a replay only for the shapes those reps took (CONTRIBUTING, "Testing a wording change").
• The instrument has a floor: five reps an arm separate the arms only on a four-in-five gap, ten only on a half, and a three-rep baseline row only on a clean sweep — so a rule whose expected effect is one run in five or fewer is restated to bind every run, or declined as below the instrument, never measured harder.
• A draw bought to re-check a reading is read without the draw that prompted it: that draw was picked out for looking wrong, so it overstates the gap on average, and pooling counts it twice - as the reason for the re-check and as evidence for its result. Size the re-check by the floor on its own draws and set its bar on them alone before they run; a re-check whose own undecided reading buys more draws is itself a prompting reading, so those are read without it too, unless the extension and its bar were fixed before its first draw; give the pooled rate beside the bar, never instead of it; read a re-check that does not convict as leaving a gap below its floor open, not cleared; and whenever one grader of many crosses a threshold, say how many were read. On the commit case read as a neighbour of the first-tool-call line and the skill gate, a body grader was 9/15 pooled against 14/15 on earlier reps with neither the line nor a gate firing (p ≈ 0.04 one-sided), which a bar set on the pooled rate read as a decline, while neither re-check draw convicts read alone (p ≈ 0.14 and 0.45; 7/10 together, p ≈ 0.16), and it was one of fourteen graders read; the line landed with that row a watched cost.
• Before the floor, check that a rep reaches the condition the rule fires on: a rule carried by a trigger no rep crosses is untestable by this instrument rather than unproven, and where staging that condition means changing the fixture in the one way that removes it — shrinking the window a rule about remaining room would be measured against — no wording can be read out of the round, since both arms score alike for a reason having nothing to do with it. A condition no rep reaches on its own may still be stageable, and what answers that is the case format's own inputs rather than the kept traces, which show only whether the condition has occurred: context.history_file seeds a prior conversation by resuming a transcript the case ships, so a rule about what a second message must do is reachable although every case here is a single prompt — a rule whose condition the format can seed is untested rather than untestable, short a fixture rather than short a verdict.
• A replay case's seeded conversation carries the always-on brief itself, since the start hook does not fire on a resume, and the one run of such a case under with-without reported a single arm - no ablation row, so no retirement signal either way. The seeded copy is frozen at generation, and so is the skill's body, which the seed carries as the load a fresh conversation's first turn makes - without it no resumed rep loaded the skill, whether or not the seeded brief carried the first-tool-call line, the seeded turn having ended with no call, and with it a probe quoted a sentence only that body holds - so a replay round tests whichever text the seed carries, brief or skill, and regenerates the fixture once per arm, each arm's seeded copy carrying its own text, with a regeneration that finds no diff as the staleness check.
• A rule about restraint (what a session must not do) needs its baseline taken on the strongest model the maintainer runs as well as on the eval instrument: the stronger model over-applies a rule it has understood, and the weaker one may never have exhibited the failure.
• A row whose no-plugin arm scores as well as its plugin arm is a retirement flag, not a result: three things produce it — a grader that passes without the behaviour, a row with no rule behind it, and a rule the model has internalised — and only the third is a retirement, confirmed by running the landing test backwards (CONTRIBUTING, "When a rule outlives the model that needed it").