slotstream

# slotstream — agent instructions

Qwen3.8-Flash-Next (125B + 51B n-gram, 4-bit, ~104 GB) on Macs via SSD-streamed
experts and a slot cache. Read [PLAN.md](PLAN.md) for design, [MEASUREMENTS.md](MEASUREMENTS.md)
for every measured number and its method, `Tools/verify.sh` for the acceptance
battery. Work lands directly on `main`.

## Sevra integration direction (September 8, 2026)

Read [the canonical engineering integration record](db/records/design/sevra-maintained-model-integration.md),
also projected into PLAN.md. Sevra selects and maintains a few qualified models
for actual hardware profiles and optimizes the whole stack around them. The
product is not permanently tied to one flagship; the current engine's exact
checkpoint, measured gates and independent CLI/API/Swift behavior remain intact.
New choices require target-specific engineering and full-stack qualification.
Update README/docs, canonical plan/support records and generated projections
alongside verified implementation. Do not present this direction as shipped
model selection or new hardware support.

## Target range (September 16, 2026)

Slotstream is built for Macs that cannot hold the model in memory, 16 to
64 GB. Macs of 96 GB and more run it and gain from a larger cache, but they are
not the optimization target, and engines that keep the model resident are
faster there. Keep their rows in the docs as evidence labeled beyond the
target; never present larger Macs as the goal, plan work against the big-Mac
preset estimates, or trade the target range's speed or memory for them.
Canonical: [target range decision](db/records/decisions/target-range-macs-that-cannot-hold-the-model.md).

## Public documentation

Keep `README.md` approachable and complete: purpose, capabilities, measured
results with limits, requirements, quick start, a plain explanation, FAQs,
community/support, author, star count/history, and links to guides. Preserve
useful sections and project character; do not optimize for minimum word or
link count. Write setup guides for newcomers with complete commands, expected
results, and practical troubleshooting. Put protocol details, full benchmark
methods, implementation rationale, and test procedures in linked engineering
pages, starting at `docs/ENGINEERING.md`.
Preserve the tested configuration and move its explanation instead of
removing settings. Keep old public anchors usable when moving sections.
Canonical decision: [newcomer documentation](db/records/decisions/newcomer-documentation.md).

## The brain (`db/`) — read before touching MEASUREMENTS.md or PLAN.md

`db/` is a public db.md store and the authority for what this project knows:
one record per MEASUREMENTS.md section (`db/records/measurements/`), one per
PLAN.md section (`db/records/design/`, `db/records/plan/`), plus every public
number as a claim, the decisions, the machines, and raw runs. **MEASUREMENTS.md
and PLAN.md are generated from those records by `Tools/projections.py`; never
edit them directly.** Edit the record, rerun the script, commit both. The
session loop is `dbmd spec` once, read `db/DB.md`, `dbmd log tail 20 --dir db`,
operate through `dbmd`, `dbmd validate --all db`, then `dbmd log`. The three
rules from `db/DB.md` that matter most here:

- **Capture before you transcribe.** Raw tool output lands in
  `db/sources/runs/` first; the measurement record links it.
- **Every public number is a claim.** A number on README, `docs/`, or
  `llms.txt` needs a `db/records/claims/` record whose `needle` appears on
  every surface it lists; `Tools/claims_gate.py` fails otherwise, and a
  superseded measurement moves its claims in the same change.
- **Never delete a superseded or withdrawn record.** Set its status and link
  the correction; the retraction is evidence.

`Tools/brain_gates.sh` runs all three checks (it needs `dbmd`;
`Tools/dbmd_install.sh` pins the version CI uses) and is part of
`Tools/static_gates.sh`.

## Generated files — regenerate in the commit that moves their source

Three files in the tree are projections, and CI fails on a commit that moves a
source without them:

| generated | from | regenerate with |
| --- | --- | --- |
| `llms-full.txt` | `README.md`, `docs/*.md`, `CHANGELOG.md` | `Tools/llms_full.sh` |
| `MEASUREMENTS.md` | `db/records/measurements/` | `Tools/projections.py` |
| `PLAN.md` | `db/records/design/`, `db/records/plan/` | `Tools/projections.py` |

`make docs` runs both; `Tools/llms_full.sh --sources` prints the doc list, which
lives nowhere else. **Never hand-edit a generated file** — edit the source and
rerun. `make hooks`, once per clone, installs `.githooks/pre-commit`, which
regenerates them and stages the result with the commit; a six-line README FAQ
that landed without the regenerate on 2026-09-03 is why it exists.

## Claims and measurement discipline (mistakes made 2026-08-29/30)

**Operating defaults and limits.** Follow the canonical
[measured operating policies](db/records/design/measured-operating-policies.md).
For each important tuning value, document its purpose, units, kind of limit,
evidence and tested scope, tradeoff, override behavior, and revision criterion
beside the code and in the linked brain record. Keep justified defaults when
new hardware cannot be measured; unused capacity alone is not a defect.
An estimator's clamp is not evidence of a physical performance plateau.
Update code/help, gates, claims and user docs together when the policy changes.

Every rule here is one this project already got wrong. They share a root:
**asserting from a document instead of checking the system.**

- **Before calling something blocked, check whether the repo already does it.**
  Four docs said closing the prefill gap "needs a Metal kernel, which needs
  Xcode", for two releases — while `GatedDelta.swift` shipped a JIT-compiled
  custom kernel as its fast path. The blocker was read off the risk register,
  which is about a *different* thing (mlx-swift's bundled shader library), and
  never tested. One `xcrun`, one four-line kernel, and it collapsed. A blocker
  that has never been reproduced is a rumour.
- **An estimator may not return a value outside the range it measured.** The
  decode curve extrapolated to 20 tok/s at 181/layer and over-promised 25 to
  45% through its own middle. Worse: immediately after fixing that, the prefill
  ladder was left returning 125 tok/s for chunk 4096, which nothing had
  measured. Cap at the last verified point, or measure a *ratio* at a config
  that fits and say so. Under-promising is the correct failure direction for a
  planner.
- **Re-anchoring a curve invalidates every number derived from it.** Fixing
  only the row in front of you produces contradictions readers will trust: the
  README's tier table ended up claiming a 32 GB Mac is faster than a 48 GB one.
  Regenerate the whole family — and prefer generating tables *from the tool*
  (`doctor --sim-ram N`) so they cannot drift from the code again.
- **A failing test is a bug in the test until proven otherwise.** Three "product
  failures" in one run were nested python-inside-shell quoting mangling the
  JSON; another two were a hardcoded version literal reporting a release bump as
  a regression. Reproduce by hand before believing a failure, and never pin a
  value a release will change.
- **`set -e` makes cleanup lethal.** `wait $PID` returns 143 after a `kill` and
  silently truncated the whole battery — it looked like a hang. Always
  `kill ... || true; wait ... || true`, or use a `trap ... EXIT` like
  `api_robustness.sh` does.
- **Benchmarks on a loaded machine are noise.** Single runs here vary 15%+, and
  one pair read 137.6 against 90.7 for the same config. Check reclaimable
  memory first, interleave A/B rounds, and discard anything measured while the
  machine is swapping. Report medians of paired rounds, never a best-of.
- **Global paging is not a correctness or process-memory gate (2026-09-11).**
  macOS swap counters cover every application. Preserve them as diagnostics;
  never abort or fail ordinary release, context-capacity or governor checks
  solely because they increase. Keep actual process-footprint ceilings, real
  headroom, OS pressure cancellation, allocation guards and completed-work
  checks. Clean benchmark eligibility is separate from functional acceptance.
  Historical frozen studies keep their original verdicts and protocols.
  [Canonical policy](db/records/decisions/global-paging-is-diagnostic.md).
- **Explicit knobs bypass the safety clamp — that is their purpose, and it
  makes them dangerous.** `--experts-per-layer 181` is never resized by the
  availability clamp, and forcing it against 26.6 GB reclaimable drove the
  machine to 158 MB free and 13 GB of swap. `Planner.availabilityOverride` is
  worse: simulating 60 GB free made the governor allocate a *real* 25.4 GB pool
  and pushed swap to 39 GB. Bound both by `deviceAvailableGB()` before use.
- **Kill your own background waiters.** A poll loop watching a log that would
  never get its line sat in the task list for six hours looking like live work,
  and the "nothing is running" check missed it because it grepped for expected
  process names. Check the task list, not your assumptions about it.

## Memory safety — READ BEFORE RUNNING ANYTHING (incident 2026-08-28)

This Mac has **48 GB of unified memory shared with Carlos's live apps and
session**. On 2026-08-28 a session stacked test processes — a ~31.5 GB soak
server, a second test server, a browser pane, and builds — overcommitted the
machine and **crashed the whole system**. Every model process here is
multi-GB. These rules are mandatory:

1. **One model process at a time.** Never two servers; never `serve` plus a
   `run`/`elastic-check` concurrently. The binary now enforces this with a
   per-user file lock before model allocation. Still inspect `pgrep -fl
   slotstream` before heavy work; do not kill a process owned by another task
   without coordinating with its owner.
2. **Check reclaimable memory before every heavy step** (model launch, big
   build, verify run). Reclaimable = `vm_stat` free + purgeable + file-backed
   pages; `slotstream doctor` prints it as "reclaimable now". If what you are
   about to start does not fit with several GB to spare, do not start it.
3. **Tests use small explicit sizes** — `--memory-gb 8.1`..`10` — never auto,
   unless the large configuration is itself the measurement, and then nothing
   else heavy may be running.
4. **Kill every test process the moment its test ends**, and confirm.
5. `Tools/verify.sh` keeps ordinary equality gates between the 8.1 GB floor
   and a 10 GB target. The full live-governor drill is an explicit exception:
   unchanged shrink/grow deadbands require `--slots 1000 --max-memory-gb 13`,
   with a 16 GB real reclaimable preflight, internal target-plus-3 GB checks,
   sampled memory/swap evidence and no other heavy work. A skipped drill does
   not pass acceptance. The full image/MTP diagnostic separately uses an
   explicitly priced 12 GB target and a 15 GB preflight. The complete original
   vision-serving photographs separately use 14.5 GB and a 3072-token prefill
   reservation, after a 20.5 GB preflight; that workspace prevents the larger
   image's correct target refusal. Keep this override local to the image
   server. Never use spare RAM to enlarge an equality profile.
6. The engine caps MLX's allocator cache at 2 GB (`Engine.swift`,
   `MLX.Memory.cacheLimit`). Do not remove it: without the cap a 10 GB-target
   server held 15.1 GB of real RSS (freed transients hoarded by the
   allocator); with it, 6.0 GB flat at identical speed. `GenStats.peakMemoryGB`
   combines the kernel lifetime physical-footprint peak, lifetime RSS and current
   footprint. RSS alone misses released GPU allocations. Verification also keeps
   sampled request intervals and swap observations; the lifetime peak includes
   loading and earlier requests. The MLX-only peak is diagnostic only.
7. **No memory-hog stress experiments without Carlos's explicit go.** The
   2026-08-28 hog experiments are done and documented in MEASUREMENTS.md;
   never rerun them casually.
8. The elastic governor protects **one auto-sized instance** against the rest
   of the system. It cannot protect against deliberately stacked processes —
   that protection is these rules, i.e. you.

## Weight download

Fresh pulls use Slotpack v1: a fully hash-pinned, lossless compressed package
in the public Hugging Face mirror at an exact repository revision. The legacy
`weights.sevra.page` hostname redirects through free static asset rules; it
must not proxy model bytes through R2 or metered Worker code. Preserve original model
bytes, pinned hashes, range coverage, and all decoder bounds. The canonical
format and release qualification are in `docs/DOWNLOAD-FORMAT.md`.

Download, bounded parallel decoding, and writes overlap. Every network worker
owns its URLSession so HTTP/2 streams do not collapse the intended independent
connections. Automatic mode starts at eight and trials increases only while
measured throughput improves; explicit counts stay fixed. CDN cache status
and actual connection observations are logged when available. Honor server
rate-limit reset headers with cancellable waits. Do not infer a speed guarantee
from byte reduction or extrapolate an unmeasured multi-gigabit connection.

Keep compressed-object, decoded-chunk, and final original-file SHA-256 checks.
Resume bits follow synced writes; final paths appear only after whole-file
verification. Cancellation drains workers. A directory lock rejects competing
writers. Optional failures wait for outstanding writes before cleanup. Run
`Tools/slotpack/checks.py` after transport changes, and qualify any new package
with an independent complete public CLI pull before enabling it in a release.

The raw downloader remains the explicit compatibility path and preserves old
`.partmap` resumes. `SLOTSTREAM_WEIGHTS_SOURCES` still selects raw sources in
automatic mode. Raw fallback inside a compressed pull uses bounded original
ranges and cached signed redirects. The Linux bandwidth harness compiles the
same production sources; it is a test instrument, not a Linux inference port.

## Serving invariants (learned the hard way)

These were all real bugs found by adversarial probing. Each is now gated by
`Tools/api_robustness.sh`; do not "simplify" any of them away.

- **SIGPIPE must stay ignored.** `Server.run` sets `signal(SIGPIPE, SIG_IGN)`
  and each accepted socket gets `SO_NOSIGPIPE`. Without it a client closing a
  tab mid-stream kills the whole daemon, and every `alive`/`send -> Bool` check
  in the handlers is dead code because `write` can never return `-1`.
- **Every sampling knob goes through `SampleParams.sanitized()`.** Clients send
  Ollama's documented defaults `seed: -1` and `num_predict: -1`; `UInt64(-1)`
  and `0 ..< -1` both trap and take the process with them. Out-of-range
  `top_p`/`min_p` used to empty the candidate set and turn `probs/probs.sum()`
  into NaN, after which the sampler emitted token 0 forever.
- **Never normalize the sampling probabilities.** The draw is scaled by the
  unnormalized CDF total instead. That removes the 0/0 and, since `u < 1`,
  guarantees the pick lands on a token with actual mass.
- **Incremental detokenization is bounded and scalar-safe.** Qwen's ByteLevel
  decoder is run over small stable token groups while incomplete UTF-8 bytes
  and stop-sequence prefixes remain buffered. Non-streaming requests do one
  final decode only; never restore full-prefix decoding after every token.
- **The pass shrinks as the context grows, and the bound is measured, not
  chosen.** `PrefillSchedule.chunk(at:maxChunk:)` halves the pass until
  physical query rows × key extent is under 4096 × 8016, the largest query-by-key product any
  prefill measurement covered, because the sparse-attention layers score every
  query of the pass against every key of the context (a chunk × context
  transient). Count cropped numerical-alignment query rows and masked key
  columns too. Preserve the original 256-row floor while it fits; there is no
  floor exemption from the product bound. Do not raise
  `measuredQueryKeyProduct` or add a prefill anchor without a staged
  `context-check` measurement recorded in MEASUREMENTS.md, and never quote a
  context number the tool did not print (`doctor`, `prefill-schedule`).
- **Prompt plus completion is capped (`--max-context`, default `auto`).** Auto
  takes the largest of 32,768, 65,536, 131,072 and 262,144 whose plan keeps MTP
  and the lookahead as the 32,768 plan has them, retains one complete
  conversation and adds at most 10% to the representative request without removing cache
  above the measured decode range, judged on RAM
  and working set. A busy start applies the same memory and performance rule; `--experts-per-layer` and `--pool-gb` keep 32,768, while `--memory-gb`
  still gets a window priced inside its target
  (`records/decisions/automatic-context-window-per-machine`). Frozen allocation
  fixtures and monotonic sweeps pin an explicit 32,768. The model limit,
  implementation limit, configured window and request cap are distinct. Carlos
  opened the implementation and MTP limits to the model's 262,144 on 2026-09-13
  with native 131,072-token runs with and without the draft head inside their
  plans and no native 262,144-token run; images stay at 65,536. The planner
  charges actual stepped active capacity, bounded retained state, resident modes
  and workspace. `Engine.generate` clamps new tokens to the remaining room. Full
  replacement buffers are charged before releasing their old readers; spare
  main, draft and indexer buffers cannot credit each other. The open native
  gates stay listed in the configurable-context plan; never describe them as
  passed.
- **One accepted request owns its guards through preparation and queuing.**
  `--max-prefill-wait` defaults to 30 minutes to the first sampled token; zero
  disables only time. Unknown ETA never disables the wall guard. Concurrent
  preparation and pending dispatches reserve shared headroom atomically;
  prepared images retain their reservation while queued and while owned.
  Failures after streaming begins end with an error, never a success tail.
- **A metadata endpoint must never take the generation lock.** `/api/tags` and
  `/api/ps` want pool numbers, and reading them through `withExclusive` made
  both block for the whole of a running request. Worse, the accept loop waited
  on the connection semaphore, so enough blocked metadata calls stopped the
  process answering anything — a client polling either endpoint could not tell
  a generating server from a crashed one. `Engine.publishPoolSnapshot`
  republishes under its own lock at every resize; the endpoints read that.
- **The accept loop must never block.** `connSlots.wait()` on the accept thread
  turned a full connection pool into a dead server. It takes the slot with a
  zero timeout now and answers 503 when there is none.
- **JSON `null` means "not set".** The OpenAI client serializes an unset
  `max_tokens` as null and the Ollama CLI sends a null `options`; treating
  either as a present-but-wrong value turned a stock default request into a
  400. `Server.withoutNulls` strips them before validation.
- **A no-op value is not a feature request.** Refusing `n: 1` or
  `frequency_penalty: 0` broke stock SDKs while protecting nothing: those name
  the behaviour this server already has. Accept the exact default, refuse every
  other value. This does not license accepting a knob that would change the
  reply — Ollama `num_ctx` and `repeat_penalty` are still refused, never dropped.
  The OpenAI adapter accepts a bounded `options.num_ctx` and enforces that
  per-request limit; `Tools/openai_tools_gate.py` checks it.
- **An unseeded request gets its seed at the API boundary.** `Sampler`'s own
  default is a constant, so an unseeded request replayed one fixed stream from
  process start while the docs promised otherwise. The draw lives in
  `Server.sampleParams` and `v1Chat`, never in `Sampler`, so every offline gate
  stays deterministic.
- **Reasoning goes in `thinking`, never in the answer.** `ThinkSplitter` routes
  everything before `</think>` to `message.thinking` in both streamed and
  non-streamed replies, withholding the last few characters so a tag split
  across two deltas is never emitted as text.
- **The incremental decoder holds back as little as it can.** Eight tokens
  before the first flush and four after it meant one delta per four tokens, and
  no streaming at all below eight. Byte-exactness rests on the U+FFFD check,
  not on the size of the backlog.
- **`Geometry` constants are checked against config.json** in
  `Qwen4ExpModel.validate`. The planner sizes memory from the constants while
  the engine allocates from the config; if they drift, every memory number the
  user sees is wrong.

- **The Ollama CLI's wire format is a gate, not an assumption.** Its
  `ShowRequest` serializes every field, so `ollama run` opens `/api/show`
  with empty `name`/`system`/`template`/`options`, its one-shot mode uses
  `/api/generate` with empty `suffix`/`system`/`template`, its interactive
  mode opens with Ollama's documented "load" request (an empty prompt,
  answered `done_reason: "load"` without touching the engine), and its chat
  can carry `keep_alive` and `options: null`. 0.1.8's strict validator rejected them
  and the CLI could not start; nothing noticed for two releases because the
  claim lived in PLAN.md, not in a test (found 2026-09-01). Accept the
  deprecated `name` alias, empty overrides, `keep_alive`, and null `options`;
  keep refusing unknown fields and non-empty overrides. `api_robustness.sh`
  gates the CLI's exact request shapes; a client claim in the README needs a
  gate like it.

## Conversation prefix cache

- **A reused state is extend-only and can never be rewound.** `LinearCache`
  holds the GDN recurrent state, a fold over every token with no inverse, and
  `ngramCtx` is carried forward the same way. So reuse requires
  `prompt.starts(with: heldIds)` and `prompt.count > heldIds.count`; anything
  else is a full rebuild. Do not add "partial rewind" or longest-common-prefix
  matching — there is nothing to rewind to.
- **The held id list is tracked, never inferred.** A token is sampled *before*
  it is fed, so both break paths in the decode loop leave the last token
  unconsumed. `Generator.generate` records exactly what the state consumed; a
  caller that recomputes this from the returned ids will be off by one and the
  next request will silently reuse a state that does not match its prompt.
- **A miss must evict before the caller allocates.** Four conversations may be
  retained, so `PrefixCache.take` evicts LRU entries until retained + active
  states fit both the four-state and shared-token ceilings. Each state has
  ~113 MB of fixed GDN memory in addition to ~27 KiB/token; both are charged.
- **A turn may resume only at its own prefill pass boundaries** (2026-09-17).
  The arithmetic depends on how tokens were grouped into passes and on whether
  each one was read or generated: a 256-row pass sums a row in a different
  order from a one-row decode step, MLX picks reduction orders by shape, and
  top-10 expert routing turns those differences into different experts. So the
  state a turn leaves behind, its prompt read in passes and then its reply
  decoded a token at a time, is not what reading those same ids computes, and
  continuing from it can move a token across the decision. It did: on a
  1,430-token agent turn a fresh read scored `>` at 0.9576 and `]` at 0.0421
  for one position of tool-call syntax, the continued turn inverted them, and
  the model's first `file.edit` call arrived malformed. `PrefixResumeRule`
  (`InferenceOptimizations.alignedPrefixResume`, on in the deployed family)
  offers a request only a state whose length is one of **its own** pass
  boundaries and whose every token was read in those passes; everything after
  the boundary is re-read. `SLOTSTREAM_OPT_ALIGNED_RESUME=0` restores the old
  behavior, for comparison work only.
- **What the rule costs.** A follow-up turn re-reads back to the last boundary,
  which is one partial pass instead of nothing. Measured at 961 slots on the
  same three-turn chat: follow-up prefill 2.47 s -> 8.56 s, against 26.3 s to
  read the conversation cold. What it buys is that the continued turn and the
  cold one produce the same tokens and bit-identical prompt logits, which is
  what `slotstream prefix-exact-check` asserts. Under the rule a conversation
  state is worth only its ids (`peek` splices them into the next prompt), so it
  is the first thing evicted and the boundary snapshot is the last; a deeper
  snapshot of the same conversation replaces the one it supersedes, or the
  resume point never advances and each turn re-reads more of itself.
- **Byte-equality against a cold rebuild holds only under the rule.** Without
  it the comparison never passes and must not be asked for: swept over a
  64-token sequence, all 63 split points differ, and a continued turn moved
  logits 3.7% to 5.9% of their spread. That is inside the band re-chunking a
  plain prefill already moves them, which is what `slotstream prefix-check`
  measures (4.37% against a 5.90% control) and is still the right gate for
  *different pass sizes*, which the rule does not make equal and cannot.
  §6.1's "streaming is math-invisible" is about the expert pool, where hit and
  miss deliver identical bytes.
- **The governor sheds it before shrinking the pool.** One re-prefill is a
  cheaper give-back than a starved cache, which taxes every token after it.
- **It holds four conversations, and must not be reduced to one.** A single slot
  passed every synthetic test and then scored 0 hits / 7 misses against Open
  WebUI, whose title-generation request lands between turns and evicted the chat
  every time. Any client that decorates a conversation (titles, tags,
  suggestions) breaks a one-slot cache. Because several held states are
  additive, the retention ceiling is charged against the memory budget.
- **The disk tier restores the saved representation; it never rebuilds.**
  `--prefix-cache-dir` (`PersistentPrefixCache`) writes a committed text state
  after its reply (live rows only, original buffer shapes) and restores it into
  buffers of those shapes, so offsets, allocated bytes and continuation match
  the saved state exactly. `optimization-state-check --variant
  persistent-prefix[-mtp]` gates that against a memory hit, and
  `Tools/persistent_prefix_e2e.py` gates identical ids across real restarts.
  Rows live in immutable segments; a state descended from a persisted head
  (`State.persistedLineage`) writes only the rows added since. Every rewind
  clears that lineage through `invalidateCheckpoints`, so a new path that lowers
  `tokenCount` must go through it, and rows are never matched by token equality
  alone: a re-prefill of the same ids produces other bits. Memory answers
  first; disk is read only for a longer state, after making room like a miss.
  A file belongs to one executable image, sampled weight content, geometry and
  optimization settings, so every rebuild starts fresh; do not loosen that key
  without a check that rejects changed arithmetic. A speculating request takes
  only states with the draft cache, or every later save of that conversation
  would lack it.

## Prefill

Each of these cost a measured experiment. Do not re-derive them, and do not
revert the constants to their older values — two of those older values are
still quoted in commit history and both are wrong.

- **A pass of 256 tokens or more is a sweep, never a pool load** (`MoELayer.sweep`,
  `SweepTuning.minTokens`). Rows are sorted by expert, the layer's experts go
  through staging groups of 32 — resident ones copied out of the pool, the rest
  read from the checkpoint in contiguous runs — and each group is one grouped
  GEMM per projection with `sortedIndices: true`. That flag is the whole
  point: it is what reaches MLX's `gather_qmm_rhs` kernel, which reads an
  expert's weights once per tile of tokens instead of once per token. The old
  per-(token, expert) gather over the pool never could, and re-read every
  expert about forty times per 2048-token pass. Measured on 0.2.2's code: an
  8k prompt at a 16 GB target went 91 → 184 tok/s (three interleaved rounds),
  prose 66 → 107, the 8.1 GB floor 51 → 88.
- **The kernel a row meets must depend on the routing alone.** MLX takes the
  grouped kernel only when a call has at least 16 rows and four per expert of
  the weight array it is handed. With the pool as that array the rule would
  have switched kernels with the cache size, so the sweep hands it a group of
  at most 32 experts and pads a short group up to the rule. This is what keeps
  the golden-equivalence invariant (§6: pool size and contents never change
  the math); `sweep-check` proves the sweep bit-identical on a cold pool and
  on one holding 638 of the prompt's experts. Do not let a group's composition
  depend on residency in a way that changes row counts per call, and do not
  drop the padding.
- **The sweep never writes the pool.** That is the scan resistance PLAN §3.3
  asked for: a long prompt cannot evict what decode was using. Only the final
  pass admits, and only each layer's fair share of the pool by frequency
  (`SlotPool.admit`, `admitOnSweep` set by the generator). Resident groups go
  first within a layer, so admission can never evict a resident expert that
  layer has not copied yet.
- **The sweep is read-bound, not compute-bound, and the GPU is idle waiting
  for reads.** `SLOTSTREAM_SWEEP_TRACE=1` on the 8k prompt at 16 GB: reads 22 s,
  waiting for the GPU 1.6 s, everything else 20 s. Reads run at 11–13 GB/s
  against the SSD's 17.3 on 2.7 MB records; queue depth 12 stays right (4 loses
  a third, 32 gains nothing). What is left is serial: the router, attention,
  and the layer tail run while no read is outstanding, because the next layer's
  experts are unknown until its router runs.
- **Prose costs more than repeated text, and the n-gram rows were why.** A
  token needs sixteen ~100 B rows, three `pread`s each at ~55 µs SSD latency;
  fetched one at a time on the calling thread, a 10k-token prompt of ordinary
  prose spent ~35 s there (in the old path and the sweep alike), while the
  acceptance prompt's repeated sentences hid it behind the row cache. The rows
  of a pass are known from its ids, so `NgramStore.prefetch` reads every
  missing row on 32 lanes before the embedding is assembled. Do not go back to
  one row at a time.
- **The pass size is part of the memory plan, not a constant.** A pass touches
  nearly every expert of every layer, so a bigger pass is strictly faster and
  strictly more memory-hungry. Measured at a matched pool of 60/layer with the
  sweep: 88 → 128 → 169 → 211 → 222 tok/s from 256 → 4096. Output is inside the
  prefill-rechunk band at every size (the `prefix-check` method); it is not
  byte-identical, and the docs must not say it is.
- **`prefillCostGB` charges ~1.30 MB per chunk token, linear from zero.**
  It previously charged `(chunk - 256) x 1.8 MB`, which conflated two different
  things — pass activations, which scale with the chunk, and KV plus indexer
  state, which scales with the *context* — and so overcharged a big pass by 2x
  and kept the planner one size below the best available. Measured directly:
  1024 → 1.30 GB, 2048 → 2.19, 4096 → 4.30. Context state is separate and
  small: 4k → 8k tokens moved peak by 0.1 GB. **Do not restore the 1.8 figure.**
- **`prefillChunkFor` takes at most a quarter of the pool budget**, raised from
  a fifth once the cost above was honest. The deciding experiment held total
  memory fixed and traded pool for pass size: 2048 dominated 1024 on every axis
  — faster prefill, faster decode, *lower* peak.
- **Staging is bounded at 32 records** (`SLOTSTREAM_EXPERT_LOAD_BATCH`), the
  sweep's group size and the pool path's load slice. A 256-token layer can
  route all 512 experts; loading them as one 1.415 GB record batch made a
  `--memory-gb 10` long prompt peak at 12.4 GB in 0.1.x. The sweep keeps at most
  two groups alive and caps MLX's buffer cache while a prompt is read
  (`SLOTSTREAM_PREFILL_CACHE_MB`), because its varying array sizes otherwise
  fill the whole 2 GB cache. Re-run the real process-RSS gate before touching
  either.
- **Cross-layer read-ahead does not work here as a background thread.** It was
  built, measured slower in every paired run, and removed (2026-08-30). The
  sweep overlaps reads with the GPU inside a layer instead, on the main thread,
  which is a different thing. Reading layer L+1's experts during layer L is
  still the one lever left for the serial part above, and it costs a layer of
  staging; measure before building it.
- **That is NOT blocked on Xcode, despite what the risk register implies.**
  The register's entry is about building *mlx-swift's own bundled shader
  library* from source, which is worked around by vendoring `mlx.metallib`.
  Writing a **new** kernel is a different thing: `MLXFast.metalKernel` JIT-
  compiles Metal source at runtime through the Metal framework, needing no
  offline toolchain. This repo already does it — `GatedDelta.swift` builds the
  gated-DeltaNet kernel that way and it is the shipped fast path. The grouped
  GEMM turned out not to need one: MLX ships it, behind a sorted-index flag.

## Sampler and governor

- **The sampler has a numpy oracle.** `Tools/sampler_ref.py` must stay in step
  with `Sampler.next`; both build logits from the same splitmix64 stream using
  only exactly representable float operations, so the comparison is exact.
  Changing the sampler means changing both.
- **The governor's policy is a pure function on purpose.** `GovernorPolicy.decide`
  is tested through all its branches by `governor-check` with no model loaded.
  Do not fold the policy back into the daemon: the alternative test is putting
  this machine under real memory pressure, which is exactly what the memory
  safety rules forbid. Note the invariant it asserts — the decision depends on
  (available + pool), never on either alone, which is why `desiredSlots` credits
  what a restart would release.
- **`elastic-drill` covers the wiring the policy test cannot**: poll, decide,
  take the generation lock, resize, log. It uses `Planner.availabilityOverride`,
  which **does not make the allocation imaginary** — simulating 60 GB free on a
  machine with 7 GB made the governor take a real 25.4 GB pool and drove swap
  from 13 to 39 GB. Anything using that seam must bound the simulated value by
  `deviceAvailableGB()`.
- **Warm decode estimates use development-Mac anchors.** 6.0 / 8.2 / 11.2 /
  11.6 tok/s at 30 / 60 / 120 / 150 experts per layer show diminishing gains
  over that measured range, not a universal plateau. Community M5 Max runs
  demonstrate gains at larger manual targets; see `docs/HARDWARE.md`.
  An older
  20.0 at 181/layer has never reproduced; the estimator holds flat above the
  verified points rather than extrapolating to it.
- **A speculative rejection rolls back, it never re-runs.** The verify pass
  records the linear layers' state after every position (`LinearCache.record`;
  the GDN recurrence stepped per token, which `mtp-check` proves bit-identical
  to the fused kernel), and `State.rollback(keeping:of:from:ngramWindow:)`
  swaps to the recorded state, trims the attention caches, and rebuilds the
  n-gram context from ids. Re-running the kept tokens cost most of a pass on
  most rounds and was the single largest avoidable cost in speculation
  (MEASUREMENTS M9). Do not reintroduce a rebuild; extend the recording.
- **Speculative decode's multiplier is a measured ratio per cache size and
  depth.** `mtp-bench` on 0.2.0 (four drafts) read ×0.55 / 0.69 / 0.88 / 0.96
  at 20 / 29 / 42 / 57 experts per layer and ×0.88 at 122, all below
  break-even; depths 1 and 2 read ×1.13 / ×1.12 at 57 and ×1.17 / ×1.13 at
  122, near the former automatic activation floor, which explained the former default
  of 1; with the rebuild eliminated depth 1 reads
  ×1.20 at 57 and ×1.24 at 122 (×1.18 sampled). The "×1.5–1.9" once written here assumed a
  five-token verify pass costs one token's pass; `mtp-passcost` measured
  1.65 with every expert resident (a sixth of a pass per extra token), so
  the ceiling is ×1.4 at depth 1 and the estimate is withdrawn. Quote the
  ladder and the ceiling, never the launch-bound arithmetic. Carlos adopted
  **two drafts as the default on 2026-09-11**. Version 0.2.16 lowers the
  activation floor to 76/layer after the head and context charges, before
  the separate lookahead reservation; older studies retain their original floor.
  The automatic-40%-RAM study found two and three effectively
  tied overall and did not qualify a universal optimum. Keep the adoption
  decision separate from those measurement limits:
  [current draft-depth policy](db/records/decisions/draft-depth-defaults-to-two.md).
  New experiments use the adopted depth unless they explicitly study another
  depth; preserve frozen historical fixtures and raw evidence.

## Repo facts

- Model weights: `models/qwen38-flash-next-mlx-4bit/` (97 GB, gitignored),
  pinned `pipenetwork` revision; `slotstream pull --verify` re-checks all
  hashes in ~14 s and is a verify.sh gate.
- Parity goldens must be generated under **mlx 0.31.1** (`.venv31`,
  `Tools/parity_ref.py`). The runtime now pins MLX 0.32.2; kernel-upgrade
  fidelity and same-backend cache equivalence are separate gates. Preserve the
  historical reference: never regenerate goldens under a newer mlx.
- SwiftPM cannot compile Metal shaders with CLT only: the Makefile colocates
  the prebuilt `mlx.metallib` next to the binary. `swift test` is unavailable
  (no XCTest in CLT) — `Tools/verify.sh` is the acceptance suite.
- CI is split by product. `ci.yml` builds and checks the engine and skips pushes
  that change only docs, the brain or `apps/`. `sevra-mac.yml` runs
  `Tools/check_sevra_mac.sh` and the Xcode build (`Tools/build_sevra_xcode.sh`)
  when `apps/macos` or the engine it builds on changes. The app's real-model
  checks need the weights and stay on a development Mac.
- Coverage percentages are advisory review feedback. Build, instrumented-test
  and report-generation failures still block CI; never make the whole coverage
  job optional. Inspect uncovered safety behavior and retain all correctness
  gates. See [coverage policy](db/records/decisions/coverage-as-review-feedback.md).
- The sandbox proxies localhost HTTP clients (curl/urllib): test the server
  with `nc` raw sockets, or the app's Browser pane (which reaches localhost).
- Launch background servers with `(nohup ... &)` subshells; TaskStop kills
  whole process groups.
- Distribution: `install.sh` (repo root) is the public one-line installer; it
  fetches the latest release asset `slotstream-arm64.tar.gz` (binary +
  `mlx.metallib`, plus a `.sha256` file) into `~/.slotstream/bin`. **Cutting a
  release**: bump `version` in `Sources/Slotstream/Version.swift` (the single
  source for `--version`, `/api/version` and
  the CI tag check) to match the tag, commit, then
  `git tag vX.Y.Z && git push origin vX.Y.Z` —
  Wait for the commit's complete main CI run to pass before pushing the tag.
  CI builds on macos-26, preserves the binary, metallib, source archive and
  build identity, runs every static/golden/catalogue/coverage/library gate,
  and confirms its tested bytes still match that archive.
  `.github/workflows/release.yml` requires successful main CI for that exact
  commit, verifies the archive hashes and reconstructed source, requires
  `--version` to equal the tag, attests provenance
  (`gh attestation verify <asset> --repo carloslfu/slotstream`), and publishes
  those same bytes without another compilation. Never build release assets locally except as a documented
  emergency fallback. Asset names are stable (the installer uses
  `releases/latest/download/`), so never rename them. The tarball's metallib
  is the macOS 26 build (CI pins it via `SLOTSTREAM_METALLIB_MACOS=26`);
  `install.sh` swaps in the macOS 14/15 builds from pinned mlx-metal wheels —
  when bumping the MLX version, update those wheel URLs + sha256s alongside
  `Tools/fetch_metallib.sh`. raw.githubusercontent caches `install.sh` for
  ~5 minutes after a push.