slotstream

# slotstream: full documentation

Generated by Tools/llms_full.sh from README.md docs/GETTING-STARTED.md docs/ENGINEERING.md docs/SEVRA-MAC.md docs/EXPERT-LOOKAHEAD.md docs/HERMES-NOTES.md docs/CLIENTS.md docs/CLI.md docs/API.md docs/LIBRARY.md docs/FX.md docs/HERMES.md docs/CODEX.md docs/CLAUDE-CODE.md docs/CODING-AGENTS.md docs/TESTING.md docs/TROUBLESHOOTING.md docs/DOWNLOAD-FORMAT.md docs/HARDWARE.md CHANGELOG.md. Edit those, then rerun it.
Relative links below point at the same sections, which are all in this file.


<!-- ===== README.md ===== -->

# slotstream

[![Latest release](https://img.shields.io/github/v/release/carloslfu/slotstream?label=latest%20release)](https://github.com/carloslfu/slotstream/releases/latest)
[![GitHub stars](https://img.shields.io/github/stars/carloslfu/slotstream?style=flat&logo=github&label=stars)](#star-history)

**Run a 105 GB AI model on a Mac that can't hold it.**

Slotstream runs [Qwen3.8-Flash-Next](https://huggingface.co/pipenetwork/Qwen3.8-Flash-Next-MLX-4bit),
a 125-billion-parameter open model, on Macs with 16 to 64 GB of memory. It
keeps most of the model on the SSD and loads the parts it needs as it writes.
Our 48 GB M5 Pro measured 15.86 tokens per second at a 22 GB memory target
([how it was measured](#speed)).

Chat with it, ask it about pictures, or code with it: `slotstream launch claude`
starts Claude Code on the local model, and Codex, Pi, opencode and Hermes work
the same way. Developers can connect their own apps through its Ollama-,
OpenAI- and Anthropic-compatible APIs or its Swift library.

After a one-time download it works offline, with no Python and no cloud
account. The whole engine is one native Swift program on Apple's MLX and
Metal; see [Built native](#built-native). Every published number has a
recorded method, and the experiments that failed stay in the
[measurements](MEASUREMENTS.md).

[Get started](#install) · [Speed](#speed) · [Guides](#guides) · [Get help](#support)

> **I'm building Sevra on Slotstream: private, personal AI optimized for your computer.**
> Sevra will choose a tested model for your hardware, keep that choice current
> as models improve, and let you control what it remembers. The Mac app is in
> development and runs Slotstream in process; see
> [how it is built](docs/SEVRA-MAC.md) and
> [join the waitlist](https://www.sevrahq.com/). Slotstream's command-line
> tool, APIs and Swift library remain independently usable.

## Who it's for

Slotstream is built for Macs that cannot hold the model in memory: **16 to
64 GB**. That is where the engineering, the measurements and the defaults go,
so that frontier-class intelligence runs on the Macs most people already own.
It also runs on 96 GB and larger Macs, where the model fits in memory, but it
is not optimized for them: engines that keep the whole model in memory report
faster replies there. See
[related projects](docs/ENGINEERING.md#related-projects) if that is your Mac.

## Will it run on my Mac?

You need an **Apple Silicon Mac with at least 16 GB of memory, macOS 14 or
later, and about 110 GB of free SSD space**. Open About This Mac from the
Apple menu to check your chip and memory. On an 8 GB Mac even the smallest
memory plan doesn't fit, so Slotstream refuses to start instead of swapping.
Windows, Linux and Intel Macs are not supported. The
[hardware guide](docs/HARDWARE.md#what-you-need) has the tested macOS versions.

## Speed

`tok/s` means tokens per second; a token is a small piece of text, often part
of a word. Reply speeds below describe generation after the model has warmed up.

**Our development Mac, a 48 GB M5 Pro, measured 15.86 tok/s with 0.2.19 at a
22 GB memory target**, in a controlled benchmark on eight prompts the engine
was never tuned on. The engine predicts which experts the
next layers will need and reads them from the SSD before they are asked for,
which changes speed and never the output. The
[expert lookahead guide](docs/EXPERT-LOOKAHEAD.md) has the measurements behind
each release. This historical test used smaller prompt passes and disabled
prefix caching, leaving more memory for experts. It is not a measurement of
today's automatic configuration. A qualified full-answer baseline on 0.2.23
[is still pending](db/records/measurements/release-speed-calibration-2026-09-22.md).

<a id="speed-by-memory"></a>
<a id="speed-by-mac-memory"></a>

### What to expect by memory

Rough planning ranges for warm replies, from community reports and our own
measurements, rounded outward. Faster chips and SSDs sit at the top of each
range; other apps and memory pressure pull results down.

| Installed RAM | Estimated warm reply speed | Example automatic context window |
|---|---|---|
| 8 GB | **Support coming soon.** The current model doesn't fit yet. | Not available yet |
| 16–<24 GB | ~1–6 tok/s | 32,768 tokens |
| 24–<48 GB | ~5–16 tok/s | 32,768 through 32 GB; 65,536 at 36 GB |
| 48–<96 GB | ~15–27 tok/s | 32,768 at 48 GB; 131,072 at 64 GB |
| 96 GB+, the model fits in memory | ~20–32 tok/s | 262,144 tokens, the model's full window |

Context examples use decimal-GB memory simulations. A Mac's marketed capacity,
Metal limits and available memory can produce a different plan; `slotstream
doctor` shows the actual choice.

The middle rows are anchored on our M5 Pro's measurement; the top ends of the
last two rows come from a 128 GB M5 Max with a larger, manually chosen memory
target, and the last row is outside Slotstream's target range. These are
estimates, not limits. The hardware guide has the
[basis of each range](docs/HARDWARE.md#planning-ranges), every result
[measured on real Macs](docs/HARDWARE.md#results) with credits and test
conditions, and [every automatic memory plan](docs/HARDWARE.md#automatic-memory-plans).

### Recent prompt-processing results

The changes shipped in 0.2.23 shorten prompt processing and repeated-history
work. These measurements use the same 48 GB M5 Pro at a 10 GB target, with
three clean pairs per comparison:

| Workload | Matched control | Median times, control → enabled | Median paired time reduction |
|---|---|---|---|
| 16K inventory prompt, MTP on | Larger-read workspace policy off | 155.22 s → 53.94 s prefill | 65.40% |
| 2K prose follow-up, MTP off | Prefix checkpoints disabled | 30.73 s → 4.42 s request | 85.62% |

Both comparisons switch a feature off in the same tested binary. They measure
prompt processing or a cached follow-up, not an increase in reply tok/s or a
whole-release speedup. Times are arm medians; reductions are medians of paired
changes. The [hardware guide](docs/HARDWARE.md#recent-prompt-processing-results)
explains the fixtures and the latest audit's exclusions.

A fresh installed-release study used ordinary caching at the same memory
target. These are observed first-read ranges and median exact-repeat request
times across the prescribed request order, with capped replies:

| Prompt | Eligible first reads / repeats | First-read prefill range | Median repeated request |
|---|---:|---:|---:|
| 2K code | 4 / 3 | 13.86–28.11 s | 2.79 s |
| 2K prose | 3 / 3 | 14.91–26.51 s | 3.22 s |

The desktop load screen passed for the included observations, but several
runs had system swap-ins. Request history changed read batching, so these
results do not replace the general speed estimates or decode headline.
See the [measurement and its limits](db/records/measurements/release-prefill-2k-2026-09-22.md).

<a id="memory"></a>
<a id="context"></a>

### Memory and context

**Auto mode picks the memory target, cache size, speculative decoding and
context window for your Mac.** It takes the largest window in the table above
that still leaves room for speculative decoding and a complete conversation,
given the memory free at startup, without an unmeasured loss of useful
expert cache. `slotstream doctor` shows the choice and
why, and `--max-context 65536` sets a window yourself, up to 262,144 tokens.

**Starting a reply takes time.** Slotstream first reads your question and the
conversation history, which can take minutes for a long prompt. Follow-up
turns reuse unchanged history, a new conversation reuses the system prompt
earlier ones started with, and `serve --prefix-cache-dir` keeps long
conversations and shared system prompts on disk so that they survive a
restart. The hardware guide
has the [prompt-reading estimates](docs/HARDWARE.md#automatic-memory-plans)
for each memory size.

## Install

Open Terminal and paste this command:

```sh
curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh
```

Run the same command to update. If `slotstream` isn't found afterward, open
a new terminal window.

## Use it

Check your Mac, then ask for a first reply:

```sh
slotstream doctor
slotstream run --prompt "Why is the sky blue?"
```

<a id="downloading-the-model"></a>

`doctor` checks memory and disk space without loading the model. The first
`run` asks to download it, then prints a reply. This download can take hours,
but you only need to do it once. Interrupted downloads resume when you try
again. Follow the [step-by-step setup](docs/GETTING-STARTED.md) for more help.

<a id="chat-apps-and-the-api"></a>
<a id="pictures"></a>
<a id="coding-agents"></a>
<a id="docs"></a>

## Guides

| What would you like to do? | Guide |
|---|---|
| Chat in Open WebUI or another app | [Connect a chat app](docs/CLIENTS.md) |
| Ask about a picture | [Use an image](docs/GETTING-STARTED.md#ask-about-a-picture) |
| Start a coding agent on the model, in one command | [Use coding agents](docs/CODING-AGENTS.md) |
| Code with Claude Code | [Use Claude Code](docs/CLAUDE-CODE.md) |
| Code with Codex | [Use Codex](docs/CODEX.md) |
| Code with Pi or opencode | [Use Pi or opencode](docs/CODING-AGENTS.md#pi) |
| Work with files and tools through Hermes | [Use Hermes](docs/HERMES.md) |
| Code with fx, Vercel Labs' coding agent | [Use fx](docs/FX.md) |
| Fix a problem, move the model, or uninstall | [Troubleshooting](docs/TROUBLESHOOTING.md) |

Install chat apps and agents separately. They provide the interface and tools;
Slotstream runs the model. Keep its server running while a connected app uses it.
`slotstream launch claude` (or `codex`, `pi`, `opencode`, `hermes`) starts
that agent already connected, and starts the server in the background first
when none is running.

<a id="use-it-from-swift"></a>
<a id="testing"></a>
<a id="building-and-testing"></a>

For developers, the [engineering guide](docs/ENGINEERING.md) links to the
OpenAI- and Ollama-compatible API references, Swift library, command options,
build instructions, and tests. [Release notes](CHANGELOG.md) show what changed.

## How it works

Qwen3.8-Flash-Next is a *mixture-of-experts* model: generating each piece of
text uses only a subset of its expert networks. Slotstream keeps shared
weights in memory and reads the needed experts from SSD into a cache.
Frequently used experts stay in RAM, reducing repeated disk reads.

The whole model stays available even though it doesn't all fit in memory.
Slotstream chooses a memory target for your Mac and adjusts its cache as
other apps need room. Cache size changes speed without removing experts
from the model. The [engineering explanation](docs/ENGINEERING.md#how-it-works)
covers the implementation.

<a id="built-native"></a>

## Built native

Slotstream is one native Mac program: the command-line tool, the HTTP server,
the memory planner, the expert cache and the model itself are Swift on Apple's
MLX framework and Metal, with Slotstream's own Metal kernels compiled at run
time where a step needed one (the gated-delta recurrence, the selected
attention, the expert routing). No Python runtime or interpreter sits between
a request and the GPU.

That is deliberate. Built around one model on one kind of hardware, each layer
is tuned for the one below it: expert records are read from the SSD straight
into the cache slots the GPU computes from, the memory plan is checked against
what the process really uses, and a governor resizes the cache while other apps
need room. It is also why the engine ships as one file that installs with one
command, and why a Mac app such as Sevra can run it in process. The speed on
this page comes from measured mechanisms that this control allows, not from
the language itself; every published number has a recorded method in the
[measurements](MEASUREMENTS.md). The trade is that the engine runs only on
Apple Silicon; Windows and Linux are planned in Sevra with their own native
engines.

## Status and limits

- **Not optimized for 96 GB and larger Macs**, where the model fits in memory;
  see [Who it's for](#who-its-for).
- **One generation at a time:** connected apps share the same running model.
- **Conversation length is limited:** longer histories take more memory and
  time. Auto mode picks a [window for each memory size](#speed-by-memory), and
  the coding agent guides include the larger window agents need.
- **No broad benchmarks yet:** image input and tool calling have integration
  tests, but there is no image-accuracy benchmark or completed comparison with
  other models on the same Mac.

## FAQ

### Does it work offline?

Yes, after downloading the model. Inference runs on your Mac. Connected
agents may still use internet services for web searches or other tools;
their settings determine what those tools send.

### Why doesn't Slotstream use all of my RAM?

`--memory-gb 48` is a maximum process budget, not a promise to keep 48 GB
resident. Slotstream allocates the expert cache at load, while conversation
state and temporary work grow only when a request needs them. A short request
can therefore peak well below the target. The startup report shows the budget,
expert cache, runtime allowances and safety headroom separately.

Without an explicit target, Auto uses a **33 GB** base ceiling, or **34.6 GB**
with speculative decoding at the 32,768-token window. This measured default
leaves memory for other apps. To use more, stop any running server and preview
the plan without loading the model:

```sh
slotstream doctor --memory-gb 40
```

If it fits with headroom, `slotstream serve --memory-gb 40` uses that budget
with a fixed cache. In the current development version, use
`--memory-limit-gb` instead to choose an upper limit while the cache adapts
to other apps. Custom limits can exceed the automatic default; the Mac's
supported budget and available memory still bound actual use.
Auto keeps expert cache when a larger automatic context would trade it away
without a measured benefit. Use `--max-context N` when you explicitly want a
longer window. Leave room for macOS and other apps. See the
[memory options](docs/CLI.md#memory-options) for details.

### Is Slotstream the fastest way to run this model?

On a Mac that cannot hold the model, 16 to 64 GB, it is the way to run it at
all, and the engineering goes into making that fast. On 96 GB and larger Macs
the model fits in memory and engines that keep it there report faster replies.
See [Who it's for](#who-its-for) and
[related projects](docs/ENGINEERING.md#related-projects).

### Why is it written in Swift and not Python?

The engine needs direct control of memory, disk reads and the GPU, and it
has to ship as one file that a Mac app can call in process. Swift on MLX and
Metal gives that; a Python runtime would put an interpreter and a second
process in the way. The Python in the repository is tooling (the reference
model the port is checked against, benchmark drivers, release checks), and
none of it runs when you use Slotstream. See [Built native](#built-native).

### Will this wear out my SSD?

Generation reads the model files without rewriting them. macOS swap adds
writes when memory runs short. Automatic memory sizing helps, but a small
Mac or an oversized manual setting can still swap heavily. Servers that
`slotstream launch` starts, and `serve --prefix-cache-dir`, also save each
turn of a longer conversation to a cache folder, and the system prompts
conversations share, within a disk quota.

### Can I run it on Linux or Windows?

Support for AMD and NVIDIA on Windows and Linux is planned for Sevra.
It isn't available in the current Slotstream engine.

<a id="related-projects"></a>

### Can I use a different model?

Not with Slotstream today. Its loader and memory planner are built for this
model. See [related projects](docs/ENGINEERING.md#related-projects) for runtimes
with different model and hardware support.

## Why this exists

I wanted to run this model on my own Mac, but the standard loader exhausted
memory before producing a reply. Slotstream grew out of that experiment.
The [published measurements](MEASUREMENTS.md#m07--the-naive-path-fails-why-slotstream-exists)
include that failed load and the experiments that followed.

The project was also [discussed on Hacker News](https://news.ycombinator.com/item?id=49524447).
The questions and hardware reports from that discussion help guide the work.

## Support

[Report a bug](https://github.com/carloslfu/slotstream/issues/new) if something
doesn't work, or [share your Mac's results](docs/HARDWARE.md#how-to-measure)
to help others know what to expect. Reports are credited to their authors.
Code and documentation contributions are welcome; see
[Contributing](CONTRIBUTING.md) for the workflow.

## Grants and sponsors

<p>
  <a href="https://github.com/rauchg">
    <img src="https://avatars.githubusercontent.com/u/13041?v=4&amp;s=160" width="80" height="80" alt="Guillermo Rauch's GitHub profile photo"><br>
    <strong>Guillermo Rauch</strong>
  </a>
</p>

Slotstream was selected for [Guillermo Rauch's personal grants for foundational
open-source software](https://rauchg-oss-grants.vercel.app/).
Thank you for supporting its development.

## Who made this

I'm [Carlos Galarza](https://www.carlosgalarza.com). I build local AI and
make it run efficiently on the computers people already own. Slotstream is
the engine, written natively for the Mac and tuned as one system, and
[Sevra](https://www.sevrahq.com/) is the private AI app I'm building on it.
I also help teams run open models on their own hardware and debug agent
workflows. For help or consulting, [email me](mailto:carloslfu@gmail.com).

## Star history

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="docs/assets/star-history-dark.svg">
  <img alt="Slotstream GitHub star history, updated weekly" src="docs/assets/star-history.svg" width="960">
</picture>

The badge at the top shows the latest star count; this chart is updated weekly.

## License

Slotstream is [MIT-licensed](LICENSE). The model weights have their own
[Qwen community license](https://huggingface.co/pipenetwork/Qwen3.8-Flash-Next-MLX-4bit/blob/main/LICENSE).
See [credits](docs/ENGINEERING.md#credits) for the model and code this project builds on.


<!-- ===== docs/GETTING-STARTED.md ===== -->

# Get started with Slotstream

This guide takes you from installation to your first reply. You don't need
Python or a developer account. You'll copy a few commands into Terminal.

## Before you start

You need an **Apple Silicon Mac, macOS 14 or later, and about 110 GB of free
SSD space**. Choose Apple menu → About This Mac to check your chip and memory.
The installer has been tested on macOS 14 and 15; runtime testing so far is
on macOS 26.

Slotstream needs a Mac with at least 16 GB of memory. On an 8 GB Mac even the
smallest memory plan doesn't fit, so it refuses to start instead of swapping.
It is built for Macs with 16 to 64 GB, where the model cannot fit in memory;
on 96 GB and larger Macs it runs but is not optimized, see
[Who it's for](../README.md#who-its-for).
Less available memory can reduce the expert cache and slow replies; the chip,
SSD and workload also matter. See the [hardware guide](HARDWARE.md)
for measurements from real Macs and estimates for each memory size.

Slotstream currently runs one model, Qwen3.8-Flash-Next, on Apple Silicon.
Windows, Linux, and other models are not supported by the current engine.

## Install

Open Terminal: press **Command+Space**, type `Terminal`, and press **Enter**.
Paste the following command and press **Enter**:

```sh
curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh
```

This installs the latest release. If a later command says `slotstream` is
not found, open a new Terminal window and try again.

Check your Mac before downloading the model:

```sh
slotstream doctor
```

This shows available disk space, the planned memory use, and estimated speed.
It doesn't download or load the model. Check that you have enough free disk
space before continuing. Close memory-heavy apps if the report says memory
is tight.

## Ask for your first reply

Run:

```sh
slotstream run --prompt "Why is the sky blue?"
```

On first use, Slotstream shows the model download size, destination, and
free space, then asks for confirmation. Press **Enter** to accept, or type
`n` and press **Enter** to decline.

Once the download finishes, Slotstream loads the model, processes your
question, and prints the reply. The first reply may take a while to begin;
Terminal shows progress. When your normal terminal prompt returns, the
command has finished. Run it again with a different question in the quotes.
Each `run` command starts a fresh conversation.

For an ongoing conversation, use a [chat app](CLIENTS.md) or
[Hermes](HERMES.md).

## Downloading the model

You only need to download the model once. To download it before asking a
question, run:

```sh
slotstream pull
```

Slotstream downloads **88.3 GB** of compressed files from
[Hugging Face](https://huggingface.co/carloslfu/Qwen3.8-Flash-Next-MLX-4bit-Slotpack)
and restores the **105.3 GB** model on your SSD. No Hugging Face account is
needed. Afterward, the model runs offline; any web tools in a connected
agent still need their own internet access.

The transfer alone is estimated at about 2 hours at 100 Mbps or 8 hours at 25 Mbps.
Connection overhead and processing add to that time. You can stop with
**Control+C** and run the same command later to resume. Downloaded files are
checked for corruption automatically. Since 0.2.19 `slotstream pull` also
downloads a small forecast-correction file (37.5 MB) that makes replies
faster; the download `run`, `serve` and `launch` offer on first use includes
it since 0.2.25. If
`slotstream doctor` says it is missing, run `slotstream pull` once more to
fetch it.

For an interrupted or damaged download, moving the files to another disk,
or reclaiming disk space, see [Troubleshooting](TROUBLESHOOTING.md).
Developers can read about compression and verification in the
[download format notes](DOWNLOAD-FORMAT.md).

## Ask about a picture

Put a picture named `cat.jpg` in your Downloads folder, then run:

```sh
slotstream run --image "$HOME/Downloads/cat.jpg" --prompt "What is in this picture?"
```

Replace `cat.jpg` with your file's name. Keep the quotes around the path if
it contains spaces. Images need extra memory, so Slotstream may refuse an
image when there isn't enough room. The image features have been tested,
but general image-answer accuracy has not been benchmarked.

## Connect an app or agent

- [Coding agents](CODING-AGENTS.md): `slotstream launch claude` (or `codex`,
  `pi`, `opencode`, `hermes`) starts the agent connected to Slotstream, and
  starts the server in the background first when none is running.
- [Hermes](HERMES.md): chat and work with files and tools through a local model.
- [Open WebUI and other chat apps](CLIENTS.md): use a chat interface with Slotstream.
- [fx](FX.md): use a coding agent. Read its permission and long-session limitations before starting.

Install the app separately, then follow its connection guide. Keep the
Slotstream server running while the app uses it. Slotstream runs one model
process at a time, so stop a server before using `slotstream run`: press
**Control+C** in its window, or run `slotstream stop`, which also stops a
server `slotstream launch` started in the background.

## Update or get help

Run the installer command again to update Slotstream. Check the installed
version with `slotstream --version`. If a server was running during the
update, stop it with **Control+C** and start it again to use the new version.

See [Troubleshooting](TROUBLESHOOTING.md) for slow replies, startup errors,
downloads, and uninstalling. If you need to report a problem, include your
Mac model, memory, Slotstream version, the command you ran, and the error
message. Remove any private file contents or credentials from the report.

## Common questions

**Will this wear out my SSD?** Generation reads the model files without
rewriting them. macOS swap adds writes when memory runs short. Automatic
memory sizing helps, but a small Mac or an oversized manual setting can
still swap heavily.

**Can I use a different model or another operating system?** The current
engine supports only `qwen3.8-flash-next:4bit` on Apple Silicon. Windows and
Linux support for AMD and NVIDIA is planned for Sevra. See
[related projects](ENGINEERING.md#related-projects) for other runtimes.


<!-- ===== docs/ENGINEERING.md ===== -->

# Engineering notes

Technical background, performance measurements, and development references
for Slotstream. For installation and a first reply, start with
[Get started](GETTING-STARTED.md).

## References

| Topic | Documentation |
|---|---|
| Commands and configuration | [Command reference](CLI.md) |
| HTTP integration | [API reference](API.md), [Hermes notes](HERMES-NOTES.md), [fx protocol](FX.md#protocol-reference) |
| Embedding in an app | [Swift library](LIBRARY.md) |
| Sevra native application | [Mac development and native philosophy](SEVRA-MAC.md) |
| Build, test, and contribute | [Testing](TESTING.md), [Contributing](../CONTRIBUTING.md) |
| Download internals | [Slotpack format](DOWNLOAD-FORMAT.md) |
| Expert lookahead | [How it works, experiments and results](EXPERT-LOOKAHEAD.md) |
| Design and evidence | [Design and plan](../PLAN.md), [Measurements](../MEASUREMENTS.md), [Hardware reports](HARDWARE.md) |
| Security and releases | [Security](../SECURITY.md), [Changelog](../CHANGELOG.md), [Latest release](https://github.com/carloslfu/slotstream/releases/latest) |

The public [db.md store](../db/DB.md) holds the measurements, claims, plans,
and raw runs. `PLAN.md` and `MEASUREMENTS.md` are generated from its records.
For AI agents, [llms.txt](../llms.txt) is the index and
[llms-full.txt](../llms-full.txt) combines the documentation.

## How it works

Qwen3.8-Flash-Next is a *mixture-of-experts* model: each token uses only a
small subset of its expert networks. Most of its storage is 68 GB of routed
experts and a 32 GB n-gram lookup table. The 3.8 GB shared part stays in RAM.

slotstream reads experts from SSD into a fixed pool of cache slots. All
48 layers share that pool, so layers that need more slots can borrow them
from others. Keeping more experts in RAM reduces disk reads. It changes
speed without changing the expert weights used in the computation.

A memory-mapped file alone doesn't solve this in MLX, Apple's machine-learning
framework. The tested expert-gather operation materialized every expert in a
layer, even though the token needed only a few. Explicit slots keep those
reads and allocations under control. The [design](../PLAN.md) covers the details.

That design targets Macs that cannot hold the model, 16 to 64 GB. On 96 GB
and larger Macs the model fits in memory, and the streaming machinery is
overhead that an engine keeping the model resident does not pay; see
[related projects](#related-projects) and [who it's for](../README.md#who-its-for).

<a id="native-stack"></a>

## Native stack

Slotstream is Swift from the command line down to the GPU. No interpreter
runs on the request path.

| Layer | Implementation |
|---|---|
| Command-line tool, HTTP server, the OpenAI-, Ollama- and Responses-compatible endpoints and the fx gateway | Swift: `Sources/slotstream-cli`, `Sources/Slotstream/Server.swift` and the dialect files beside it |
| Memory planner, expert store and slot cache, governor, sampler, speculative decode, prefix cache | Swift: `Sources/Slotstream` |
| Tensor operations | [mlx-swift](https://github.com/ml-explore/mlx-swift), Apple's MLX, with its prebuilt `mlx.metallib` beside the binary |
| Gated-delta recurrence, selected attention, partial rotation, block and router selection | Slotstream's own Metal kernels, compiled at run time through `MLXFast.metalKernel` |
| Tokenizer | [swift-transformers](https://github.com/huggingface/swift-transformers) |
| Slotpack download decoder | C, `Sources/CSlotpack`, with no external codec |
| Expert and n-gram reads | `pread` into a fixed slot pool over parallel lanes, with lane counts chosen by measurement |

The Python under `Tools/` never runs in the product. It is the reference
implementation the Swift port is checked against layer by layer, the numpy
sampler oracle, the trace simulators and benchmark drivers, the model-free
gates and the release checks. The [testing guide](TESTING.md) lists them.

The stack is native on purpose. The engine exists for one model on one kind
of hardware, and each layer is tuned for the layer below it: expert records
are read from the SSD into the slots the GPU computes from, process memory is
accounted from real CPU and GPU allocations, and the governor resizes the
cache under memory pressure. The same design ships as one binary, installs
with one command and can be called in process from a Mac app; the
[Swift library](LIBRARY.md) and the [Sevra Mac notes](SEVRA-MAC.md) describe
that use.

What the native stack does not claim: the measured gains on this page come
from the mechanisms named with them, the decode lookahead, the corrected
expert forecast, speculative decoding, the prefix cache, the GPU keepalive and
direct demand reads. Warm decode is
dominated by SSD reads and GPU waits, so the work goes to fewer reads and
more overlap rather than host-side micro-optimization
([decision](../db/records/decisions/decode-host-time-is-waiting-not-graph-construction.md)).
The engine runs only on Apple Silicon, and there is no completed same-Mac
comparison with other engines; see [related projects](#related-projects).

## Speed

On the 48 GB M5 Pro:

| Measurement | Result |
|---|---|
| Reply generation after the cache warms up | ~12 tok/s |
| Reply generation with speculative decoding and the 0.2.16 decode lookahead, 20 GB target | 13.47 tok/s, 1.11x faster than without the lookahead |
| Reply generation with the 0.2.19 corrected forecast, 22 GB controlled benchmark (smaller prompt passes, prefix caching off) | 15.86 tok/s, 1.10x faster than the 0.2.18 forecast |
| Engine load in the original experiment, before processing the prompt | ~2 s (historical) |
| Planned memory with automatic sizing | 32 GB (estimate) |

That memory figure is the historical planner estimate, not a measurement of
the corrected kernel lifetime peak. Older reported values can miss GPU memory
freed before the observation. Current usage, lifetime peaks and request samples
are explained in the [memory controls](CLI.md#memory-options).
The engine-load timing also comes from an early measurement; it excludes the
current CLI's full weight-verification step and is not a current cold-start
or first-answer estimate.

**Long prompts take time before the first reply token.** Processing the prompt
is called *prefill*. The estimates for this Mac are about 9 s for 2,000 tokens
and 39 s for 8,000. Ordinary prose can take longer than the synthetic prompt
used by the estimator. `slotstream doctor` shows estimates for your memory
plan, and the terminal prints progress during long prompts.

The conversation cache avoids processing unchanged history again. In a
historical eight-turn test at a 16 GB target, the last turn started replying
after 6.0 s with reuse, compared with 25.8 s without it. The current aligned
cache accepts only checkpoints compatible with the incoming prompt's compute
passes and backend, preserving exact cached-versus-fresh results for that
computation. Use `--no-prefix-cache` to measure the cost of processing the
whole prompt. See the [current reuse qualification](../db/records/measurements/prompt-speed-qualification-2026-09-21.md)
for checkpoint and app-restart evidence.

### Prefill and speculative decode measurements

The prefill sweep groups work by expert and reads weights in contiguous
batches. On the development Mac, at a 16 GB memory target, an 8,000-token
prompt improved from 91 → 184 tok/s and prose from 66 → 140 tok/s. At the
8.1 GB floor, prefill improved from 51 → 93 tok/s. The planner estimates
about 220 tok/s for a 4,096-token pass on the M5 Pro. These results depend on
the prompt and configuration; they aren't measurements on a 16 GB Mac.

Speculative decode uses a small draft head to propose tokens for the main
model to verify. The current operating choice is two drafts (default 2).
The [adoption decision](../db/records/decisions/draft-depth-defaults-to-two.md)
records the workload tradeoff and the limits of the recent comparison.
In the historical one-draft test, the draft was accepted 86% of the time.
At a 28 GB memory target, that one-draft configuration improved greedy decode
by ×1.24 (10.3 → 12.8 tok/s); the improvement was ×1.18 with default server
sampling.

`--mtp auto` enables this when the expert cache can still hold 28 experts per
layer after the head's charge, before the separate lookahead reservation, a
12 GB target at the 32,768-token window. Availability and context can change
activation. The head's 512 experts are 1.42 GB of its 1.47 GB. On a cache of
76 experts per layer or more after the full 1.6 GB charge they stay resident.
Below that the head reads them from the SSD through a 64-expert cache of its
own, a 0.4 GB charge, and the main cache keeps the other 1.2 GB. A draft row
routes to ten experts; about half are already in that cache. At a 12 GB
target this made the head 1.23x faster than plain decode with the lookahead,
where a resident head only tied, and the head now runs on 24 GB Macs. The
floor was 120 until 0.2.16 and 76 until now; on 0.2.14, two drafts decoded
31.7% faster than plain decode on the same memory at 76 per layer. The
automatic ceiling is 34.6 GB with the head enabled at the 32,768-token window;
larger windows add their context charges. `--mtp off` disables the head.

With the head on, 0.2.16 also runs the decode lookahead, and without the head
it now runs in plain decode too, where it made plain decode 1.11x faster at a
10 GB target. After each layer, the
router of the layer two ahead runs on the current hidden state, and the experts
it picks are read from the SSD straight into cache slots before that layer asks
for them. FP32 copies of the router weights save a conversion on every routing
call, and the GPU is drained every four layers instead of every layer, with
each forecast riding the next routing readback. On twelve held-out prompts at a
20 GB target with two drafts, decode was 1.11x faster than the previous default
(11.79 to 13.47 tok/s median) with identical output. In separate attribution
runs on the tuning prompts, the router copies and fewer drains added
about 2% each over prefetch alone. Those component results are specific to
that workload and are not separate held-out speedups. The
[decision](../db/records/decisions/decode-lookahead-default-with-the-draft-head.md)
records its 373 MiB charge, overrides and limits.
The short [expert lookahead guide](EXPERT-LOOKAHEAD.md) explains the mechanism
and the experiments that led to it.

### GPU keepalive and direct demand reads

Streamed decode is stop-and-go. At every layer the host reads the routing
back, reads the experts the cache is missing and only then submits the next
burst of GPU work, so a one-token pass is a few hundred short command buffers
with the GPU idle in between. An idle Apple GPU lowers its clock and starts
the next buffer late. While a request generates, Slotstream now keeps the GPU
busy with a one-thread kernel on its own command queue. The kernel computes
nothing and touches no model memory, so outputs are unchanged.

Cache misses used to be read into staging arrays and then scattered into the
cache on the GPU, one more dispatch and wait per layer. They are now read into
host memory and copied straight into their cache slots: the same bytes in the
same place, without the scatter.

On the development Mac, paired and interleaved with identical output, the two
together made decode 1.28x faster at a 10 GB target without the draft head
and 1.22x faster at 22 GB with the draft head and lookahead, counting only
pairs with no swap activity. The keepalive costs power: energy per generated
token rose 7% at 16 GB, so `--gpu-keepalive auto`, the default, runs it only
on AC power outside Low Power Mode. `--gpu-keepalive off` and
`SLOTSTREAM_OPT_DIRECT_DEMAND=0` restore the previous behavior. The
[measurement](../db/records/measurements/decode-perf-2026-09-24.md) has every
comparison, both screens and the ideas that did not help.

[MEASUREMENTS.md](../MEASUREMENTS.md) includes the configurations, comparisons,
and failed experiments behind these results.

## Context

**Prompt, conversation history, images, and reply share one window, which auto
picks for each Mac.** It takes the largest of 32,768, 65,536, 131,072 and
262,144 tokens that keeps speculative decoding as the 32,768-token plan has
it, including a draft head's resident experts, retains one complete
conversation and adds at most 10% to the planner's estimate for a typical
request. A flat estimate beyond its measured cache range is not evidence that
extra cache has no value: auto declines reductions in that range and reports
their cost as unmeasured. The same rule applies to a busy startup.
`serve --max-context 65536` fixes the window Hermes uses, and any size
up to the pinned model's 262,144 tokens is accepted; requests with images stay
within 65,536. The planner charges extra state and transient memory before
allocating the expert cache, and the automatic ceiling rises by the window's own
charge. Native runs on the development Mac cover 65,536 tokens and, since
0.2.17, 131,072 tokens with and without the draft head, both inside their memory
plans; 262,144 tokens is planned from the same ledger without a native run. The
long-context qualification is a capacity and memory check, not a long-context
answer-quality benchmark.

At 32,768 tokens, the estimated wait before the first token is about 3.0 min for
the 48 GB M5 Pro plan and 6.4 min for the 16 GB plan. The latter comes from
the M5 Pro's curve; a slower SSD can take longer. Follow-up turns reuse
unchanged history while it remains cached.

The main sequence cache uses about 27 KiB per allocated token of capacity,
rounded to allocation steps. Recurrent state, retained conversations, draft
state and transient workspace are additional charges, so this is not the
whole process cost per input token. Long prompts also cost processing time.
Slotstream reduces the prefill batch size as context grows to keep temporary
memory within the measured range.

To measure a long prompt on your Mac, stop any running server, then run:

```bash
slotstream context-check --tokens 16384
```

It reports time, speed, and peak memory, checking available memory between
passes. `slotstream prefill-schedule --chunk 4096 --tokens 32768` shows the
batch schedule without loading the model.

## Memory

Memory defaults follow the [measured operating policies](../db/records/design/measured-operating-policies.md).
That contract distinguishes model facts, safety and qualification limits,
operating defaults, and bounded estimates. Tuning choices carry evidence,
scope and revision criteria; maintaining those choices is part of the engine.

By default, slotstream chooses a memory target for your Mac and prints it at
startup. It takes the lowest of 33 GB, 70% of RAM, and 2 GB below the Metal
working-set limit, then reduces that target if other apps are using memory.
The draft head raises the base ceiling to 34.6 GB at the 32,768-token window;
the larger windows auto picks on bigger Macs add their context charges. See [Speed](#speed).

The 33 GB ceiling is a conservative default based on development-Mac
measurements. Those tests showed diminishing speed gains as
the expert cache grew. This supports a conservative default; it does not
establish an optimum for every Mac or workload. We'll adjust the default as
real measurements show a better tradeoff. The historical larger-target sweep
inspected planner estimates, which hold flat beyond the verified cache sizes;
it was not a benchmark of those larger allocations. See the
[cache measurements](../db/records/measurements/warm-decode-re-anchored-and-the-live-governor-finally-observed-2026-08.md)
and [sizing interpretation](../db/records/measurements/automatic-memory-default-evidence-scope-2026-09-09.md).

The [community M5 Max cache sweep](HARDWARE.md#does-more-memory-help) reports
faster replies with larger manual targets on the same machine. Auto has not
been calibrated to that hardware, and its fixed ceiling must not be read as
the maximum useful allocation.

The chip and SSD still matter. The plan uses decimal GB, so a Mac sold as
48 GB appears as about 52 GB in its device line.

While the server runs, it checks memory pressure every 15 s and resizes its
cache between requests. It gives memory back under pressure and grows again
when space is available. The cache-size and resize gates check byte-identical
greedy output with the other generation settings fixed. Changing the total
memory target can also change prefill grouping or enable speculative decoding;
those are separate changes, not part of that equality claim.

Warm growth also needs room for temporary replacement tensors. The governor
checks this extra allocation against both the process target and current
system availability. If it cannot fit, the existing warm cache stays usable
and growth waits. The copy preserves slot positions and appends capacity one
tensor at a time, without gathering another copy of all occupied slots.

To set a memory target yourself:

```bash
slotstream doctor --memory-gb 16
slotstream serve --memory-gb 16
```

`--memory-gb` sets the total process target, with a minimum of 8.1 GB for the
default text context; larger windows and resident components need more room.
An explicit target disables automatic cache resizing, while loading and
request-memory safeguards remain active. Preview it before starting.
The development version also offers `--memory-limit-gb`: an upper process
budget with automatic cache resizing. It can exceed the default model ceiling
while remaining bounded by supported GPU/system headroom and live memory.
The chosen limit is retained across shrink and recovery. Diagnostics and
budgeted startup share the same feasibility check.
See the [memory options](CLI.md#memory-options)
for the other controls and their precedence.

## Status and limits

- **Hardware:** the development measurements use a 48 GB M5 Pro. Community
  reports cover other Macs; several memory tiers remain estimates. See
  [Hardware measurements](HARDWARE.md).
- **Concurrency:** one model process per user, with one generation at a time.
- **Compatibility:** macOS 14/15 runtime testing is still needed. Tool calling
  works through OpenAI chat completions and the fx gateway; the Ollama subset
  doesn't support it.
- **Vision:** the image encoder is checked against an independent reference
  and the APIs are tested with images. There is no general vision accuracy
  benchmark or comparison with another runtime yet.

## Why this exists

I have a 48 GB MacBook Pro and wanted to run this model on it. The stock loader
pushed the machine into 48 GB of swap before producing a token. I built
slotstream to keep the shared weights in memory and stream the experts from
SSD, with a cache that leaves room for other apps. That is still the target:
Macs that cannot hold the model, 16 to 64 GB.

The [measurements](../MEASUREMENTS.md#m07--the-naive-path-fails-why-slotstream-exists)
start with that failed load. The launch was also
[discussed on Hacker News](https://news.ycombinator.com/item?id=49524447), with
227 points and 114 comments, reaching No. 1 on Show HN and No. 8 on the front
page on September 1, 2026.

## Related projects

If your Mac holds the whole model, 96 GB and up, an engine that keeps it in
memory is the faster choice. [MTPLX](https://github.com/youssofal/MTPLX) runs
this model with native speculative decoding on such Macs and publishes its
measurements. Slotstream is built for the Macs below that line; see
[Who it's for](../README.md#who-its-for).

Other projects approach local inference with different models, hardware,
and memory strategies:

- [llama.cpp](https://github.com/ggml-org/llama.cpp): inference across many
  models and CPU/GPU backends.
- [Rapid-MLX](https://github.com/raullenchai/Rapid-MLX) and
  [oMLX](https://github.com/jundot/omlx): local inference servers for Apple Silicon.
- [Whallm](https://github.com/yanun0323/Whallm),
  [SwiftLM](https://github.com/SharpAI/SwiftLM), and
  [Mference](https://github.com/NeelM0906/Mference): other approaches to running
  large models on Macs.
- [mlx-flash](https://github.com/matt-k-wong/mlx-flash),
  [samosa-chat](https://github.com/deepanwadhwa/samosa-chat),
  [deepseek-v4-flash-mlx](https://github.com/ssd-moe/deepseek-v4-flash-mlx),
  [streamlx](https://github.com/srcterm/streamlx), and
  [mlx-moe-offload](https://github.com/huckiyang/mlx-moe-offload): related work
  on inference with limited memory.

There isn't a completed comparison on the same Mac yet. Each project's
reported speeds use its own setup and shouldn't be read as a ranking.

## Star history

The [README](../README.md#star-history) shows the star count and history.
The chart is updated weekly by this repository's
[workflow](../.github/workflows/star-history.yml).

## Image memory and measurements

Each resized image uses up to 2,304 tokens of the conversation's context.
The image encoder, or *vision tower*, loads on the first image and reserves
0.9 GB inside an auto or `--memory-gb` process target, reducing expert capacity
as needed. An explicit pool-size setting retains its pool and adds the tower
to the expected footprint. Image pixels and attention also need workspace;
the server rejects the request if the budget or real headroom is insufficient.
Use `slotstream serve --vision off` to disable images.

In a measured conversation, the first image turn took 15.4 s and the
follow-up took 1.8 s because its image state was reused. This tests the image
path and reuse; the project has not measured general image-answer accuracy.

## Credits

MIT. [`Sources/Slotstream/Vendored/GatedDelta.swift`](../Sources/Slotstream/Vendored/GatedDelta.swift) is ported from
[mlx-swift-lm](https://github.com/ml-explore/mlx-swift-lm) (MIT).
[`Tools/reference/`](../Tools/reference/) includes the community `qwen4_exp.py` used as the test
reference. Model weights come from
[pipenetwork/Qwen3.8-Flash-Next-MLX-4bit](https://huggingface.co/pipenetwork/Qwen3.8-Flash-Next-MLX-4bit)
and remain under the [Qwen community license](https://huggingface.co/pipenetwork/Qwen3.8-Flash-Next-MLX-4bit/blob/main/LICENSE).


<!-- ===== docs/SEVRA-MAC.md ===== -->

# Sevra for Mac

Sevra is being built as a native personal AI application around Slotstream.
The development app lives under `apps/macos`. This is ongoing implementation,
not an announced alpha or a supported installer.

## Native philosophy

Product behavior is shared through Markdown/text specifications, schemas and
declarative fixtures. Each platform owns its UI, runtime, memory, tools,
permissions and inference integration. Independent upstream libraries are
allowed; sharing application source is not a requirement.

Mac uses SwiftUI and AppKit/TextKit over a Swift application runtime. That
runtime calls Slotstream in process and uses the official bundled dbmd tool
for deterministic file-database operations. Metal and dbmd keep their own
implementations. The native interface needs no web server or account.
Slotstream itself is native Swift on MLX and Metal end to end
([native stack](ENGINEERING.md#native-stack)), which is what makes the
in-process call possible.

The [engineering specification](../db/records/design/sevra-spec/overview.md)
owns the detailed contracts. Its
[implementation ledger](../db/records/design/sevra-spec/implementation-status.md)
distinguishes code, observed behavior and unpassed qualification. Windows and
Linux have independent implementation plans; this change builds the Mac app.
Cloud development, paid sync and remote inference remain deferred until users
ask for them.

## Build and run locally

Use a Mac development checkout with Swift Command Line Tools, the existing
Slotstream dependencies and Metal resource, and the official dbmd executable.
Set `SEVRA_DBMD` to the reviewed absolute executable path if it is not installed
at the default location used by the build script.

```bash
bash Tools/build_sevra_mac.sh
open .build/Sevra.app
```

The script copies the runtime dependencies into the development bundle and
ad-hoc signs it. Developer ID signing, notarization, the installed updater and
clean-machine qualification are separate work. A successful ad-hoc build does
not establish a trusted public distribution.

By default the app opens `~/Sevra/Home`. For disposable development data, launch
the executable directly with a dedicated Home:

```bash
SEVRA_HOME="$HOME/Sevra/DevelopmentHome" .build/Sevra.app/Contents/MacOS/Sevra
```

The current integration verifies and uses the existing Slotstream model
location. Settings can check that installation or explicitly download, resume
or repair its pinned files. Opening Sevra starts no download. Fresh-download
and cancellation qualification remain separate from the installed-model test. No model is loaded merely to inspect history or edit a draft. Before
running real inference, follow the repository's memory-safety rules and stop
any other model process you own. Never stop somebody else's process without
coordination. The bounded functional-test configuration is documented in the
[Mac baseline](../db/records/design/sevra-spec/mac-platform.md); it is not yet a
qualified automatic product recommendation.

The current engine keeps its model-process lock until the process exits.
After running inference in Sevra, quit Sevra before starting another model
process; the Unload control alone does not release that process-wide lock.

## Exercise the local workflow

Open a new thread and attach files or folders with the paperclip, the
**File → Attach Files…** command (⌘⇧A), drag and drop, or paste. A thread holds
up to eight attachments, each shown as a chip above the composer. Ask Sevra
about them, or ask for a cited briefing and review the complete document before
choosing **Save document**. The model is told each attachment's name, kind and
access, so a question such as "what is this?" reads the attached file instead
of asking what you mean. Saved artifacts use create-only publication inside
the Home.

### What Sevra can read

- **Text and code:** Markdown, plain text, CSV, JSON, logs and common source
  files up to 8 MB each.
- **Documents:** PDF, Word (`.docx` and `.doc`), RTF, OpenDocument text, Excel
  (`.xlsx`) and EPUB, up to 64 MB. Sevra searches their text and cites the page
  a PDF excerpt came from.
- **Scans and images:** PNG, JPEG, HEIC, TIFF, GIF, WebP and BMP, plus PDF pages
  without a text layer. Sevra recognizes their text on this Mac when a page is
  read, at most 40 pages per request, and marks such citations as recognized
  text.
- **Folders:** live access without an upfront scan or a fixed file-count limit.
  Sevra browses subfolders, finds filenames and searches content as needed.
  New files, renames and edits appear on subsequent operations. Hidden entries
  and symbolic links are excluded. Broad searches skip dependency folders such
  as `node_modules`; Sevra can inspect a specific dependency folder when needed.

Long listings and searches return bounded pages. Sevra can continue from where
it stopped, including further matches in the same file. If a directory changes
between pages, Sevra must restart the operation with current names. Search reports
skipped or unreadable entries instead of treating them as searched. An attachment stays
usable regardless of the size of its tree. Access remains limited to the files
or folders you selected.

Without an attachment, Sevra explains how to add one with the paperclip. Thread
only affects remembered context; attached files remain available in that mode.

Password-protected, damaged and oversized documents are refused with a reason.
Sevra reads the text inside a document, not its layout; tables and multiple
columns come out as plain text.

Rich documents are opened by `sevra-extract`, a small helper that ships inside
the app. Each document gets a fresh helper process that receives only the
document's bytes. Before it parses anything, the helper puts itself in a macOS
sandbox that denies reading your files, writing outside its own work folder,
all network access, launching programs and the window server. Text recognition is
split in two. A strict helper turns the file into plain grayscale pixels, and a
second helper that may use the GPU and Neural Engine only ever sees those
pixels. Sevra stops a helper that runs too long, prints too much, uses too much
memory or is cancelled, together with anything it started.

### Reviewed changes

Each chip has an access menu. **Read only** is the default. **Can propose
changes** lets Sevra stage new files and edits in that folder or file. Staged
changes are never written during a response. When it ends, **Review changes**
shows every file with a line-by-line diff, and **Show complete text** shows
exactly what will be written.

**Write Files** applies the whole set. Sevra first checks that each file is
unchanged since it was read and saves the previous version in the Home. It then
replaces the file in one atomic step, keeping its permissions and Finder tags.
A file edited after the review, replaced by a symbolic link, or hard-linked
elsewhere is left alone and reported. If Sevra closes while writing, the
review reports which files were written; nothing is replayed.

**Undo Changes** restores the previous versions and moves new files to the
Trash, where **Show in Trash** finds them. A file edited since Sevra wrote it
is left as it is. Sevra never changes shell scripts, HTML, SVG or notebooks,
and keeps access only while a folder is attached. After a restart, attach the
folder again to write or undo. Incognito threads can read but not change files.

### Knowledge bases

A folder with a `DB.md` file attaches as a db.md knowledge base. Sevra searches
and queries it through the pinned dbmd tool, which runs inside the same kind of
sandbox, confined to that store. With changes allowed, Sevra can stage new
records and body edits, appends and replacements, for review. Records are
written through dbmd, so indexes stay current; db.md files each new record by
its type, and the review reports where it went. Frontmatter, the store's
`DB.md`, its indexes and log, and anything under `sources/` are never edited.
Undo restores the records and rebuilds the index.

### Skills and mini-apps

The sparkles button in the composer lists skills. **/app** asks Sevra to build
or change a mini-app, and **/skill** saves a repeatable workflow. Your own
skills appear after you approve them; type `/name` to use one. A skill only adds
instructions to a request. It never grants access to files, and anything it
proposes still waits for your review.

A mini-app is one HTML file with its own CSS and JavaScript. Its review shows
the data it asks for and how many records each collection already holds, notes
about what it tries to do, its source, and a live **Try it** preview backed by
scratch data that is discarded. **Turn On App** publishes that exact version and
grants the listed access on this Mac. Open apps from **Apps & Skills** (⌘2),
where you can switch versions, turn an app off or remove it. Removing an app
keeps its versions and data in the Home.

Apps store records through `window.sevra` (`list`, `get`, `create`, `update`,
`archive`, `restore`) in named collections. Each record is a db.md file under
`db/records/app-data/` in your Home, with a revision number that prevents lost
updates. An app sees only the collections its active version declared and you
approved. Apps that use the same collection share its records. Sevra assigns
record ids, tells an app about changes made elsewhere with a `sevra-change`
event but never about its own saves, and slows down an app that saves too often.
A save based on a stale revision is refused instead of overwriting newer data. A
changed app file, a restored Home or a Home file edited outside Sevra stops
access until you act.

Each app runs in its own web view that serves only its approved bytes. It has no
network: loads are blocked by content rules and a content security policy, and
anything that would connect anyway meets a proxy that refuses it. Peer
connections, link preconnects and DNS prefetching are switched off, and the app
does not run at all unless WebKit confirms they are. Navigation, popups,
downloads and camera or microphone access are refused, and browser storage ends
with the view. Opening a web link from an app asks you first. Backups carry app
versions and data but never their access grants, so a restored app stays off
until you turn it on.

Settings exposes System, Light and Dark. System follows macOS; overrides
persist. Saved drafts and threads reopen with the same Home. Incognito is a
separate session and cannot create persistent memories, staged artifacts,
file changes, apps or skills.

After a completed Home exchange, **Continue in thread** opens a work thread
with that exchange visibly labeled **From Home**. The original messages stay
in Home and are quoted by reference. Further messages in Home do not become
part of the continuation. **Open thread** returns to the existing continuation;
repeated clicks do not create duplicates. **View in Home** returns to Home.
The continuation keeps the memory scope, and both composers retain their own
drafts. File attachments and approval authority stay with the original thread;
attach a file again to let the continuation read it. Search, copy and export
include the visible quoted exchange. Forget still suppresses its source from
future AI context without erasing the inspectable history.

## Draft saving and regression checks

The composer serializes draft writes and tracks each acknowledged revision
independently of the display refresh. Typing during a save stays in the editor;
the writer drains the latest text before switching threads or closing. Send
accepts the message and updates its draft in one durable transaction. A pending
save cannot restore a sent message. Incognito closing drains pending work
before removing the thread.

Recoverable revision changes are handled quietly. A real competing draft offers
both versions for an explicit choice. Save failures keep the text visible, offer
Retry Save and Copy Draft, and prevent closing with unpersisted edits. Send
retries reuse an acceptance nonce so an uncertain result cannot create a
duplicate message. Successful recovery clears its own warning.

Journal uses the same draft coordinator. Unsaved entries survive reopening,
and Save entry commits the entry and remaining draft together. Repeated clicks
and uncertain-result retries keep one accepted entry. Text typed while saving
remains available for the next entry. Model-file verification also checks for
cancellation between bounded reads, so Stop does not wait for a complete hash
scan.

Run the Mac development regressions without loading a model:

```bash
bash Tools/check_sevra_mac.sh
```

The script builds the app and local CLI, then checks the production composer
state machine with delayed and failing storage, native Markdown presentation,
and the runtime using scripted inference and real dbmd persistence. Disposable
Homes exercise restart, crash recovery, tool boundaries and data isolation.
The runtime suite also covers document reading, the helper sandbox, reviewed
changes, crash recovery, knowledge bases, skills and mini-app data. It finds
`sevra-extract` beside itself, or through `SEVRA_EXTRACT`. The script also runs
two offscreen view checks. The mini-app check loads a hostile page into the
production app host and requires that a local listener sees no connection and
no datagram, then clicks through a file-change review and an app review:

```bash
bash Tools/check_sevra_apps_ui.sh
```

The thinking-controls check can also run on its own:

```bash
bash Tools/check_sevra_thinking_ui.sh
```

That check renders the production views over the scripted engine in a scratch
Home, finds each control by its rendered label and clicks it: Think longer on
and off, Send, the live clock and the last lines of the thought, the details
popover with the working notes, Answer now, the thinking line above the reply,
the optional speed line with the live writing speed, and the typical-time
hint, in light and dark appearance. Its
window is ordered far outside every display and the process never activates,
so nothing appears on screen; snapshots land in `.build/sevra-thinking-ui/`.
Native UI walkthroughs, VoiceOver passes and real-model checks remain separate
evidence; this suite does not qualify every OS, input method, accessibility
mode or release.

## Home backup and recovery

Settings → General → **Back up Home…** saves a verified folder containing saved
conversations, drafts, memories, journal and owned files. **Reveal backup** opens
its location. Attached source folders and model weights remain separate; a
retained citation is only an excerpt of its original source. Device credentials,
source grants and Incognito content are excluded.

**Restore backup…** creates a new Home beside the backup or in another folder.
It refuses an existing destination and verifies the declared files before
publishing the restored copy. Opening that copy keeps AI paused for a dated
privacy review. Inspect Knowledge first. Known newer Forget decisions from the
current Home are retained; a backup restored without that current state may be
missing later privacy choices. Enabling AI does not restart old jobs, reattach
sources or approve a document. Large Homes and physical power-loss recovery
still need broader qualification.

When a saved draft changes in another editor, **Inspect Home changes…** shows
the changed text. **Adopt reviewed drafts** accepts only the exact version you
reviewed and preserves local conflict records. If the composer also has unsaved
text, the existing two-version choice protects it. This works after restart,
although a previous draft that was never retained is explicitly unavailable.
Drafts and conflict records are excluded from AI context. Changed conversation
evidence, missing records and malformed drafts remain paused for recovery;
arbitrary edits to owned state are not silently imported.

## Long conversations and inspectable context

Sevra keeps the full saved conversation while selecting a bounded recent window
of complete exchanges for each request. Older partial excerpts are labeled as
such. **Context** shows which messages and saved memories were used and what was
omitted. An oversized current request or pinned Home exchange is refused with
its text preserved; attach long material as a source instead.

Saved memories are selected by word overlap with the request, thread scope and
recency, within a bounded budget. This is deterministic retrieval, not a claim
of semantic recall. Forget excludes the source event from future windows and
derived excerpts. New Thread only conversations may read shared memories but
keep newly saved memories within that thread. Older Thread only conversations
retain their stricter reading scope until you explicitly allow shared memories.
Incognito uses neither saved memories nor persistent conflict records.

## Thinking and response details

**Think longer**, in the composer bar, lets Sevra reason before it answers. It
is off by default and stays on for the thread until you turn it off; a thread
with attached sources thinks too. While Sevra thinks, the status reads
"Thinking…" with a clock, and the last few lines of its reasoning appear under
it, newest at the bottom. Click them to read all the working notes, or press
**Answer now** to end the thought and answer from what Sevra has so far. When
the answer arrives, a line above it says how long Sevra thought. Click that line
for the notes and the rest of the response's details. Working notes are not
saved or remembered: they stay only while Sevra is open, for the eight most
recent responses, and never enter the conversation's copy, export or search.
Only how long the thought took and how it ended is saved with the response.

Every response also records what it cost on this Mac, measured by the engine:
the tokens it wrote and how fast, the time to its first token, how much of the
conversation it read and how much it reused, the context it used, any model
load it waited for, the share of the model's experts already in memory while
it wrote, and the memory budget it ran with. Speed depends on that budget, so
the details always show it. With a custom limit, they distinguish that saved
limit from the smaller budget available for the response. Turn on **Show response details** in the
conversation options menu or the View menu to add a line under every reply
with tokens per second, tokens written and time to first token, and to show
the live writing speed in the status while Sevra writes. To open the full
details, click a reply's thinking line or speed line, choose **Show Response
Details** from the reply's context menu, or press ⌥⌘I for the latest reply.
**Copy** there copies the numbers as text, without any notes or messages. The
numbers stay in your Home and are never sent anywhere.

## Brand and native UI review

The unified macOS toolbar brings the window controls, sidebar toggle, current
page, memory status, Search and three-dot actions into one row. Long thread
titles truncate while their full text remains available to accessibility and
in the tooltip. Control-Command-S toggles the sidebar or compact navigation.
The actions menu follows the current thread and includes Settings.

The heading follows the sidebar divider as navigation resizes, and sits beside
the native navigation controls when the sidebar is hidden. Conversation text,
composer, status and document actions share a reading column. Search, Journal
and Knowledge use matching gutters. Sidebar icons and row spacing follow a
consistent grid; document titles, text and footer actions align in split review.

Settings separates General, Model and Keyboard into native sections. General
holds appearance and Home location; Model keeps automatic memory and readiness
controls together, with usage details available through a disclosure. Model-file
checks show their own progress and readiness. Keyboard lists the same shortcuts
used by the app menus.

Search supports arrow-key selection and Return to open the selected thread.
Arrow keys remain with the input method while composing text. Escape closes
native Find before leaving a document, and returns from a panel to the
conversation. Memory choices show their selected scope; Incognito explains its
restriction instead of offering scope changes it cannot apply. Knowledge uses
plain save/forget wording and explains memories that are too long to save.

The sidebar uses the approved Observer mark and lowercase Poppins wordmark.
The app bundle includes a multiresolution Observer icon for Finder, the Dock
and About. Both the command-line bundle script and Xcode target package it.
To regenerate the icon from the same native vector used by the sidebar:

```bash
bash Tools/generate_sevra_icon.sh
bash Tools/build_sevra_mac.sh
```

The source mark retains the approved geometry; the rounded warm tile is specific
to the Mac app icon. Modern layered-icon qualification through
[Apple's Icon Composer workflow](https://developer.apple.com/documentation/Xcode/creating-your-app-icon-using-icon-composer)
remains release work.

A native walkthrough verified the logo in Light and Dark, appearance persistence,
Save and relaunch, saved-document viewing, large text, narrow navigation, draft
continuity through appearance changes and scoped Find. **Open document** now
shows the current saved UTF-8 file inside Sevra, with a separate Finder action.
Its owner-mediated preview has a bounded read and refuses symbolic links.
Saved documents also remain available from the conversation’s **Documents** menu
after later messages. Earlier citations resolve to their own retained run and
source excerpt, including after reopening the Home.

The runtime validates the complete tool response before executing any call.
It can request bounded schema feedback for unsupported argument keys or a
recognized tool-name spelling mistake. The rejected response executes nothing;
the model must return a valid new response, and saving still requires exact
document approval. The final adversarial replay passed this path with the real local model,
including correction, cited proposal, exact approval and reopening the saved
file. This qualifies the bounded fixture, not every possible user task.
What the model writes before calling a tool says what it is about to do, so
it is shown as one line under the reply's **Activity**, ahead of the calls it
introduces, and never becomes part of the answer. The answer is the final
round's text, or the text beside a proposal. If the final round has no text
of its own, the reply keeps what the model wrote along the way.
See the [feature audit](../db/records/design/sevra-spec/implementation-status.md#adversarial-mac-app-review)
for the current evidence and open requirements.

The conversation and saved-document view render headings, emphasis, lists,
quotes, syntax-colored code and native tables. Code has an exact-copy action;
the outline jumps to headings, code blocks and tables. Standard selection and
Find operate within the loaded history page. Earlier/Newer navigate bounded
pages; Copy conversation and Export Markdown preserve the complete source.
Raw HTML stays literal, images do not download, and opening a web link requires
a destination review. Evidence links resolve only to excerpts supplied by the
native owner. Unsupported or oversized Markdown remains readable as source.

Hover over the conversation icons for their native help: **Outline** jumps
to a heading, code block or table and appears when the rendered view has
sections to navigate. **Conversation options** holds Markdown source, copy,
export and Find. Document previews have their own options and Find; source
mode and Find stay scoped to the pane you choose.

**Jump to latest message** is a round down-arrow above the composer. It appears
when newer content is below or you are reading an earlier page, and disappears
at the latest message. Click it or press **⌃⌘↓** to return to the newest content.
New output follows while you are at the bottom; scrolling up or selecting text
preserves your reading position. Sending your own message returns to the latest
page while keeping the composer focused. The shortcut also appears in the View
menu and Keyboard settings.

The composer grows with the draft, retains native Undo and marked-text handling,
and accepts files and folders through its picker, drag or paste. Thread controls
cover pin, rename, lifecycle and memory scope. Review opens beside the conversation
when space allows; compact windows use explicit navigation and a single document
pane. Search, Settings and document commands have deliberate focus behavior.

Markdown parsing uses the upstream Swift Markdown library off the UI thread.
Completed messages are cached during streaming. Native tables use AppKit's
compatible text-layout path deliberately; this does not claim a wholly
TextKit 2 implementation. The Markdown dependencies' notices ship in the app.

This remains a development interface. Complete screen-reader/IME qualification,
live OS accessibility transitions, sustained interaction and measured frame,
latency and memory budgets still require the full native test matrix. Ad-hoc
builds do not establish installed-release quality. The
[implementation ledger](../db/records/design/sevra-spec/implementation-status.md)
records the exact observed scope and remaining gates.

## Memory and readiness

Settings → Model → Memory budget defaults to **Automatic (Recommended)**. The app uses
Slotstream’s memory planner and elastic cache governor, keeping the current
text model and context fixed while adapting cache residency to the Mac and
other applications. The recommendation inherits the engine’s measured
operating ceiling; it is not a promise of optimal performance on every Mac.

Desktop also uses the engine's automatic speculative decoding when the optional
draft head is installed and its full memory cost fits. Smaller budgets keep
ordinary decoding. This is independent of **Think longer** and needs no user
switch. Short chats use smaller prompt-processing batches to create useful
conversation checkpoints; longer inputs retain the engine's throughput schedule.
Crossing between schedules can require a fresh read because incompatible
checkpoints are never reused.
The [operating-policy record](../db/records/decisions/sevra-app-speed-defaults-2026-09-23.md)
contains the comparisons, costs and conditions for revising these choices.

**Custom limit** means “use up to” the selected budget within the displayed
supported range, which comes from this Mac's hardware rather than the automatic
default. It retains automatic pressure protection. Switching to Custom starts
at the current budget; returning to it restores your last chosen limit. The
saved limit stays stable when available memory changes. Settings distinguish
the app’s physical memory use from the budget available now. Unified
CPU/GPU memory is counted once.
If a saved limit exceeds the current Mac's supported range, Settings shows
the saved value and asks you to lower it or choose Automatic.

The model loads with the first request and verifies the pinned files. Within
that app session, unchanged files on APFS can reuse the successful verification
after unloading. File identity, size and modification/change timestamps are
checked again; changes require fresh hashing. Other filesystems and new app
launches always hash again. No verification proof is stored on disk. An immediate
reload lets macOS refresh its memory statistics before sizing the next model,
so memory just released is not incorrectly counted as still occupied.
**Keep model ready → Automatic** keeps it loaded while Sevra is in the foreground,
including while you read or compose. Leaving the foreground starts an inactivity
interval, with a bounded delay informed by observed preparation time. Memory
pressure, power saving and sleep can release it sooner. **While app is open** favors warm
follow-ups but still yields to memory pressure and sleep. **Release memory
now** preserves saved conversations and personal memory. Ordinary conversations
can reuse a disposable prompt cache in that Home after a reload. Thinking and
incognito sessions keep their inference state off disk. Backups exclude the
prompt cache; it can be rebuilt from the saved conversation.

Budget changes during a response apply after that job finishes. New messages
wait through the short resource handoff. Closing the window follows the
existing accepted-work policy; quitting drains work and releases the model.
Sleep stops active work and interrupts queued work. Wake permits new requests
without replaying interrupted actions. An unavailable memory reading or a
configuration that cannot fit produces an explicit refusal. The full
conversation stays on disk while each request uses its disclosed context window.

The native lifecycle and policy checks are in the ordinary regression runner.
The separate `--performance-real --home <new-disposable-directory>` check uses
a bounded custom budget to exercise lazy loading, warm reuse, changing a budget
during generation, release/reload, responsive metadata and automatic idle
release. Follow the same model-process and headroom rules as the real fixture
below. Full hardware qualification and clean paired performance measurements
remain separate from these functional checks.

Real cache, response-metrics, thinking and source-tool checks also accept
`--mtp-profile-gb <budget>` for an explicitly requested MTP performance profile.
They require the chosen budget plus the ordinary headroom, refuse a profile
that does not actually enable MTP, and use disposable Homes. The default checks
retain their small budgets. `--real-speed --memory-gb <budget> --arm <policy>`
compares complete allocations and checks reused output against a fresh run;
`--extended` includes a longer inventory. These are development measurements,
not public hardware qualification.

## Checks and internal CLI

```bash
swift build --package-path apps/macos -c release --product sevra-mac-checks
apps/macos/.build/release/sevra-mac-checks
swift build --package-path apps/macos -c release --product sevra-presentation-checks
apps/macos/.build/release/sevra-presentation-checks
swift build --package-path apps/macos -c release --product sevra-local
apps/macos/.build/release/sevra-local --help
```

The check runner uses fake inference and real dbmd mutations in disposable
Homes. It does not load the large model. Real-engine and native interaction
results must be recorded separately. The internal `sevra-local` executable
does not replace the existing `sevra` compatibility CLI. An active Home is
exclusive. The internal CLI attaches through a user-local authenticated Unix
socket to the existing owner, or acquires the Home lock itself when given an
explicit dbmd executable. It never starts a second owner for an active Home.
Incognito is unavailable to this initial CLI attachment.

```bash
apps/macos/.build/release/sevra-local status --home "$HOME/Sevra/Home"
apps/macos/.build/release/sevra-local chat --home "$HOME/Sevra/Home" --prompt "Hello"
```

The app must already own that Home for these commands. A CLI connection may
detach while the app continues its accepted job. Preserve the printed nonce
to reconcile uncertain submission; a lost response is not permission to retry
with a new request id. The optional network server is not implemented.

The real-model fixture additionally exercises source reading, citation review,
create-only document publication and reopening its durable Home. Run it only
under the repository memory rules, with a new disposable destination:

```bash
cp Tools/lib/mlx-0.32.2.metallib apps/macos/.build/release/mlx.metallib
apps/macos/.build/release/sevra-mac-checks --real \
  --source "$PWD/apps/macos/Fixtures/private-workspace-brief" \
  --home "$PWD/.build/sevra-disposable-real-check"
```

The harness reviews a synthetic fixture after checking its frozen rubric. It
does not certify the native Save button, appearance, accessibility or public
release. See the implementation ledger for observed and pending evidence.

A second real-model fixture asks a question about a generated PDF, proposes an
exact edit to a text file and proposes a counter mini-app, acting as reviewer
for each. The generated PDF is the same bytes on every run, so two runs on the
same Mac and memory plan give the model the same input. It needs the helper,
the metallib copied as above and the same memory rules:

```bash
SEVRA_EXTRACT="$PWD/apps/macos/.build/release/sevra-extract" \
  apps/macos/.build/release/sevra-mac-checks --real-basics \
  --home "$PWD/.build/sevra-disposable-real-basics"
```

To try the counter app it built, run the apps check against the app's file. The
app opens twice in the production host over one scratch store. After a few
clicks on plus, the reopened app must show the same count and keep a single
saved record:

```bash
SEVRA_APP_UNDER_TEST="$(find "$PWD/.build/sevra-disposable-real-basics/Home/extensions/miniapps" -name index.html | head -1)" \
  SEVRA_APP_DATA=counter:write bash Tools/check_sevra_apps_ui.sh
```

A third real-model check compares the numbers Sevra records for a response
with the engine's own statistics for the same requests: a thinking turn that
loads the model, a second thinking turn that resumes the conversation instead
of reading it again, and a plain turn after switching thinking off, which reads
it again because the switch changes the system instructions. The engine
resumes a request only at its own prefill pass boundaries, so the opening
message is long enough for the conversation to pass the first one. It needs
the metallib copied as above and the same memory rules:

```bash
apps/macos/.build/release/sevra-mac-checks --real-metrics \
  --home "$PWD/.build/sevra-disposable-real-metrics"
```

The live-source fixture checks the model's explanation of file access, then
attaches a large disposable folder, locates and cites a file, rediscovers it
after a rename and edit, and proposes a reviewed change. It verifies the
approved bytes and undo. Use a new destination and the same memory rules:

```bash
SEVRA_EXTRACT="$PWD/apps/macos/.build/release/sevra-extract" \
  apps/macos/.build/release/sevra-mac-checks --real-sources \
  --home "$PWD/.build/sevra-disposable-real-sources"
```

Its `receipt.json` records answers, actual tool traces, source excerpts and
failures. The ordinary checks also cover live navigation and search pagination
without loading a model; `sevra-mac-checks --sources` runs that group alone.

Slotstream's original CLI, serving APIs, library products and package coordinates
remain independently usable. This application work does not rename the public
repository, publish a release or change existing credentials and services.


<!-- ===== docs/EXPERT-LOOKAHEAD.md ===== -->

# Expert lookahead

Expert lookahead starts reading likely needed experts from SSD while the GPU
is still working on earlier layers. It reduces the time spent waiting for
weights during reply generation.

## How it works

In a mixture-of-experts model, a small *router* chooses which expert networks
each token needs. Slotstream keeps some experts in RAM and loads missing ones
from SSD. Normally, it discovers a missing expert only when its router runs.

We reuse a future layer's own router on the model's current internal state to
guess its choices early. That state will still change before the layer runs,
so this is an approximate forecast. It needs no separately trained predictor.

- **Forecast:** as each layer's routing comes back, read the model's state after the
  previous layer's attention step, apply the next layer's router to it and correct
  the result with a small learned table shipped for the checkpoint (0.2.19). Without
  that file, the router of the layer two ahead runs on the state two layers back, as in
  0.2.16. Either way the SSD reads get time to overlap GPU work.
- **Prepare:** reserve cache slots for promising missing experts. Background
  workers read into reusable temporary buffers, then copy into those slots.
- **Use:** when the real router asks for a prepared expert, make its slot
  available with a bookkeeping update. No further weight copy is needed.
- **Recover:** if a needed read is still running, finish it; if an expert was
  missed, load it normally. Discard unused forecasts and reclaim their slots.

The real router still makes every final choice with the original weights.
Wrong forecasts waste reads, without changing the computation. The draft head
has a separate job: proposing tokens for the main model to verify.

## What we tried

| Experiment | Result |
|---|---|
| [Train small predictors](../db/records/measurements/expert-lookahead-pilot-offline-stop-2026-09-11.md) | Too inaccurate within the allowed read budget. Stopped before speed qualification. |
| [Reuse the model's routers](../db/records/measurements/expert-lookahead-2-router-reuse-prefetch-native-screens-2026-09-12.md) | Much better forecasts, but the first implementation was slower: preparing prefetched weights for use cost more than the disk waits it saved. |
| [Prepare weights in reserved cache slots](../db/records/measurements/expert-lookahead-2-b0-cohort-2026-09-12.md) | Moving that work into background workers made prefetch useful: **1.105x** decode throughput on the first held-out benchmark. |
| [Cache router conversions and reduce GPU waits](../db/records/measurements/decode-path-serialization-attribution-2026-09-13.md) | Keeping router weights in FP32 and draining queued GPU work every four layers (when cache space permits) added **about 2% each** over prefetch in separate tuning runs. |
| [Forecast from the previous layer's attention](../db/records/measurements/decode-forecast-taps-b1-cohort-2026-09-15.md) | Reading the model's state after the previous layer's attention, at the same lead time, raised forecast agreement from 0.62 to 0.73 and decoded 1.058x on the held-out benchmark with identical outputs, but the run failed a hygiene condition (host swap noise left one prompt with a single clean pair), so it is not a confirmed result on its own. |
| [Learn a small correction of the forecast](../db/records/measurements/decode-forecast-taps-learned-confirmation-2026-09-15.md) | A per-layer linear correction of the forecast's router logits, 35 MiB of FP16 factors fitted on the pilot's requests, raised agreement to 0.80 and decoded **1.031x** over the uncorrected tap on ten held-out prompts (bootstrap 1.013 to 1.041), identical outputs, 9% fewer reads during decode. |
| [Issue more speculative reads](../db/records/measurements/decode-forecast-taps-threshold-screen-2026-09-15.md) | Lower margin thresholds cut demand reads but the extra reads arrived late and wasted bandwidth: 1.013 and 0.994 against the shipped threshold, which stays. |
| [Compute the next layer's attention early](../db/records/measurements/decode-forecast-taps-readout-timing-screen-2026-09-15.md) | Running the target layer's attention on the forecast's input against the resident caches is exact and reaches 0.86 agreement (0.88 corrected), near the ceiling: the rest is the previous layer's routed experts, unknowable before reading them. It still decoded 5% slower than the corrected forecast, and 26% slower when its GPU work was deferred behind the demand reads, because that work costs more than the reads it saves. |
| [Corrected forecast against the shipped bundle](../db/records/measurements/decode-forecast-taps-readout-timing-confirmation-2026-09-15.md) | The direct measurement for a default decision: **1.111x** over the shipped configuration on eight held-out prompts (bootstrap 1.102 to 1.123), all 23 pairs faster, identical outputs, 21% fewer records read in decode and 73% fewer wasted speculative bytes. Shipped as the 0.2.19 default; see below. |

Cache replacement policies and remembering routes for repeated tokens also
failed their [adoption gates](../db/records/measurements/expert-lookahead-2-router-reuse-prefetch-native-screens-2026-09-12.md).
The existing CLOCK cache policy stayed.

## The 0.2.19 default: a corrected forecast

**The shipping build's default decodes 1.10x faster than the 0.2.18 forecast** on
eight held-out prompts (two code, two prose, two reasoning, one dialogue, one
structured) at a 22 GB memory target, aggregate 1.108 (bootstrap 1.092 to 1.147),
every kind at 1.079 or above, 24 counted pairs, identical output in every
cell, 14.38 to 15.86 tok/s median. It reads about 20% fewer expert records
during decode and wastes about 74% fewer speculative bytes. The two arms are
the same binary: the default (the correction file located next to the weights)
against `SLOTSTREAM_EXPERT_PREFETCH_TAP=boundary`, the 0.2.18 forecast. The
22 GB target is where the automatic plan runs the lookahead under the
benchmark's settings (prefix cache off, two drafts); at a 20 GB target it does
not, so a pre-release measurement of the same contrast used environment-configured
arms there and read 1.111x, 13.10 to 14.83 tok/s. See the
[release benchmark](../db/records/measurements/corrected-forecast-release-benchmark-2026-09-16.md)
and the [default's qualification](../db/records/measurements/corrected-forecast-default-2026-09-16.md).

## Final result and limits

The complete bundle achieved **1.11x decode throughput** against the previous
default, which already used speculative decoding. Median speeds were
**11.79 to 13.47 tok/s**, with identical output tokens in every compared run.
The speedup uses paired comparisons; the medians use different samples.

Tests used Qwen3.8-Flash-Next on one 48 GB M5 Pro, a 20 GB memory target and
two draft tokens. The final benchmark used twelve held-out prompts across six
task families, with repeated, interleaved comparisons. Every family was faster.
See the [final benchmark](../db/records/measurements/decode-path-serialization-b1-cohort-replication-2026-09-13.md).

Other Macs, cache sizes and prompt processing were not qualified by this test.
The component gains came from tuning runs and cannot be added to this result.

Lookahead is enabled by default with the draft head when the cache meets its
activation threshold. Its **373 MiB** reservation, 409 MiB with the 0.2.19
correction file, comes out of the memory budget before sizing the expert cache.
Set `SLOTSTREAM_OPT_EXPERT_PREFETCH=0` to disable the default bundle, or
`SLOTSTREAM_EXPERT_PREFETCH_TAP=boundary` to keep the 0.2.16 forecast with the
file present. The [adoption decision](../db/records/decisions/decode-lookahead-default-with-the-draft-head.md)
and the [corrected forecast decision](../db/records/decisions/corrected-decode-forecast-default-with-the-sidecar.md)
record the settings, memory safeguards and overrides.


<!-- ===== docs/HERMES-NOTES.md ===== -->

# Hermes engineering notes

The [Hermes setup guide](HERMES.md) is the user walkthrough and the source
for the complete tested configuration. These notes explain the integration
and its limits for developers and people adjusting that configuration.

## Protocol and ownership

Hermes uses the OpenAI chat-completions endpoint. Slotstream translates its
function definitions, calls, results, and reasoning into the same native model
format used by the fx gateway. Tool execution and argument validation belong
to Hermes. Slotstream also serves the OpenAI Responses API for clients such as
Codex; constrained JSON generation is not implemented. See the [API reference](API.md).

## Context and memory

The startup command in the [setup guide](HERMES.md#start-slotstream) leaves
memory planning and speculative decoding on their automatic defaults.
Slotstream decides whether to enable the MTP draft head from its availability
and the planned expert cache. Hermes does not require an MTP override.
Hermes requires a larger context than the 32,768 tokens auto keeps on Macs
through 32 GB, so the setup guide fixes 65,536 tokens on every Mac. The
explicit flag charges that window's extra active state and a measured
transient reserve before allocating the expert cache. The
minimum memory target rises with this larger window. If overriding automatic
planning, inspect `slotstream doctor --max-context 65536` first and choose a
target that fits your Mac. Long prompts must still be read before the first
answer token.

## Discovery, images, and timeouts

The placeholder key is for this local endpoint. Hermes uses chat completions.
The explicit context setting is reproducible; the server also exposes the
same runtime window through model discovery, so automatic discovery works.
Vision discovery also reflects whether this server accepts images. With the
vision weights available and vision enabled, Hermes can send image attachments.
The first image reserves the vision tower inside the automatic process
target, reducing expert capacity as needed. Image admission also checks
attention workspace and real headroom; it can refuse an image even when
text fits. Explicit pool-size overrides retain their pool and add the tower
to the expected footprint.
The local stream timeout allows a long cold prefill to finish; it remains
bounded and should be adjusted to measurements on slower hardware.
The auxiliary task timeouts are configured separately from the main stream
watchdog. Otherwise a long summary can time out while an equally long main
prefill would still be allowed to continue.

## Provider selection

The named `providers.slotstream` entry binds the endpoint, placeholder key,
API mode, and request overrides together. The launch command selects that
entry explicitly. A missing or disabled entry produces an initialization
error in the tested versions. Hermes can still log `provider=custom` for this
named endpoint; the resolved URL, not that internal label alone, identifies
the connection.

The older generic `custom` setup could collide with a saved provider also
named `custom`, including the legacy `custom_providers` list. `CUSTOM_BASE_URL`
from the shell or the profile's `.env` could also override the old
`model.base_url`. `OPENAI_BASE_URL` is not the routing override for that generic
Hermes path. The named setup is tested with all of these stale settings present.
This does not prevent someone from editing `providers.slotstream` itself or
deliberately configuring a fallback provider.

Explicit HTTP proxy variables are a separate routing layer. In both tested
Hermes versions, an HTTP proxy without local exclusions is selected even for
the local model URL. Include `localhost` and `127.0.0.1` in `NO_PROXY` and
`no_proxy` when using a proxy, preserving existing exclusions. The profile's
`.env` can override shell values. This condition does not explain a log that
already reports OpenRouter as the selected model endpoint.

## Output budgets

In the tested Hermes versions, CLI initialization does not forward
`model.max_tokens` to the agent. Our earlier guide put the limit there, and our
earlier integration gate constructed the agent directly with an explicit limit.
That bypassed the failing configuration path. Without a limit on the actual
request, Slotstream used its ordinary completion default and could cut off
long replies or tool arguments.

The guide now sets the main limit through
`providers.slotstream.extra_body.max_tokens`, which reaches the wire through
Hermes's named-provider request overrides. Auxiliary requests need their own
`extra_body.max_tokens`; they do not inherit this override. These are generation
ceilings. They do not repair Hermes's separate internal output-reservation
accounting, so leave compression enabled and room for input history. Larger
limits still have to fit the server's advertised `max_output_tokens` and
remaining context.

## Summaries and titles

Route auxiliary compression and titles to `main` to keep them on Slotstream.
Hermes omits the usual output-limit argument on custom-provider auxiliary
calls. Each task's `extra_body` supplies its own budget. Compression can take
several minutes on a long history. Compressing a very short conversation may
increase its size because Hermes adds a structured handoff and retains recent
messages; test it on a history with a substantial middle to summarize.
Title generation can request constrained JSON first; Slotstream explicitly
rejects that unsupported mode, and Hermes retries without the constraint.
This fallback does not provide a strict JSON-schema guarantee.

## Integration checks

The automated protocol gate runs against an already-running server:

```sh
python3 Tools/openai_tools_gate.py --output /tmp/slotstream-openai-tools.jsonl
```

The gate preserves requests and responses and executes no tools. It checks
streamed and non-streamed tool loops, typed arguments and IDs, reasoning,
discovery, tool-choice behavior, and rejection of invalid histories. The
ordinary API and image regression suites remain separate.

Run the command from a source checkout, with the server already running.
For tests through the actual Hermes client, including compression and images,
see [OpenAI agent integration](TESTING.md#openai-agent-integration).
The [configuration correction](../db/records/measurements/hermes-configuration-hardening-2026-09-08.md)
records the corrected guide and tests through the actual CLI configuration path.
Those live Hermes runs explicitly disabled MTP. The guide now follows
Slotstream's automatic default, but a live Hermes run with automatic MTP is
still pending. The gate records the selected server plan so that later runs
can distinguish enabled and disabled MTP coverage.
The earlier [protocol integration measurement](../db/records/measurements/hermes-context-and-openai-integration-2026-09-05.md)
records exact client commits, build identities, observed failures, passing
checks, and the limits of the qualification. These results cover those tested
versions and paths; they do not establish compatibility with every future client.

## Server request deadlines

Slotstream's `--max-prefill-wait` is a separate request deadline, including
queueing, preparation and image processing. If deliberately raising that
server budget, raise the client's stale-stream timeout and HTTP read timeout
enough to let the server return its own terminal result. Keepalives do not
extend the server budget. A deadline or resource error must not execute a
pending tool call or be treated as a completed summary. The first image
reserves the tower's memory inside the server target; a target that can only
fit text at the selected context size can refuse that additional load.

See [request limits and errors](API.md#request-deadlines-and-resource-failures).


<!-- ===== docs/CLIENTS.md ===== -->

# Connect apps and agents to Slotstream

Slotstream runs the model; your app provides the chat interface or tools.
Install both separately, then connect the app to the running Slotstream server.

## Choose a connection

| I use… | Follow… |
|---|---|
| Claude Code, Codex, Pi or opencode | [Coding agents](CODING-AGENTS.md): `slotstream launch` starts each one connected |
| Hermes | [Hermes setup](HERMES.md), which includes all the settings it needs |
| Open WebUI or the Ollama CLI | [Ollama-compatible clients](#ollama-compatible-clients) below |
| An app with an OpenAI-compatible or custom provider | [OpenAI-compatible clients](#openai-compatible-clients) below |
| fx | [fx setup](FX.md), including its permission and long-session limitations |

Start with [Get started](GETTING-STARTED.md) if Slotstream isn't installed.
For agents that use tools, you need **Slotstream 0.2.8 or later**. Check with
`slotstream --version`; run the [installer](../README.md#install) again to update.

## Start one server

**Hermes and fx users:** follow your app's guide above for the server command
and configuration. For an ordinary chat app, open Terminal and run:

```sh
slotstream serve
```

On first use, accept the model download and wait for it to finish.
Once you see `slotstream listening on http://127.0.0.1:11434`, leave this
window open. Stop the server with **Control+C** when finished.

The settings below are for an app running directly on the same Mac.
Apps running in Docker or on another computer need separate networking
configuration. See [Security](../SECURITY.md) for the server's local-access limits.

## OpenAI-compatible clients

In your app's provider settings, choose **OpenAI-compatible** or **custom
OpenAI**, then enter:

| Setting | Value |
|---|---|
| API mode, if offered | Chat Completions or Responses |
| Base URL | `http://127.0.0.1:11434/v1` |
| Model | `qwen3.8-flash-next:4bit` |
| API key, if required | `unused` |

No OpenAI account or key is needed for this local connection. Leave `unused`
as written. Apps that use the OpenAI Responses API work on the same base URL;
Codex needs the setup in the [Codex guide](CODEX.md). Apps that use the
Anthropic Messages API use `http://127.0.0.1:11434` as the base URL, without
`/v1`; see the [Claude Code guide](CLAUDE-CODE.md).

If the app asks for a **full endpoint** instead of a base URL, use:

```text
http://127.0.0.1:11434/v1/chat/completions
```

Apps may have separate model settings for conversation titles, summaries,
and fallbacks. Set those to the same local provider if you want those model
requests to stay on your Mac. Hermes's [configuration](HERMES.md#configure-hermes)
already does this. Tools that use web services still need their own connections.

## Ollama-compatible clients

In Open WebUI or another app with an Ollama provider, use:

| Setting | Value |
|---|---|
| Provider | Ollama |
| Server URL | `http://127.0.0.1:11434` |
| Model | `qwen3.8-flash-next:4bit` |

These settings support chat and images. For agents that use tools, choose
OpenAI Chat Completions or follow the [Hermes guide](HERMES.md).

If you already have the Ollama CLI installed, you can also open a chat in
a second Terminal window:

```sh
OLLAMA_HOST=http://127.0.0.1:11434 ollama run qwen3.8-flash-next:4bit
```

## Check the connection

Select the model in your app and send a short message, such as “Hello.”
You should see activity in the Slotstream terminal, followed by a reply in
your app. A first reply may take a while to begin.

For an agent, try the [file-reading example](HERMES.md#try-it). Check that
it actually reads the file and returns its contents.

## Troubleshooting

| Problem | What to check |
|---|---|
| Connection refused or the wrong model appears | Keep the Slotstream server running and copy the address and model name exactly. Check for [port conflicts](TROUBLESHOOTING.md#the-server-cant-listen-on-port-11434). |
| Tools don't work | Use OpenAI Chat Completions with Slotstream 0.2.8 or later. The Ollama connection does not support tools. |
| The app calls `/v1/responses` | Supported from Slotstream 0.2.20. Update with the install command if `slotstream --version` is older. Codex needs the [custom provider setup](CODEX.md); it cannot use `--oss`. |
| The app calls `/v1/messages` | The Anthropic Messages API, supported from Slotstream 0.2.21. Set the base URL without `/v1`. |
| The app sends `store` and gets a 400 | Update Slotstream; from 0.2.21 chat completions accept `store: false` without effect. `store: true` still returns 400, since the server keeps no completions to fetch later. |
| The conversation is too long | The window includes instructions, history, and reply. Auto picks it per Mac, often 32,768 tokens; `slotstream doctor` shows yours. Hermes needs the `--max-context 65536` setup in its guide. |
| The first answer times out | Check progress in the Slotstream window. Long prompts can take minutes. See your agent's guide for its timeout settings. |
| Summaries stop early or use a cloud model | Check the separate summary-provider and reply-length settings. Use the full configuration in the Hermes guide; fx has known summary limitations. |
| The app requires strict structured output | JSON-schema constrained output and strict tool schemas are unsupported. The app needs a mode that works without that requirement. |
| An image fails | Check available memory and that you haven't started Slotstream with `--vision off`. Developers can check the [image request format](API.md#images). |

<a id="reporting-an-integration-problem"></a>

[General troubleshooting](TROUBLESHOOTING.md) covers startup, memory, and
model files. For a bug report, include the app and Slotstream versions, Mac
model and memory, server command, connection settings, and error text.
Remove credentials and private conversation or file contents.

Developers: see the [API reference](API.md) for supported fields and request
examples, and [integration tests](TESTING.md#openai-agent-integration) for
protocol and real-client checks.


<!-- ===== docs/CLI.md ===== -->

# Command reference

This page covers everyday commands, memory settings, and common diagnostics.
Run `slotstream <command> --help` for the options in your installed version.
Only one model process can run per user at a time.

The shared `run` context option, request-wait controls, feasibility metadata
and expanded `context-check` flags below are available starting in Slotstream 0.2.14.

<a id="where-things-live"></a>

## File locations

| Path | What |
|---|---|
| `~/.slotstream/bin/` | Symlink to the active release: the `slotstream` binary and its `mlx.metallib`. |
| `~/.slotstream/releases/<sha256>-macos<NN>/` | Each installed release, content-addressed. The installer stages a release here, verifies it, then switches the `bin` symlink. |
| `~/.slotstream/launch/codex/` | The model descriptions and base instructions `slotstream launch codex` gives Codex, one per Codex version and window. |
| `~/.slotstream/launch/server-<port>.json` | The server `slotstream launch` started on that port, by process id and start time, so a later launch knows it may restart it. |
| `~/.slotstream/launch/start-<port>.lock` | Held while a launch may start a server on that port, so a second launch waits for that server. |
| `~/.slotstream/logs/serve.log` | The log of the server `slotstream launch` started (`serve-<port>.log` on another port), with the previous start's as `.1`. |
| `~/.slotstream/prefix-cache/` | Prompt caches the server `slotstream launch` started keeps on disk: token ids and model state, 20 GB at most. `slotstream prefix-cache` lists or clears them. |
| `~/.slotstream/models/qwen38-flash-next-mlx-4bit/` | The weights: 25 files, 105.3 GB (the 1.5 GB draft head is optional). Compressed pulls use `.slotpack-state.json` and `.slotpack.part` files while in progress; legacy raw pulls use `.partmap` and `.part`. |
| `/usr/local/bin/slotstream`, or a PATH line in `~/.zshrc` / `~/.bash_profile` | How the installer puts the command on your PATH (the wrapper when `/usr/local/bin` is writable, the profile line otherwise). |
| `/tmp/slotstream-model-<uid>.lock` | The one-process lock, held while a model is loaded. |

## Reading tokens per second

`slotstream run` prints a `-- decode` line with average `tok/s` after the
response, and a separate prefill rate for reading the prompt. Add
`--stats-json stats.json` to save the raw generation measurements.

When using an existing `slotstream serve`, the Ollama-compatible `/api/chat`
and `/api/generate` endpoints return `eval_count` and `eval_duration` in the
final response. For streaming, read the final frame. Duration is in
nanoseconds: divide the count by `eval_duration / 1e9` when that duration is
positive. This measures the decode phase and excludes prefill. The ordinary
OpenAI-compatible response supplies token usage without these duration fields.

These are end-of-response statistics. A planner's expected speed is an
estimate; it is separate from the measured rate of a completed request.
[The serving benchmark and throughput expectations](../MEASUREMENTS.md#three-prompt-serving-benchmark-and-throughput-expectations)
show the observed variation, prompt-reuse behavior and measurement limits.

## Everyday commands

### `slotstream run`

Generate once from a prompt, with no server.

| Flag | Meaning |
|---|---|
| `--prompt <text>` | The prompt (default: "Why is the sky blue?"). |
| `--max-context <auto or n>` | Window shared by the final templated input and reply. Default `auto`; the same planning and bounds as `serve`. |
| `--max-prefill-wait <minutes>` | Accepted request to first sampled model token, including preparation. Default 30 minutes; `0` disables only the time policy. |
| `--max-tokens <n>` | Tokens to generate; `<= 0` means as many as the context allows (default 128). |
| `--greedy` | Deterministic greedy sampling. |
| `--raw` | Send the prompt without the chat template. |
| `--think` | Enable the model's thinking mode. |
| `--image <path>` | Attach a local image; repeat the flag for multiple images. |

Plus the [memory options](#memory-options) below.

### `slotstream serve`

Start the local API server. See the [API reference](API.md) for the Ollama
and OpenAI endpoints and the [fx guide](FX.md) for the AI SDK gateway.

| Flag | Meaning |
|---|---|
| `--port <n>` | Listen port on 127.0.0.1 (default 11434). |
| `--max-context <auto or n>` | Maximum tokens shared by prompt and reply. Default `auto`: the largest of 32768, 65536, 131072 and 262144 tokens that keeps speculative decoding, retains one complete conversation and adds at most 10% to the estimated request time. Auto declines cache reductions above the measured decode range because their performance cost is unknown; `doctor` shows each tradeoff. A number fixes the window, from 1 to 262144, the model's limit. Requests with images stay within 65536. A larger window is priced before allocating the expert cache; a prompt above the configured cap returns 400. Main sequence-cache capacity costs about 27 KiB per token, plus recurrent, retained, draft and transient allocations. |
| `--max-prefill-wait <minutes>` | Accepted request to first sampled model token, including queueing, tokenization and images. Default 30 minutes; `0` disables only time. |
| `--no-elastic` | Pin the cache at its startup size. By default an auto-sized cache resizes between requests as memory pressure changes; explicit sizes are always pinned. |
| `--no-prefix-cache` | Process each prompt from scratch. Useful for reproducibility comparisons. |
| `--prefix-cache-dir <dir>` | Also keep conversation states on disk, so a restarted server, or a conversation longer than the in-memory cache holds, resumes from its last committed state instead of processing its prompt again. Off unless set; the servers `slotstream launch` starts set it. A request writes its state during prefill, at the last pass boundary before the end of its prompt, and records the conversation's token ids after its reply; the next turn resumes from that state. The first write stores the fixed recurrent state and every cached token; each later turn writes the recurrent state plus only the tokens it added, keeps the previous turn's state so that reply can still be regenerated after a restart, and removes older states of the conversation. Files hold the conversation's token ids and model state and are used only by the same binary, model files and settings that wrote them; starting the server removes files from other builds. Requests with images are not written. The prefix conversations share is written too: when a prompt's system message ends 512 tokens or more in, or a prompt shares at least 512 tokens with a prompt already kept and parts from it there, that head is stored during the prompt's own prefill, rounded down to the prefill pass grid, the 256-token grid by default, so the next conversation with the same system prompt resumes from it, in the same process or after a restart. A shared prefix is kept once, is never replaced by the conversations that extend it, and is listed as such. A file the system cannot read is kept, and the server runs without the disk cache until the file is fixed or the directory cleared and the server restarted. `slotstream prefix-cache` lists or clears the directory. |
| `--prefix-cache-disk-gb <gb>` | Disk quota for `--prefix-cache-dir` (default 20), applied at startup and before each write. When it is full, states nobody continued go first, then previous-turn states kept for regenerating, then conversations, then the prefixes several conversations start from, least recently used first within each. |
| `--prefix-cache-min-tokens <n>` | Shortest state written to `--prefix-cache-dir` (default 1024 tokens), for conversation states and shared prefixes alike; a shorter shared prefix is still kept in memory. Through 0.2.25 the default was 2048. |
| `--prefix-cache-max-age-days <days>` | Remove states in `--prefix-cache-dir` unused for this many days (default 30), at startup and before writes. `0` keeps them until the quota needs room. |
| `--idle-exit <minutes>` | Stop after this many minutes with no request and no registered agent still running (default 0, keep serving). `slotstream launch` starts its server with 30 and registers each agent it opens through `POST /slotstream/clients`. The log says why the server stopped. |

Plus the memory options.

### `slotstream launch [agent] [arguments...]`

Start a coding agent connected to Slotstream. The agent is `claude`, `codex`,
`pi`, `opencode` or `hermes`; everything after its name is passed to it.
Without a name, the command lists the agents installed and asks which one to
start. It asks the server for its model, context window and reply limit,
prepares the agent's connection, and replaces itself with the agent, so the
agent runs in the same Terminal window. [Coding agents](CODING-AGENTS.md)
describes what each agent receives and which files are written.

When no server answers on the port, the command starts one in the background
and shows its start until it answers:

- with the window the agent needs (the automatic window, or 65,536 tokens for
  Hermes when the automatic one is smaller) and the `--memory-gb` given here;
- with prompt caches kept on disk in `~/.slotstream/prefix-cache`
  (`--prefix-cache-dir`, 20 GB at most), so a restarted server does not read an
  agent's instructions again;
- with its log in `~/.slotstream/logs/serve.log` (`serve-<port>.log` on another
  port), the previous start's kept as `.1`;
- in its own session, so Control-C and a closed Terminal reach only the agent.

That server keeps running while any agent `slotstream launch` opened is
running, and stops 30 minutes after the last one exits and its last request
ends (`--idle-exit`). `slotstream stop` stops it sooner. Control-C while it
starts stops it too. A server you started yourself is used as it is; the
command never restarts it. Two launches at the same time start one server:
the second waits for the first to finish starting, then uses its server.

| Flag | Meaning |
|---|---|
| `--port <n>` | The port the server listens on (default 11434). |
| `--memory-gb <gb>` | Memory target for a server this command starts, as in `serve`. Default: automatic. When a server with another target is already running, it is left as it is and a note names its target. |
| `--memory-limit-gb <gb>` | Adaptive ceiling for a server this command starts (development version). Cannot be combined with `--memory-gb`. An existing server keeps its settings; a note explains when the requested limit does not apply. |
| `--idle-exit <minutes>` | How long a server this command starts keeps running after its last agent exits (default 30, at most 10080); `0` keeps it running until `slotstream stop`. |
| `--no-start` | Use a running server only; never start or restart one. |
| `--dry-run` | Print the server it would start, the command, the variables it sets or removes, the files it would write, and notes; start, download and write nothing. Keys and tokens in the output are hidden, and for Pi only the `slotstream` entry of its models file is shown. |

The served window must reach the agent's minimum: 32,768 tokens for Claude Code
and Codex, 16,384 for Pi and opencode, 65,536 for Hermes. When the running
server's window is smaller, a server this command started and no agent is
using is restarted with the larger window; any other server is left running,
and the message says how to restart it.

It stops with a message, before starting the agent, when another server
answers on the port, when the server is busy with every connection it takes,
when another Slotstream model process runs for this user, when the model is
not downloaded and there is no terminal to ask on (with one, it asks first),
when the server is too old for the agent's API, or when the agent is not on
`PATH`. It also refuses what would not run on this Mac: `codex cloud`, Codex's
`--output-schema`, Pi with another `--provider` and no `--model`, and a Hermes
configuration whose `context_length` is larger than the served window. The
agent's arguments and files are checked before a server starts, so these
refusals come before the server's start, and before the model download is
offered. Codex needs its version's base instructions for the model
description; the first launch of each Codex version downloads them from the
Codex repository and keeps them under `~/.slotstream/launch/codex/`.

### `slotstream stop`

Stop the Slotstream server on a port: the one `slotstream launch` started in
the background, also while it is still starting, or one running in a Terminal
window. The server's requests end at once; prompt caches already on disk stay
for the next server. When no server answers, it says so and exits
successfully.

| Flag | Meaning |
|---|---|
| `--port <n>` | The port the server listens on (default 11434). |

### `slotstream prefix-cache`

Show what a `serve --prefix-cache-dir` directory holds, or clear it. Nothing is
loaded, so this works while no server runs; listing also works while one does.

| Flag | Meaning |
|---|---|
| `--dir <path>` | The directory given to `--prefix-cache-dir`. Default: `~/.slotstream/prefix-cache`, the one servers that `slotstream launch` starts use. |
| `--clear` | Remove every state file. Refused while a server or app holds the directory. |
| `--json` | Print JSON instead of text. |

Each state is listed with the build that wrote it, its token count, the size of
its own file and when it was last used. Cached tokens that several states share
are stored once and reported as one total.

### `slotstream pull [model]`

Download losslessly compressed model weights from Hugging Face, reconstructing
the original files with resumable transfers and hash verification.
The only model name is `qwen3.8-flash-next:4bit`, which is also the default.

| Flag | Meaning |
|---|---|
| `--dir <path>` | Destination directory (default `~/.slotstream/models/qwen38-flash-next-mlx-4bit`). |
| `--connections <n>` | Fixed independent connections, 1–32. Omit to start at 8 and test increases only while throughput improves. |
| `--transport automatic\|compressed\|raw` | Automatic uses compressed Hugging Face objects for new pulls and preserves legacy raw resumes. Explicit raw selects file-based mirrors. |
| `--verify` | Check existing files against pinned SHA-256 hashes without downloading. |

The complete compressed package uses **16.12% fewer bytes**. Decode and writes
overlap the transfer. Historical raw-path testing measured 112 MB/s on a
1 Gbit/s link; that is not a guarantee for another connection. Ctrl-C safely
preserves verified chunks. See [the download guide](DOWNLOAD-FORMAT.md).

Since 0.2.19, `pull` also fetches one optional file outside the compressed
package: the 37.5 MB decode forecast correction
`lookahead/tap-correction-attention-rank128-v1.safetensors`, verified by size and
SHA-256 against the pinned mirror commit. `--verify` reports it, an installed
model gets it by running `slotstream pull` again, and a model directory without
it runs the 0.2.16 forecast.

Weights placed elsewhere are used by passing that directory to `--model`, or
by symlinking it into the default location. Symlinked directories work from
0.2.1 onward.

### `slotstream doctor`

Show your Mac's memory plan, disk space, and estimated speed. This command
never loads the model and can run while the server is working.

| Flag | Meaning |
|---|---|
| `--sim-ram <gb>` | Preview this much RAM in decimal GB. Simulates memory capacity, not another chip or SSD. Assumes no other apps are using memory unless `--sim-available` is set; working set defaults to 75% of RAM. |
| `--sim-working-set <gb>` | Use this Metal working-set limit in the simulation. |
| `--sim-available <gb>` | Use this much available memory in the simulation. |
| `--max-context <auto or n>` | Default `auto` reports the automatic window, every candidate and the reason it was or wasn't taken. A number previews that window's allocation and reports the largest memory-feasible window under these inputs. |
| `--max-prefill-wait <minutes>` | Preview the request deadline separately from memory feasibility; default 30 minutes, `0` disables only time. |
| `--json` | The resolved plan as JSON, with estimates unrounded (`max_context_tokens`, `est_prefill_s_at_max_context`), plus `context_window_source` and, in auto mode, `automatic_context_window`. |

Plus the memory options, so `doctor --memory-gb 16` shows exactly what
`serve --memory-gb 16` would plan under the same conditions. Its estimated
prefill times use M5 Pro measurements; they exclude startup, queueing,
image preparation and reasoning before visible answer text. The full-window
estimate is a planning reference; actual requests also need room for a reply.

### `slotstream context-check`

Run an explicit capacity diagnostic with a synthetic prompt. Conversation
reuse is disabled by default; the retained-state mode first fills distinct
conversations and exercises their interleaved follow-ups. The report includes
actual prompt and reply IDs, compute shapes, sampled physical footprint and
swap observations. Incomplete output, exceeded process budgets and missing
required process-memory evidence fail qualification. Global paging is recorded
separately and does not fail capacity acceptance; it can exclude clean timing.

Stop any running model process first. Results are printed without writing
files; contributors can register them in the measurement records under `db/`.

| Flag | Meaning |
|---|---|
| `--tokens <n>` | Prompt length (default 8192); prompt plus the required reply must fit the model limit. |
| `--reply-tokens <n>` | Required nonempty output, reserved before model loading (default 16). An early stop is incomplete qualification. |
| `--max-prefill-wait <minutes>` | The same request deadline; `0` disables only this deadline. |
| `--wall-seconds <seconds>` | Independent hard duration bound per rung (default 7200). |
| `--plan-only` | Print the unqualified memory plan and resolved runtime controls without loading an engine. |
| `--warm-conversations <n>` | Fill and revisit retained conversations before the main request. Requires a single rung and enough configured room. |
| `--warm-tokens <n>` | Length of each distinct warm-up prompt. Its reply and follow-up must also fit the configured window. |
| `--sample-footprint` | Compatibility flag; this diagnostic always samples physical footprint in addition to lifetime process RSS and allocator telemetry. |
| `--ladder` | Run 2048, 4096, … up to `--tokens`, stopping at the first rung that leaves the plan. |
| `--min-free-gb <gb>` | Abort a pass when reclaimable memory falls below this (default: the planner's slack, 5% of RAM, at least 1.5 GB). |
| `--json` | One JSON object per rung. |

Plus the memory options; give it the same target you would give `serve`.
With retained conversations, the wall-clock ceiling covers both warm-up and
the main request. Diagnostic access above the public implementation ceiling
does not make that window supported by `serve` or `run`.

## Memory options

Shared by `run`, `serve`, `doctor`, and every check that loads the model.
With no sizing override, auto sizes the process to the machine (see
[memory defaults](../README.md#why-doesnt-slotstream-use-all-of-my-ram)).

| Flag | Meaning |
|---|---|
| `--model <name or dir>` | Model name (resolves to `~/.slotstream/models`, or a dev checkout's `models/`) or a directory path. |
| `--memory-limit-gb <gb>` | Adaptive total process ceiling, in decimal GB (development version). Can exceed the default model ceiling. The cache shrinks when other apps need memory and can grow back when it is available, within the saved limit and the Mac's supported budget. Cannot be combined with the fixed memory/cache options below. |
| `--memory-gb <gb>` | Total process memory budget, in decimal GB. The cache gets what remains after runtime, context, workspace and a nominal 1 GB margin. Near the minimum cache size, the plan can use part of that margin; `doctor` shows the actual planned headroom. Minimum 8.1 for the 32,768-token window; larger windows raise the minimum. This is a planning allowance, not an instruction to fill RAM. Conversation state and workspace use memory as needed, so measured usage can be lower. Auto picks the context window inside this target and preserves cache whose loss it cannot price; `--max-context N` chooses the context tradeoff explicitly. |
| `--experts-per-layer <n>` | Expert cache size directly, 1…512. Each of the 48 layers has 512 experts of 2.76 MB and the cache holds `n × 48` of them, so the pool is `n × 0.133 GB`: 30/layer is 4 GB, 181 is 24 GB, 226 is 30 GB. The pool is one global cache; hot layers borrow slots from cold ones. |
| `--pool-gb <gb>` | Raw expert-pool size (1 GB is about 7.5 experts per layer). |
| `--vision auto\|on\|off` | Accept images (default `auto`). `auto` loads the image encoder on first use; `on` also requires the checkpoint to contain vision weights; `off` rejects images. |
| `--mtp auto\|on\|off` | Speculative decode (default `auto`); see [Speculative decode](#speculative-decode). |
| `--gpu-keepalive auto\|on\|off` | `run` and `serve`: keep the GPU busy while a request generates (default `auto`). Streamed decode leaves the GPU idle between short bursts of work, and an idle GPU lowers its clock and starts the next burst late. A one-thread kernel on its own queue keeps it awake; outputs are unchanged. It costs power: `auto` keeps it on with AC power outside Low Power Mode and off on battery. See [GPU keepalive](ENGINEERING.md#gpu-keepalive-and-direct-demand-reads). |
| `--max-ram-percent <p>` | Auto only: the largest share of RAM auto may target (default 70). Alone it cannot raise the 33 GB base ceiling plus enabled draft/context charges. With `--memory-limit-gb`, it can further lower that ceiling; without this percentage option the adaptive limit is bounded by hardware and availability. Ignored when a fixed size is given. |

Precedence when several are given: `--experts-per-layer` beats `--pool-gb`,
which beats `--memory-gb`. An explicit size keeps its expert-pool policy. Physical headroom is still
checked before model loading and request growth. Preview it with `doctor`;
an unavailable window is reported separately from a request that may take
too long to prefill.

`--memory-limit-gb` keeps resizing enabled unless you also pass `--no-elastic`.
The requested limit stays saved even when the current target is lower. `doctor`
and model startup use the same budget feasibility check, including at the
default context window. Metal's recommendation is read from the system;
Slotstream keeps additional headroom and checks live reclaimable memory too.


A larger context reserves allocated cache capacity, retained conversations,
resident components and transient work before assigning the expert pool.
`doctor --json` includes the byte ledger and a discrete feasibility result;
a smaller history does not turn unused long-context reservation into free RAM.
Unknown prefill estimates appear as JSON `null` and remain deadline-bound.
Neither a fixed pool nor `--no-elastic` disables request memory checks.

The target is a budget, so current usage can be lower while reserved context
and temporary workspace are unused. `doctor` previews a new plan; it does not
change an already running server. `/api/ps` includes that server's actual
`details.memory_plan`, including its sizing source, target and expert pool.
For an adaptive limit, `memory_limit_gb` is the saved ceiling and `target_gb`
is the current budget, which can be smaller.

For `run`, the default memory summary separates the **lifetime footprint peak**
from **current footprint**. The lifetime value includes loading and all earlier
requests in that process, including GPU memory already freed. With
`--sample-footprint`, the summary instead labels the sampled generation peak.
Saved statistics retain `peakMemoryGB` as the maximum of lifetime footprint,
lifetime RSS and current footprint; `lifetimePhysicalFootprintPeakBytes` exposes
the native footprint peak separately. Sampling remains useful for attributing
memory to a particular request and can miss allocations between samples.

## Optimization defaults

The CLI resolves the selected optimization family automatically: compact
runtime state and n-gram rows, bounded prompt-read grouping, committed prompt
checkpoints, bounded output buffering and memory-governor response. Long
prompt grouping is admitted only when its additional workspace fits; ordinary
chronological passes remain the fallback. Explicit prefill and optimization
controls retain their precedence and validation. With the draft head, the
[decode lookahead](#decode-lookahead) is on by default too, and so is the
[split verify attention](#speculative-decode) from 6,144 tokens of context.
Experts missing from the cache are read into host memory and copied straight
into their cache slots, without staging arrays or a GPU scatter, and on AC
power the [GPU keepalive](#memory-options) runs while a request generates.

Prompt checkpoints help only when the token and image history actually
matches. Use `--no-prefix-cache` for comparisons that require fresh prompt
computation. Query tiling bounds vision attention workspace; it does not grant
extra permanent expert-cache capacity. The specialized fused rotation keeps
its qualified platform and shape checks, with the original rotation elsewhere.

The `SLOTSTREAM_OPT_` switches remain available for reference comparisons and
qualification. They include experimental paths that were rejected or remain
conditional. Enabling every switch is not the selected configuration.
[Integrated measurements](../MEASUREMENTS.md#final-integrated-optimization-results)
report the tested workloads and limits; [the unified plan](../PLAN.md) retains
the disposition of each candidate.

The automatic ceiling is a conservative default based on development-Mac
measurements, separate from RAM-share and physical-memory bounds. The
[M5 Max community sweep](HARDWARE.md#does-more-memory-help) demonstrates gains
from larger manual targets; auto is not calibrated to every hardware profile.
See
[why auto retains a ceiling](ENGINEERING.md#memory) and the
[operating-policy contract](../db/records/design/measured-operating-policies.md).

## Environment variables

| Variable | Read by | Meaning |
|---|---|---|
| `SLOTSTREAM_WEIGHTS_SOURCES` | `pull` | Comma-separated raw-file bases tried in order. Selects raw transport in automatic mode; every file must match the original pins. |
| `SLOTSTREAM_COMPRESSED_SOURCES` | `pull` | Comma-separated Slotpack package bases; every object must match the embedded package. |
| `SLOTSTREAM_PULL_TRANSPORT` | `pull` | `automatic`, `compressed`, or `raw`; an explicit non-automatic CLI flag takes precedence. |
| `SLOTSTREAM_PULL_CONNECTIONS` | `pull` | Fixes parallel connections, capped at 32; the CLI flag takes precedence. |
| `SLOTSTREAM_PREFIX_CACHE` | engine | `0` disables conversation prefix reuse, like `--no-prefix-cache`. |
| `SLOTSTREAM_PREFILL_CHUNK` | engine | Override the largest prefill pass in tokens instead of taking it from the memory plan; the schedule still shrinks it as the context grows. Measurement work only. |
| `SLOTSTREAM_IO_QUEUE_DEPTH` | engine | Expert read parallelism, 1…128 (default 12). The development-Mac sweep found little benefit from 12 to 32; other SSDs and workloads can differ. |
| `SLOTSTREAM_EXPERT_LOAD_BATCH` | engine | Expert records staged at once during prefill, 1…512 (default 32): the sweep's group size on a pass of 256 tokens or more, the pool's load slice below that. Bounds peak memory on long prompts. |
| `SLOTSTREAM_SWEEP` | engine | `0` selects the older pool path instead of the prefill sweep when ordinary prefill is used. A/B work only; slower in the recorded development-Mac comparisons. Other optimization controls can select a different qualified path. |
| `SLOTSTREAM_SWEEP_ADMIT` | engine | `0` stops the last pass of a prompt from admitting the prompt's hottest experts into the pool, so decode starts cold. A/B work only. |
| `SLOTSTREAM_SWEEP_TRACE` | engine | `1` prints, after each prefill, where the sweep's time went: reads, waiting for the GPU, sorting rows, copies out of the pool, and MLX's peak and cache. |
| `SLOTSTREAM_PREFILL_CACHE_MB` | engine | MLX buffer-cache cap while a prompt is read. The plan sets 512 at targets of 12 GB and under (the sweep's varying array sizes otherwise fill the 2 GB cache, 1.7 GB of peak at the floor) and no cap above, where it costs ~6% of prefill; this forces a value at any target. |
| `SLOTSTREAM_OPT_EXPERT_PREFETCH` | engine | `0` turns off the decode lookahead and its 373 MiB charge (409 MiB with the checkpoint's forecast correction file). `1` selects an experimental prefetch configuration from the `SLOTSTREAM_EXPERT_PREFETCH_*` tuning variables instead; for comparisons only. |
| `SLOTSTREAM_OPT_ROUTER_WEIGHTS` | engine | `0` or `1` overrides the FP32 router weight cache that the decode lookahead turns on. |
| `SLOTSTREAM_OPT_VERIFY_SPLIT` | engine | `0` runs the speculative verify pass through the dense attention kernel at every context, the previous behavior. The default splits it into two-row vector-kernel calls from 6,144 tokens of context. |
| `SLOTSTREAM_OPT_VERIFY_SPLIT_CONTEXT` | engine | Context, in tokens, from which the verify pass splits (default 6144, the measured crossover on the development Mac); `0` splits at every context. |
| `SLOTSTREAM_OPT_ROW_INVARIANT` | engine | `1` selects the exact mode: the model's small dense matmuls run through one kernel at every row count, and the split verify attention makes one call per row. With `SLOTSTREAM_OPT_VERIFY_SPLIT_CONTEXT=0`, speculative and plain decode give identical output for draft depths up to 4. It changes plain decode's rounding, so it is off by default. |
| `SLOTSTREAM_MTP_EXPERTS` | planner | `resident` or `streamed` forces where the draft head keeps its 512 experts; unset or `automatic`, they stay resident on a cache of 76 experts per layer or more after the resident charge and stream below it. For comparisons. |
| `SLOTSTREAM_GPU_KEEPALIVE` | engine | `auto`, `on` or `off`: the default for `--gpu-keepalive`. Any other value is refused. |
| `SLOTSTREAM_OPT_DIRECT_DEMAND` | engine | `0` reads cache misses through staging arrays and a GPU scatter, the previous path; the default reads them into host memory and copies each record straight into its slot. Both put the same bytes in the same slots. |
| `SLOTSTREAM_DECODE_BARRIER_LAYERS` | engine | Layers between GPU drains, 1…48. The decode lookahead uses 4; `1` drains after every layer. A pass that could not keep that many layers of experts pinned drains after every layer anyway. |
| `SLOTSTREAM_EXPERT_PREFETCH_TAP` | engine | `boundary` keeps the 0.2.16 forecast (the layer-boundary router forecast at stride 2) when the correction file is present, charging 373 MiB instead of 409; the configuration 0.2.19 was benchmarked against. Other values belong to the experimental configuration and are for comparisons only. |
| `SLOTSTREAM_ROOT_DIR` | installer | Install somewhere other than `~/.slotstream`. |
| `SLOTSTREAM_RELEASE_BASE` | installer | Fetch the release from another base URL (CI uses it to test unpublished builds). |

## Checks and diagnostics

These checks help diagnose an installation. The first group needs no weights;
the second loads the model. Use small explicit memory targets for model
checks (`--memory-gb 8.1` to `10`) and stop other model processes first.
See [Testing](TESTING.md) for the full suites.

Fixed-profile diagnostics reject `--memory-limit-gb` because they use their
own bounded allocation. `prefix-exact-check` accepts it with `--plan`;
`elastic-drill` can exercise it within the diagnostic's separate memory ceiling.

**Weights-free**

| Command | Proves |
|---|---|
| `runtime-check` | Process RSS accounting and the prefix cache's four-conversation bound. |
| `governor-check` | The elastic resize policy across pressure, availability, and cooldowns. |
| `sampler-golden` | Sampling from reproducible synthetic logits, compared against `Tools/sampler_ref.py`. Flags: `--vocab`, `--draws`, `--seed`, `--logit-seed`, `--temperature`, `--top-p`, `--top-k`, `--min-p`, `--presence-penalty`, `--accumulate`. |
| `pull-check` | Same-size corruption detection and HTTP range validation in the downloader. |
| `prefill-schedule` | The prefill passes a prompt runs at a given pass size and the wait they imply; the same wait arithmetic `doctor` and the 400 message use. JSON includes actual canonical pass sizes, physical query rows and key extents, including numerical padding. `--chunk` (4096), `--tokens` (32768), `--from` (0), `--json`. |

**Load the model**

| Command | Proves |
|---|---|
| `elastic-check` | Greedy output is byte-identical across a live pool grow and shrink. `--max-tokens` (24), `--big-slots` (960; lower it on small machines). |
| `elastic-drill` | Drives the live governor through shrink, cooldown, recovery and exact output. `--slots` (4000), `--max-memory-gb` (10), `--quick` skips recovery. Use `--memory-limit-gb 10 --max-memory-gb 10 --mtp off` for small-cache pressure recovery, or `--slots 1000 --max-memory-gb 13 --memory-limit-gb 13 --mtp off` for the full availability drill. Both require the target plus 3 GB physically reclaimable and report sampled memory and swap. An insufficient ceiling refuses before model allocation. |
| `prefix-check` | Conversation prefix reuse is equivalent, bounded, and deterministic. `--slots` (640), `--max-tokens` (24). |
| `prefix-exact-check` | A continued conversation computes what a cold one does: same tokens and bit-identical prompt logits whether a turn resumed a retained state or read its whole prompt, with reuse still happening, an identical prompt reusing its complete state, an edited history rebuilding, and a second conversation resuming a shared prefix. `--slots` (640), `--max-tokens` (24), `--plan` to run a real memory plan so `--memory-gb` and `--mtp` apply. |
| `sweep-check` | The prefill sweep (passes of 256 tokens or more) stays inside the prefill-rechunk band against the pool path, is deterministic, gives bit-identical logits on a cold and a warm pool, and leaves the pool consistent after admission. `--slots` (640). |
| `decode-overlap-check` | Direct demand reads and the GPU keepalive leave output exact: the same ids as the staged reads with the keepalive off, on a cold floor-sized cache, with and without the draft head, and a failed direct read leaves no stale slot. `--tokens` (24). |
| `draft-stream-check` | A draft head whose experts stream decodes the same ids as a resident head at a 12 GB target, reads and reuses its cached experts, and a failed draft expert read ends only its own request; plain decode with the lookahead decodes the same ids as without it at 10 GB. `--tokens` (32). |
| `parity` | N truncated layers match the Python reference dumps. `--layers` (4), `--tokens`, `--compare <dir>`, `--out <dir>`. |
| `template-check` | Renders the chat template for a canned conversation and prints token ids. `--think`. |
| `ngram-golden` | Prints n-gram row ids for a token sequence, for comparison with Python. `--tokens`. |
| `dequant-golden` | CPU-dequantizes one n-gram row for comparison with `mx.dequantize`. `--gid` (12345). |

<a id="new-in-020"></a>

## Speculative decode

- `--mtp auto|on|off` on `run`, `serve`, and `doctor`: speculative decode with the
  model's draft head, `mtp.safetensors`, which `pull` fetches with the
  weights. The file is optional; downloads can complete without it. `on`
  without the file is an error; `auto`, the default, turns it on when the
  cache still reaches 28 experts per layer after the head's charge, before the
  separate lookahead reservation. A 12 GB target qualifies at the default
  context; actual availability and the selected window can keep the head off
  on a larger Mac. On a cache of 76 experts per layer or more after the full
  1.6 GB charge, the head keeps its 512 experts resident. Below that it reads
  them from the SSD through a 64-expert cache of its own, which charges 0.4 GB
  and leaves the rest to the main cache; the output is the same either way.
  At a 12 GB target the head with streamed experts decoded 1.23x faster than
  plain decode with the lookahead, where a resident head only tied
  ([decision](../db/records/decisions/draft-head-streams-its-experts-below-76-per-layer.md)).
  `SLOTSTREAM_MTP_EXPERTS=resident` or `streamed` forces one placement. Before
  0.2.16 the floor was 120, and then 76; two drafts measured 31.7% faster
  than plain decode on the same memory at 76 per layer
  ([decision](../db/records/decisions/draft-head-auto-floor-76-per-layer.md)).
  The floor is separate from draft depth. The historical one-draft measurement
  at the former 28 GB memory target was ×1.24 decode; MEASUREMENTS.md M9
  preserves its configuration, ladder and ceiling.
- `mtp-parity`, `mtp-accept`, `mtp-check`, `mtp-rowcheck`: the draft head's
  parity with the Python reference, its measured accept rate (`--depth`,
  default 4), the speculative-decode gates, and the row-equality gate: in the
  exact mode, every row of a two-row and a three-row verify pass must equal
  the one-row pass at its position in the same mode bit for bit, and the
  state a three-row pass leaves must equal the state three one-row passes
  leave, on a prompt whose positions cross 1,024 keys (`--cross`), where the
  attention kernel changes, and one above the indexer budget (the stock
  deviation is reported alongside).
- The verify pass checks the drafts in one pass of draft depth plus one rows.
  The backend's vector attention kernel takes at most two such rows at this
  model's head layout, so a three-row pass used to fall to the dense kernel,
  which reads every cached key and whose cost grows with the context. From
  6,144 tokens of context the pass now runs two rows at a time through the
  vector kernel, the kernel plain decode's attention uses. The gain grows with
  the context. On a quiet machine at a 22 GB target, speculative decode
  measured 11.82 against 11.67 tok/s (x1.013) with a 16,356-token prompt,
  x1.085 on a second 16,356-token prompt and x1.29 with a 32,740-token prompt;
  the fetch-free pass is 32% cheaper at 32,740 tokens and 45% at 65,508. The
  split changes which drafts are accepted, so the 16k gain follows the prompt,
  1% and 8% in the two measured, while at 32k both arms accept alike.
  `SLOTSTREAM_OPT_VERIFY_SPLIT=0` restores the dense pass. Below the threshold
  the dense kernel is faster
  ([measurement](../db/records/measurements/speculative-verify-pass-split-attention-2026-09-17.md)).
- Exact mode: a multi-row pass rounds a little differently from one-row
  passes, so speculative and plain decode can pick different tokens at near
  ties. `SLOTSTREAM_OPT_ROW_INVARIANT=1` with
  `SLOTSTREAM_OPT_VERIFY_SPLIT_CONTEXT=0` removes the difference for draft
  depths up to 4. The small dense matmuls use one kernel at every row count,
  and each verify row attends in its own call over the keys plain decode
  reads, so the backend picks the same attention kernel. Every row of a verify
  pass then equals plain decode in the same mode, and a speculative run's
  output equals a plain run's. The mode changes plain decode's rounding too,
  and it costs 4 to 6% of plain decode's speed, so it is off by default;
  `mtp-rowcheck` gates it.
- `SLOTSTREAM_DRAFT_DEPTH`: draft chain depth, 1–16 (default 2).
  Two drafts are the adopted operating choice for mixed workloads. The recent
  automatic-memory comparison found two and three effectively tied overall;
  this is not a universal performance optimum. Rejected drafts roll back to
  recorded state. Explicit valid overrides still work; invalid values fall
  back to the default. See the [decision and evidence](../db/records/decisions/draft-depth-defaults-to-two.md).
  This changes draft depth when MTP is enabled, not the automatic activation
  floor or the RAM-share default.

### Decode lookahead

From 0.2.16 the decode lookahead runs by default with the draft head when
the cache reaches the head's floor before the lookahead's own reservation.
Without the head it now runs in plain decode too, from 20 experts per layer
before its reservation: at a 10 GB target it made plain decode 1.11x faster.
The final printed cache can therefore be smaller. As each layer's
routing comes back, the engine reads the model's state after the previous layer's
attention step, applies the next layer's router to it, corrects the result with a
small learned table shipped for the checkpoint
(`lookahead/tap-correction-attention-rank128-v1.safetensors`, 37.5 MB, which
`pull` fetches next to the weights), and reads the experts it picks from the SSD
straight into cache slots before that layer asks for them. It also keeps FP32
copies of the router weights and drains the GPU every four layers instead of
every layer. Output is unchanged.

The memory plan charges 373 MiB for it before sizing the expert cache, 409 MiB
when the correction file is present, and `doctor` prints `lookahead: on` with the
charge. On twelve held-out prompts at a 20 GB target with two drafts, the 0.2.16
lookahead decoded 1.11x faster than without it; the corrected forecast of 0.2.19
decoded 1.10x faster than the 0.2.18 forecast on eight held-out prompts at a 22 GB target,
14.38 to 15.86 tok/s (see the [expert lookahead guide](EXPERT-LOOKAHEAD.md)). A head
forced onto a smaller cache runs without it. `SLOTSTREAM_OPT_EXPERT_PREFETCH=0`
turns it off, `SLOTSTREAM_EXPERT_PREFETCH_TAP=boundary` keeps the 0.2.16 forecast
with the file present, and the two variables above override its router cache and
drain period. The
[decision](../db/records/decisions/decode-lookahead-default-with-the-draft-head.md)
and the [corrected forecast decision](../db/records/decisions/corrected-decode-forecast-default-with-the-sidecar.md)
record the evidence and limits.


<!-- ===== docs/API.md ===== -->

# HTTP API

For client configuration and integration troubleshooting, start with
[Connect apps and agents](CLIENTS.md).

Start the server with `slotstream serve`. It listens on **127.0.0.1:11434**;
use `--port N` to choose another port. It has no authentication, so local
processes can use it. Browser requests must come from an allowed loopback
origin. See [Security](../SECURITY.md).

This page covers the Ollama-style `/api/*` endpoints, the OpenAI-style
`/v1/*` endpoints, and the Anthropic Messages API at `/v1/messages`. For the
AI SDK gateway, see the [fx guide](FX.md). OpenAI tool calling is described
below. The OpenAI tool and reasoning additions require Slotstream 0.2.8 or
later; the Responses API requires Slotstream 0.2.20 or later; the Messages
API requires Slotstream 0.2.21 or later. Use `qwen3.8-flash-next:4bit` as the
model name.

Unknown fields, unsupported features, and malformed values return a 400
error describing the problem, except on `/v1/messages`, which ignores unknown
top-level fields as described there. A wrong model name returns 400, or 404
on `/api/show` and `/v1/messages`. Some client compatibility fields are
accepted without an effect; these are listed below.

## Endpoints

| Endpoint | What it does |
|---|---|
| `POST /api/chat` | Chat completion in Ollama format; streams by default |
| `POST /api/generate` | Prompt completion in Ollama format; streams by default |
| `POST /v1/chat/completions` | Chat completion in OpenAI format; doesn't stream by default |
| `POST /v1/responses` | Response in OpenAI Responses format, the API Codex uses; doesn't stream by default |
| `GET`/`DELETE /v1/responses/{id}` | Returns 404; responses aren't stored |
| `POST /v1/messages` | Message in Anthropic Messages format, the API Claude Code uses; doesn't stream by default |
| `POST /v1/messages/count_tokens` | Counts a Messages request's prompt tokens |
| `GET /v1/models` | Lists the model in OpenAI format |
| `GET /api/tags` | Lists the model in Ollama format |
| `GET /api/ps` | Reports the loaded model and its current memory use |
| `POST /api/show` | Returns model metadata and capabilities |
| `GET /api/version` | Returns `{"version": "..."}` |
| `POST /api/embed`, `/api/embeddings` | Returns 400; embeddings aren't supported |
| `POST /api/pull`, `/api/create` | Returns 501; use `slotstream pull` on the host |
| `GET /slotstream/status` | Reports the server's process, window and use; see [server status](#server-status) |
| `POST /slotstream/clients` | Keeps a server started with `--idle-exit` running while a process runs |

`/api/show` accepts `model` (or the deprecated `name` alias) and optional
`verbose`. Empty `system`, `template`, and `options` fields are accepted for
Ollama CLI compatibility; non-empty overrides return 400.

## `/api/chat`

Accepted fields: `model`, `messages`, `stream` (default `true`), `think`
(boolean), `options`, and `keep_alive`. `keep_alive` has no effect because
the server keeps the model loaded.

Each message has a `role` and `content`, with optional `images`. Content can
be text or an array of supported image/text parts; see [Images](#images).
Tool calls aren't supported on this endpoint. Tool clients should use
`/v1/chat/completions` with OpenAI function definitions and tool-result messages.

```bash
curl localhost:11434/api/chat -d '{
  "model": "qwen3.8-flash-next:4bit",
  "messages": [{"role": "user", "content": "Hello"}],
  "stream": false,
  "options": {"temperature": 0.2, "seed": 7}
}'
```

`options` accepts `temperature`, `top_p`, `top_k`, `min_p`,
`presence_penalty`, `num_predict`, `seed`, and `stop` (a string or array).
JSON `null` is treated as an unset field.

With `think: true`, reasoning appears in `message.thinking` and the answer
in `message.content`, for both streamed and complete responses. If the token
budget runs out during reasoning, `content` is empty.

## `/api/generate`

Accepted fields: `model`, `prompt`, `system`, `raw`, `stream`, `think`,
`images`, `keep_alive`, and the same `options` as chat. Empty `suffix` and
`template` fields are accepted for Ollama CLI compatibility. A non-empty
suffix or template override returns 400.

`think: true` returns reasoning in `thinking` and the answer in `response`.
`raw: true` sends the prompt without the chat template and can't be combined
with a system prompt, thinking, or images.

An empty prompt acknowledges Ollama's load request with
`done: true, done_reason: "load"`. `/api/chat` does the same for an empty
message list. The model is already loaded in either case.

## `/v1/chat/completions`

Set an OpenAI-compatible client's base URL to `http://localhost:11434/v1`.
If it requires an API key, use any placeholder string. For example, with the
Python OpenAI SDK installed:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="unused")
reply = client.chat.completions.create(
    model="qwen3.8-flash-next:4bit",
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)
```

Accepted fields: `model`, `messages`, `stream`, `temperature`, `top_p`,
`top_k`, `min_p`, `presence_penalty`, `max_tokens` / `max_completion_tokens`,
`seed`, `stop`, `stream_options` (`{"include_usage": true}`), `tools`,
`tool_choice`, `parallel_tool_calls`, and `reasoning_effort`. `top_k`,
`min_p`, `think` (boolean), and `options.num_ctx` are slotstream extensions.
Without `max_tokens` or `max_completion_tokens`, a reply may use a quarter of
the served window, at most 8,192 tokens and never more than the room the
prompt leaves. `num_ctx` may lower the
request's prompt-plus-reply budget; it cannot exceed the served context.
JSON `null` is treated as unset.

For SDK compatibility, these fields are accepted only at the listed values:
`n: 1`, `frequency_penalty: 0`, `logprobs: false`, `logit_bias: {}`,
`response_format: {"type": "text"}`. Other values for these options return
400. These are accepted and have no effect, because nothing is stored or
billed: `store: false`, `metadata` (an object of text values), and the text
fields `user`, `prompt_cache_key`, `prompt_cache_retention`,
`safety_identifier` and `service_tier`. `store: true` returns 400: it asks
for a completion to fetch later, and the server keeps none.

Function tools use OpenAI's `{"type":"function","function":{"name":...,
"description":...,"parameters":...}}` shape. The server renders their schemas
with the model's native template and converts complete generated calls into
`message.tool_calls`, each with an `id`, `type: "function"`, and a function
name plus JSON argument string. The finish reason is `tool_calls`. Send the
assistant message back unchanged, followed by a `role: "tool"` message with
the matching `tool_call_id` and textual result. Each outstanding call needs
exactly one result before the next conversation message. Results may arrive
in any order; the adapter matches their IDs and restores call order for the
native model template.

`tool_choice` accepts `auto` (default), `none`, `required`, or a named
function object. A required/named choice is both prompted and checked; an
unsatisfied choice produces an inference error unless the output token budget
was exhausted. `parallel_tool_calls: false`
ends generation after the first complete call. The default allows multiple
calls, with distinct IDs and stream indices. The caller executes tools.
Chat Completions streams a declared string argument while it is generated,
including large file contents. Reassemble `delta.tool_calls` by `index`,
concatenating each function's `arguments` fragments. Calls become complete
only when their closing tags arrive. If the output budget is exhausted,
the reply ends with `finish_reason: "length"`, requested usage, and `[DONE]`,
including when a required tool call has not finished. Partial argument strings
can be incomplete JSON; do not execute them. A completed tool reply ends with
`finish_reason: "tool_calls"`. Non-streaming replies use the same length
semantics. Malformed calls on a normal stop and undeclared function names
produce an inference error. Strict
schema enforcement is unavailable: omit `strict` or use `false`, and validate
arguments in the caller before execution.

`reasoning_effort: "none"` or `"minimal"` disables reasoning. `low`, `medium`,
`high`, `xhigh`, and `max` enable it, using the same model mapping as the
gateway. Reasoning is returned separately in `reasoning_content`, which is
accepted on assistant history messages. A conflicting `think` flag is a 400.
Initial `system` and `developer` instructions are combined in order.

## `/v1/responses`

The OpenAI Responses API, which Codex and newer OpenAI SDK code use. Set the
client's base URL to `http://localhost:11434/v1`; a placeholder API key works.
For Codex, follow the [Codex guide](CODEX.md).

```bash
curl localhost:11434/v1/responses -d '{
  "model": "qwen3.8-flash-next:4bit",
  "instructions": "Answer briefly.",
  "input": "What is 2+2?"
}'
```

Accepted fields: `model`, `input` (text or an array of items),
`instructions`, `tools`, `tool_choice`, `parallel_tool_calls`,
`reasoning.effort`, `max_output_tokens`, `temperature`, `top_p`, and
`stream` (default `false`). JSON `null` is treated as unset.

Input items are `message` (roles `system`, `developer`, `user`, `assistant`;
content as text or `input_text`, `output_text`, and `input_image` parts),
`reasoning`, `function_call`, and `function_call_output` (text, or content
items with text and `input_image` parts). System and developer messages
before the conversation join `instructions`; a later one renders as user
text, which is what Codex does with its own context messages. Function call
outputs may arrive in any order; the adapter matches `call_id` and restores
the call order for the native model template. An `input_image` needs an
inline `image_url` data URL; see [Images](#images).

Tools use the Responses shape, `{"type":"function","name":...,
"description":...,"parameters":...}`. A `namespace` tool is flattened: each
member renders as `namespace.name`, and a call to it is reported with the
`name` and `namespace` fields split again. Hosted `web_search` and
`file_search` tools are dropped because the model cannot run them. Other
tool types, `strict: true`, and the `text.format` constrained-output
setting return 400, as on `/v1/chat/completions`. `tool_choice` accepts
`auto`, `none`, `required`, `{"type": "function", "name": ...}`, or
`{"type": "custom", "name": ...}` for a freeform tool; `allowed_tools`
returns 400.

Accepted without effect, because the server has nothing to change:
`store` (nothing is stored either way), `include`, `metadata`,
`prompt_cache_key`, `prompt_cache_retention`, `safety_identifier`, `user`,
`service_tier`, `stream_options`, `text.verbosity`, `truncation:
"disabled"`, and the Codex fields `client_metadata` and `access_programs`.
Refused with 400: `previous_response_id` and `conversation` (no responses are
stored, so every request carries its whole conversation), `background:
true`, `prompt` templates, `context_management`, `max_tool_calls`,
`top_logprobs`, `truncation: "auto"`, `input_file` and `input_audio` parts,
`file_id` images, encrypted reasoning or tool output from another provider,
and an input that ends with an assistant item.

`reasoning.effort` uses the same mapping as `reasoning_effort` on the chat
endpoint: `none` and `minimal` disable thinking, the other levels enable it.
The model's reasoning streams as `reasoning_summary_text` deltas and is
returned as the reasoning item's `summary`, which clients echo back on the
next turn. Without `max_output_tokens` the reply budget is the
`max_output_tokens` value `/v1/models` reports, a quarter of the served window
up to 8,192 tokens, bounded by the room the prompt leaves.

The response object, returned whole without `stream` and carried by the
`response.created` and `response.completed` events, has `id`, `status`,
`model`, `output`, and `usage`, and repeats the request's `tools`,
`tool_choice`, `parallel_tool_calls`, `instructions`, `temperature`,
`top_p`, `max_output_tokens`, `reasoning`, and `text` as the API does. It
reports `store: false` and a null `previous_response_id` because nothing is
kept.

A streamed reply is Server-Sent Events, each with an `event:` line and a
`data:` line: `response.created`, `response.in_progress`,
`response.output_item.added` and `.done`, `response.content_part.added` and
`.done`, `response.output_text.delta` and `.done`,
`response.reasoning_summary_part.added`, `.delta` and `.done`,
`response.function_call_arguments.delta` and `.done`, and
`response.completed` with `usage`. During a long prompt read the server sends
a `response.in_progress` event every 10 seconds, because Codex's idle timeout
counts events rather than bytes. Running out of tokens before the reply ends
is `response.incomplete` with `incomplete_details.reason:
"max_output_tokens"`, unless a complete function call was already delivered,
in which case the response completes. An inference failure after the stream
starts is a `response.failed` event carrying `error.code` and `error.message`,
with no `response.completed` after it. Function calls are delivered whole:
`response.output_item.added`, one arguments delta, `arguments.done`, and
`output_item.done`, only once the model's call block is complete.

## `/v1/messages`

The Anthropic Messages API, which Claude Code and the Anthropic SDKs use. Set
the client's base URL to `http://127.0.0.1:11434`, without `/v1`; any API key
or token works. For Claude Code, follow the [Claude Code guide](CLAUDE-CODE.md).

```bash
curl localhost:11434/v1/messages -H 'content-type: application/json' -d '{
  "model": "qwen3.8-flash-next:4bit",
  "max_tokens": 256,
  "messages": [{"role": "user", "content": "What is 2+2?"}]
}'
```

With the Python SDK:

```python
import anthropic

client = anthropic.Anthropic(base_url="http://127.0.0.1:11434", api_key="unused")
reply = client.messages.create(
    model="qwen3.8-flash-next:4bit",
    max_tokens=256,
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.content[0].text)
```

Accepted fields: `model`, `messages`, `max_tokens` (required), `system` (text
or text blocks), `stop_sequences`, `stream` (default `false`), `temperature`,
`top_p`, `top_k`, `tools`, `tool_choice`, `thinking`, and
`output_config.effort`. JSON `null` is treated as unset. `metadata`,
`context_management` and `service_tier` are accepted without effect.
`container`, `mcp_servers` and `output_config.format` return 400, because
they ask for work this server cannot do.

Other top-level fields are ignored rather than refused, because Claude Code
adds request fields in most releases. The reply names them in an
`X-Slotstream-Ignored-Fields` header, and the server's output, in its window
or its log, names each one once. Inside messages and tools, an unknown content block type or tool
type returns 400, and other fields, such as `cache_control`, are ignored.
Headers such as `anthropic-version` and `anthropic-beta` are not required
and have no effect.

Messages alternate between `user` and `assistant`; consecutive messages of
one role are joined into one turn, as the Messages API treats them. User content is text or blocks: `text`, `image` (a
`base64` source in JPEG, PNG, GIF or WebP; see [Images](#images)),
`document`, `search_result` (read as its title, source and text), and
`tool_result`. A `document` with a plain-text source is read inline; a PDF,
URL or file document is replaced by a note telling the model it cannot see
it, so the conversation can go on. Assistant content is
`text`, `thinking`, `redacted_thinking` (skipped, since it is encrypted by
another provider) and `tool_use`. A `system` message before the conversation
joins `system`; a later one renders as user text, which is where Claude Code
puts its environment details. Its `tool_addition` and `tool_removal` blocks
are skipped, and any other non-text block returns 400. `cache_control`
markers have no effect: the server reuses prompts on its own. Image `url` and
`file` sources return 400, and so does a conversation that ends with an
assistant message, except in a token count.

Every `tool_use` needs a `tool_result` with its id in the next user turn,
which may span consecutive user messages, and the results may come in any
order. A result's content is text or `text`,
`image`, `document`, `search_result` and `tool_reference` blocks (read as
`Tool available: <name>`); `is_error: true` prefixes the text with `Error:`.
Tools use `name`, `description` and `input_schema`. Server tools the API runs
itself, such as `web_search`, `code_execution` and `advisor`, are dropped
because the model cannot run them, and a `tool_choice` that names one returns
400; other typed tools return 400. `strict` is
accepted but not enforced: calls are checked to be complete JSON objects,
not validated against the schema. `tool_choice` accepts `auto`, `any`,
`none`, and `{"type": "tool", "name": ...}`, with
`disable_parallel_tool_use`, and is prompted and checked as on
`/v1/chat/completions`.

Thinking is off unless `thinking` asks for it. `{"type": "adaptive"}` turns
it on at `output_config.effort` (`high` when absent), and `{"type":
"enabled", "budget_tokens": N}` at an effort chosen from the budget; both use
the same model mapping as `reasoning_effort` on the chat endpoint.
`{"type": "disabled"}` turns it off. `display: "omitted"` returns thinking
blocks with empty text; any other `display` value shows it. Each thinking block carries a `signature` that holds
its reasoning, so a client that sends the block back, even with its text
omitted as Claude Code does, gives the model its earlier reasoning.

Without a stop, the reply ends with `stop_reason` `end_turn`, or `tool_use`
after a complete tool call, `max_tokens` when the budget runs out, or
`stop_sequence` with the matched `stop_sequence`. When `max_tokens` was
larger than the room the prompt left in the window and the reply filled that
room, the reason is `model_context_window_exceeded` instead of
`max_tokens`. `usage` reports
`input_tokens` (the prompt tokens read for this request),
`cache_read_input_tokens` (the tokens reused from an earlier request),
`cache_creation_input_tokens` (always 0) and `output_tokens`; the first two
add up to the whole prompt.

A streamed reply is Server-Sent Events in Anthropic's order:
`message_start` (sent once the request is admitted), `ping`,
`content_block_start`, `content_block_delta` (`thinking_delta`,
`signature_delta`, `text_delta` or `input_json_delta`), `content_block_stop`,
`message_delta` with `stop_reason` and `usage`, and `message_stop`. During a
long prompt read the server sends a `ping` every 10 seconds, or, while a
thinking block with omitted text is open, an empty `thinking_delta`.
`message_start` reports the prompt tokens read and reused for the request. A tool call is
delivered whole: its start, one `input_json_delta` with the complete input,
and its stop, once the model's call block is complete. An inference failure
after the stream starts is an `error` event, and nothing follows it.

Errors use Anthropic's shape, `{"type": "error", "error": {"type": ...,
"message": ...}}`, with `invalid_request_error` for 400, `not_found_error`
for 404, `request_too_large` for 413, `api_error` for 500, and
`overloaded_error` for 503. A prompt longer than the served window fails
with `prompt is too long: N tokens > M maximum`, the message Claude Code
reads to compact its conversation.

`POST /v1/messages/count_tokens` takes the same body without `max_tokens`
and returns `{"input_tokens": N}`: the prompt the model would read, with the
same template, tools, thinking setting and images. It needs no generation and
answers while another request runs.

## Sampling defaults

The table applies to ordinary chat. Tool-enabled requests default to temperature
0.2, top_p 0.9 and presence_penalty 0 to preserve the repeated tool grammar.
Reasoning without tools uses the existing thinking profile. Explicit request
values override these defaults.

| Option | Default |
|---|---|
| `temperature` | 0.7 |
| `top_p` | 0.8 |
| `top_k` | 20 |
| `min_p` | 0 |
| `presence_penalty` | 1.5 |
| `num_predict` / `max_tokens` | 512 on the Ollama endpoints; a nonpositive `num_predict` uses the remaining context. On `/v1/chat/completions` and `/v1/responses`, a quarter of the served window, at most 8,192 tokens; their output limits must be positive. |
| `seed` | Random for each request |
| `stop` | None |

Set `seed` for reproducible sampling. For comparisons, keep the model,
prompt, and generation settings fixed and start the server with
`--no-prefix-cache`: reusing conversation state can change nearly tied
outputs. Out-of-range sampling values are clamped to supported ranges.

## Streaming

Ollama endpoints stream newline-delimited JSON. The final object has
`done: true`, `done_reason`, `prompt_eval_count`, and `eval_count`.

The OpenAI endpoint streams Server-Sent Events (SSE) as `data:` lines ending
with `[DONE]`. Its first delta includes `"role": "assistant"`.

Text is sent incrementally. Incomplete UTF-8 characters and possible stop
sequences are held back until resolved. Concatenating the text deltas gives
the same text as a non-streamed response under the same generation
conditions; `Tools/api_robustness.sh` checks this.

Tool streams carry `delta.tool_calls` with an `index`, ID, function name and
complete argument string. The final choice has `finish_reason: "tool_calls"`.
Reasoning uses `delta.reasoning_content`; it is excluded from answer text.
An inference error after streaming starts is an SSE `error` object followed
by connection termination, without a successful finish or `[DONE]` marker.

## Images

Every API accepts images, using these request shapes:

| API | Image field |
|---|---|
| Ollama chat | `images: [base64]` on the user message |
| Ollama generate | `images: [base64]` on the request |
| OpenAI chat | An `image_url` content part with a `data:` URL |
| OpenAI Responses | An `input_image` part with an `image_url` data URL, in a message or a `function_call_output` |
| AI SDK gateway | A `file` part with an `image/*` media type and inline `data`: a base64 string, or the `{type: "data", data}` object `@ai-sdk/gateway` sends |

This Python 3 example sends `cat.jpg` to the Ollama chat endpoint. It uses
only the standard library:

```python
import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

image = base64.b64encode(Path("cat.jpg").read_bytes()).decode("ascii")
body = {
    "model": "qwen3.8-flash-next:4bit",
    "messages": [{"role": "user", "content": "What is in this picture?",
                  "images": [image]}],
    "stream": False,
}
request = Request(
    "http://localhost:11434/api/chat",
    data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json"},
)
with urlopen(request) as response:
    print(json.load(response)["message"]["content"])
```

For the OpenAI endpoint, replace the message above with:

```python
{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64," + image}},
    {"type": "text", "text": "What is in this picture?"},
]}
```

Send it to `/v1/chat/completions` and read
`choices[0].message.content` from the JSON response. Use a media type that
matches your image. From the terminal, `slotstream run --image cat.jpg
--prompt "What is in this picture?"` is the shorter option.

The server accepts **inline bytes only**, as bare base64 or a `data:` URL.
It rejects `http://`, `https://`, and `file://` URLs. It applies EXIF
orientation, composites transparency onto white, and rejects truncated files.

Each resized image uses one token per 32×32 pixels, up to 2,304 tokens, from
the shared context, which a request with images may fill up to 65,536 tokens. The decoded image file must be at most
24 MiB, with an aspect ratio no greater than 200:1.

The vision tower uses 0.9 GB and loads on the first image request. For auto
and `--memory-gb` plans, that reservation stays inside the original process
target, reducing expert capacity as needed. Explicit pool-size settings retain
their pool and add the resident cost to the expected footprint.
Image attention and decoded pixels also need workspace;
a request is rejected before dispatch if its budget or real headroom is insufficient.
`serve --vision off` disables images. Follow-up turns reuse image state while
the matching conversation remains cached; image identity is checked by a
digest of its bytes.

<a id="server-status"></a>

## Server status and idle stop

These endpoints are available starting in Slotstream 0.2.21. `slotstream
launch` and `slotstream stop` use them to find the server's process and to
see whether it is in use.

`GET /slotstream/status` returns:

```json
{
  "server": "slotstream",
  "version": "0.2.21",
  "pid": 4242,
  "port": 11434,
  "model": "qwen3.8-flash-next:4bit",
  "context_window": 32768,
  "started_at": 1789660800,
  "active_requests": 0,
  "clients": 1,
  "idle_seconds": 0,
  "idle_exit_minutes": 30,
  "memory_source": "--memory-gb",
  "memory_target_gb": 12
}
```

`started_at` is in Unix seconds. `active_requests` counts requests being
handled, other than status checks. `clients` counts registered processes
still running. `idle_seconds` is how long the server has had neither, and 0
while it has either. `idle_exit_minutes` is the `serve --idle-exit` setting,
or `null` when the server does not stop by itself. `memory_source` says what
sized the memory plan (`--memory-gb`, `auto`, `--pool-gb` or
`--experts-per-layer`) and `memory_target_gb` the whole-process target, or
`null` for a plan without one. Reading the status is not activity.

The development version also reports `memory_limit_gb`: the saved adaptive
ceiling, or `null` when none was selected. Adaptive plans still use
`memory_source: "auto"`; their current `memory_target_gb` can be lower than
the saved limit while other apps need memory.

`POST /slotstream/clients` with `{"pid": 4242}` registers a running process
of the user the server runs as, and returns `{"clients": 1}`, the number
registered. A server started with `--idle-exit` keeps running while any
registered process runs, and counts from the last one's exit. A pid that is
not a positive process id, or not a running process of that user, returns
400. `slotstream launch` registers the agent it starts this way.

Once a server with `--idle-exit` has decided to stop, every request other
than the status returns 503 with `the server is stopping`, and the process
exits. A request that is running when the server decides has already made it
not idle, so the stop never interrupts one. `slotstream stop` sends the
process `SIGTERM`, which stops it at once, also during a request.

<a id="errors"></a>
<a id="limits"></a>

## Errors and limits

Ollama errors use `{"error": "message"}`. OpenAI errors use
`{"error": {"message": "..."}}`; validation failures also include
`"type": "invalid_request_error"`. Anthropic errors use the shape described
under [`/v1/messages`](#v1messages).

| Status | Meaning |
|---|---|
| 400 | Invalid or unsupported request, including tools on the Ollama endpoints, JSON-schema output, strict tool schemas, logprobs, embeddings, or named reasoning levels for `think` |
| 500 | Inference failure, including an incomplete generated tool call or unsatisfied required tool choice |
| 411 | Chunked request body; send `Content-Length` instead |
| 413 | Request body exceeds 32 MiB |
| 431 | Request headers exceed 64 KiB |
| 503 | Too many open connections, insufficient memory, an expired request-to-first-token deadline, including a request that waited behind others until its prefill no longer fit the wait budget, or a server that is [stopping by itself](#server-status) |

A query string doesn't affect routing. `HEAD` returns 200 or 404 for the
requested path.

Prompt plus completion is capped by the served window. By default the server
picks it for the Mac: 32,768 tokens through 32 GB of RAM, 65,536 at 36 GB,
32,768 at 48 GB, 131,072 at 64 GB and 262,144 from 96 GB, or less on a Mac
that is busy at startup. Use `serve --max-context 65536` to fix a
65,536-token window, or any size from 1 to 262,144. Requests with images stay
within 65,536 tokens. The
planner charges extra state and transient memory before allocating the pool.
A prompt over the configured cap returns 400 with the actual limit. A known
prefill estimate can also refuse work that exceeds the remaining wait budget. `/v1/models` and `/api/show` report the actual served window;
the model's training window must not be used as the request limit.

Generation requests run one at a time; a second waits for the first. Metadata
endpoints read a separate snapshot and remain responsive during generation.

The server log identifies each accepted request and periodically reports its
elapsed time and current guarded phase, including prompt preparation and
waiting for inference. Cache diagnostics say whether memory or disk supplied
the prefix, or why retained state could not be reused. Exact token prefixes,
compatible prefill boundaries and available memory remain required; a saved
state does not guarantee a cache hit for a changed prompt.
The disk cache also retains the exact generated token IDs needed to reconstruct
history when a client omits reasoning. These IDs share the checkpoint's quota,
expiry and deletion rules; the reusable numerical state still ends at its
original prefill boundary.

Prefill progress reports completed passes by elapsed time. Its remaining-time
estimate uses recent throughput, so a slow tail replaces the faster early
rate. The initial plan estimate cannot predict other applications' memory or
SSD contention. A stalled pass still appears in the request-phase heartbeat.
Socket output failures are logged separately from request completion.

## Request deadlines and resource failures

The request policy and structured resource failures in this section are
available starting in Slotstream 0.2.14.

`--max-prefill-wait` bounds the interval from accepting a complete request to
sampling its first model token. Its default is 30 minutes; `0` disables only
time. Upload and model startup are outside this clock; queueing, prompt
preparation, image work and prefill are inside it. Decode after the first token
remains subject to memory and cancellation checks. SSE keepalives preserve
transport liveness and never reset the clock.

Admission resolves the current memory plan and exact reusable prefix while
holding the generation gate. It prices missing input from its absolute position;
an unknown estimate stays unknown. `/api/show`, `/v1/models` and the gateway
catalog expose an additive `context_policy` with configured, model and qualified
mode limits. `doctor --json` additionally computes memory feasibility, which is
independent of the wait policy.

| Code | Before streaming headers | Action |
|---|---|---|
| `context_length_exceeded` | 400 | Send less input or restart with a supported larger window. |
| `prefill_wait_exceeded` | 400 | The estimated prefill alone exceeds the wait budget. Reduce missing input, reuse a valid prefix or raise the wait budget. |
| `insufficient_memory` | 503 | Free memory, lower the target/context, or resize an image. |
| `prefill_deadline_exceeded` | 503 | The deadline passed, or the request waited behind others until its estimated prefill no longer fit. Retry when the server is free, with less work, or with a deliberate longer wait budget. |
| `inference_error` | 500 | Inspect the error and retry after correcting its cause. |

After headers, failures use the dialect's terminal error frame and close the
stream. They never emit a successful OpenAI finish or `[DONE]`, Ollama
`done: true`, or gateway success terminal. Disconnection stops bounded work.
A tool proposal from an errored, truncated or incomplete turn is not a completed
tool request; receiving clients must require successful termination before
executing it. The engine's strict consumer fixture covers its wire contract;
application-side tool authority remains the receiver's responsibility.

Memory checks precede the next bounded allocation and leave safety headroom.
They cannot prevent another application from allocating between checks. The
engine joins readers and synchronizes GPU users before releasing request pins,
and failed state is not reused. A subsequent request either succeeds or receives
an explicit still-unavailable error.


<!-- ===== docs/LIBRARY.md ===== -->

# Using slotstream from Swift

Use the `Slotstream` Swift package to plan memory, download weights, run the
model, or start its HTTP server from your own Mac app or tool. It is the same
native engine the command-line tool uses, Swift on MLX and Metal end to end,
so your app runs the model in process with no local server or Python runtime
in between; see the [native stack](ENGINEERING.md#native-stack).

Add the package and product to `Package.swift`:

```swift
// Package.swift
dependencies: [
    .package(url: "https://github.com/carloslfu/slotstream.git", .upToNextMinor(from: "0.2.3")),
],
targets: [
    .executableTarget(name: "YourApp", dependencies: [
        .product(name: "Slotstream", package: "slotstream"),
    ]),
]
```

The example stays within the 0.2 release series because the library API is
still evolving. Library products are available from 0.2.3 onward.

Two products:

| Product | For |
|---|---|
| `Slotstream` | Running the model: weights, planning, generation, serving. |
| `SlotstreamDiagnostics` | Checks, goldens and benches. What the CLI and CI import; skip it in an app. |

## The Metal library

For a command-line build, place a prebuilt `mlx.metallib` **next to the
running executable**. MLX looks there for its Metal shaders. SwiftPM doesn't
compile them in the Command Line Tools setup.

- **Sevra's Xcode project**: its build phase bundles the pinned library.
  Other app targets must also include the matching MLX shaders at runtime.
- **A command-line build** (`swift build`): copy the matching library beside
  each executable after building.

  ```bash
  Tools/fetch_metallib.sh                       # from a slotstream checkout
  cp Tools/lib/mlx-0.32.2.metallib .build/debug/mlx.metallib
  ```
  Without it, the first MLX call fails with `Failed to load the default
  metallib`. A test bundle needs its own copy in `.xctest/Contents/MacOS/`.

The runtime pins mlx-swift to an exact revision and packages the matching MLX
shader library. Upgrade those together. The engine's qualified fused prefill
path uses upstream MLX attention; the experimental selected-block kernel
remains a separate control. `SLOTSTREAM_OPT_FUSED_PREFILL=0` restores MLX's
own dispatch heuristics, and `=1` requests the fused path on supported NAX
hardware. Automatic activation is limited to the measured M5 Pro profile;
other profiles retain MLX dispatch. Short decode and speculative verify calls
retain their existing paths. Persistent prompt-cache identities include the
attention settings, MLX backend overrides, GPU architecture and OS build, so changing arithmetic
starts a fresh cache.

The qualified profile also chooses expert-read groups automatically using the
fused kernel's workspace requirements. No application setting is needed. For
supported BF16 text prefill, including MTP, groups may share reads across more
chronological compute passes while staying within the existing memory target.
The larger groups end within the supported key range. Each candidate still
has to fit the actual process footprint, live headroom and request reservation;
the scheduler chooses a smaller group or an ordinary pass when necessary.
MTP prices the main and draft phases separately, including the hidden states
retained between them, and admits against the larger peak. The draft phase
keeps its full attention allowance regardless of whether its own kernel fuses.
When MTP's memory budget limits grouping, the engine can assemble expert
buffers one at a time. It uses that extra synchronization only when it can
at least double the group that fits with batched writes. This is an internal
performance policy, with the same memory and cancellation guards.
Images, CPU, other dtypes, small-query padding and attention fallbacks keep
full main-attention accounting and their established automatic grouping limits.

Diagnostic overrides remain available: `SLOTSTREAM_OPT_FUSED_WORKSPACE=0`
restores both the original accounting and automatic group cap;
`SLOTSTREAM_OPT_FUSED_PREFILL=0` also disables the combined policy.
`SLOTSTREAM_OPT_AUTO_SCOPE_LIMIT=8192|16384` can test grouping independently.
These controls do not increase the memory target or select a larger compute
pass. `InferenceOptimizations.environment()` selects the deployment policy;
the explicit reference initializer and older serialized settings retain their
original reservation. The [initial automatic-policy validation](../db/records/measurements/automatic-prefill-policy-2026-09-21.md)
and [MTP qualification](../db/records/measurements/mtp-prefill-policy-2026-09-21.md)
record the tested scope, with earlier experiments and exclusions preserved.

Planning, weight checks, prefill estimates, and most diagnostics run without
loading Metal. An app can show a memory plan and download status before the
user downloads the model.

<a id="is-a-model-here-and-what-would-it-cost"></a>

## Check and download weights

```swift
import Slotstream

let store = WeightStore.default            // ~/.slotstream/models, honours $HOME
switch store.status() {
case .ready:
    break
case let .missing(need, free), let .incomplete(need, free):
    print("need \(need / 1_000_000_000) GB, \(free / 1_000_000_000) GB free")
case let .corrupt(paths, _, _):
    print("damaged: \(paths.joined(separator: ", "))")
}
```

`status()` hashes the files before returning `.ready`, including files whose
sizes already match. Allow several seconds for a complete copy.

To download missing weights with resume, hash verification, and progress:

```swift
let cancellation = PullCancellation()
try store.download(PullOptions(cancellation: cancellation)) { line in print(line) }
// From another queue, call cancellation.cancel() to preserve resume progress.
```

The default uses the compressed CDN and tunes connection count. Explicit
`connections` fixes that count. `transport: .raw` selects raw mirrors;
setting `sources` also selects raw mirrors in automatic mode. Existing raw
partial downloads continue automatically. Download returns only after final
original-file verification; an immediate second `verify()` is unnecessary.
`status()` reports remaining reconstructed model bytes, not compressed wire
bytes, and permits an absent optional draft head.

`download` fetches the weights only. The 37.5 MB decode-forecast file that
`slotstream pull` also fetches is optional and makes decode faster; fetch it
the same way after the weights. A failure is logged and returns `false`
instead of throwing. A cancelled fetch returns `false` too, so check
`cancellation.isCancelled` afterwards to tell a stop from a failure:

```swift
for file in TapCorrectionSidecar.files {
    TapCorrectionSidecar.ensure(modelDir: store.modelDirectory, file: file,
                                cancellation: cancellation) { line in print(line) }
}
```

<a id="what-will-it-do-on-this-mac"></a>

## Plan memory

```swift
let machine = Machine.current()
let plan = try Planner.plan(PlanRequest(memoryGB: 16), on: machine)
print(plan.banner())
print(plan.expertsPerLayerCached, "experts per layer,",
      plan.expectedPeakGB, "GB planned full-workload envelope")
```

`Machine.simulated(ramGB: 16)` previews a decimal-GB memory size, like
`slotstream doctor --sim-ram 16`; it does not simulate another chip or SSD.
`Engine` rejects simulated plans;
use `Machine.current()` for a plan that will allocate memory.

In the development version, `PlanRequest(memoryLimitGB: chosenLimitGB)` sets
an adaptive process ceiling. The supported hardware budget and available
memory may lower `plan.targetGB`; `plan.memoryLimitGB` keeps the saved ceiling.
Existing `memoryGB`, `poolGB` and `expertsPerLayer` controls keep a fixed cache
and cannot be combined with this option.

An embedding app must retain a `MemoryGovernor(engine:)` and call `start()`
to enable live resizing. Call `await governor.stopAndWait()` before releasing
the engine. Construct plans through `Planner.plan`; directly constructed
adaptive plans must use `.auto` and a positive target within the saved limit.
The engine validates these conditions before model allocation. The original
planner and initializer signatures remain available for existing Swift code.

<a id="pricing-a-prompt-before-you-send-it"></a>

## Estimate prompt-processing time

```swift
let seconds = PrefillSchedule.estSeconds(tokens: 8_000, maxChunk: plan.prefillChunk)
print("about", PrefillSchedule.describe(seconds: seconds), "to the first token")
```

This estimate needs no model loaded. An app can show the expected wait
before starting a long prompt.

## Serving

The library exposes the same `Server` used by the CLI. It listens on loopback
and provides the [Ollama/OpenAI endpoints](API.md) and [AI SDK gateway](FX.md).

## A thinking phase followed by an answer

`Engine.generatePhased` runs two phases under one generation gate, with
independent sampling parameters and output budgets. Its `transition` callback
returns nonempty separator or closure token ids. The engine transfers the
live state and consumes any pending sampled token exactly once, then reads
only the suffix needed for the second phase. Both phases obey the caller's
context and memory limits; cancellation also reaches the second phase.

Callbacks must not re-enter generation or configuration on the same engine.
The first phase throws on failure; inspect the returned second phase's
`stats.requestFailure` and `stats.runtimeError` as with ordinary generation.
Neither phase writes its private working state to the disk tier. Generated
rows remain ineligible for cold-equivalent reuse by a later conversation.

## Persistent prefix cache

`Engine.enablePersistentPrefixCache(_:)` adds a disk tier under
`prefixCache`. A restarted process, or a conversation longer than in-memory
retention allows, restores the longest persisted state its prompt extends
instead of processing that prompt again, and continues exactly as the saved
state would have.

```swift
let tier = try engine.enablePersistentPrefixCache(
    PersistentPrefixConfiguration(directory: URL(fileURLWithPath: "/path/to/prefix-states")))
tier.onEvent = { print("prefix cache disk:", $0) }
```

With aligned resume enabled, the generator saves states at completed prefill
pass boundaries, for text prefixes of at least `minimumTokens`. A restored
state must match the incoming prompt's boundaries and the producing prefill
pass size as well as the binary, model and optimization settings. Legacy
heads with unknown pass sizes are not eligible for aligned state reuse.
Generated conversation ids can still help render a later prompt, but do not
certify a numerical checkpoint. Unchanged persisted rows can be referenced
by later checkpoints instead of being written again. `maxBytes`
bounds the directory: when it is full, states nobody continued go first, then
kept previous turns, then conversations, then prefixes several conversations
start from, least recently used first. `maxAge`
(30 days by default, `nil` to keep states until the quota needs room) removes
unused states. Opening the directory removes files from other binaries, models
or settings, expired and damaged files, and anything over the quota;
`tier.maintenance` reports what it removed. A file the system refuses to read,
because of its permissions or an I/O error, is kept: opening throws
`PersistentPrefixCache.InaccessibleFile`, which names it, and `slotstream
serve` then runs without the disk tier.

The prefix conversations share is kept too, with or without the disk tier.
While a prompt is processed, the head other conversations will start with is
stored as a shared prefix: the system message when it ends
`Generator.sharedPrefixMinimumTokens` (512) tokens or more in, and the longest
head it shares with a prompt already kept in memory or on disk, up to where it
parts from that prompt. A conversation's next turn, which sends back the reply
the model generated, parts from nothing another conversation starts with. The
save
point is the last prefill pass end at or before that boundary, so no pass is
reshaped and outputs are unchanged; the state is forked into `prefixCache` and,
at `minimumTokens` or more, written to disk. The next conversation starting
with the same system prompt reuses it instead of processing it again. A shared
prefix is kept once and is never replaced by the conversations that extend it;
when the quota is full, one that two or more conversations start from is
removed after them. An app that knows where its stable preamble ends, for
example one without a system message, names it on the request; `0` leaves only
the shared-head rule:

```swift
let request = try engine.beginRequest()
request.sharedPrefixTokens = preambleTokenCount
```

`GenStats.sharedPrefixBoundaries` lists the save points a request wrote,
`sharedPrefixHint` and `sharedPrefixCommon` the two boundaries it considered,
and `persistentPrefix.sharedSaveOutcome` what the disk tier did with each.

The directory is created owner-only and holds conversation token ids and model
state. To keep a conversation off disk, set `persistsPrefixState = false` on the
controller of each of its requests and pass it to `generate`:

```swift
let request = try engine.beginRequest()
request.persistsPrefixState = false
```

`tier.removeStates(overlapping: ids)` removes the states of a conversation being
deleted, given its latest prompt ids, and `tier.clear()` removes everything.
Each request's `GenStats.persistentPrefix` reports what it restored, wrote and
reused, and `prefixCache.json()["persistent"]` holds the tier's totals.

`Engine.disablePersistentPrefixCache()` detaches the tier without deleting
saved files. Detach before encoding a private conversation, because chat
encoding can consult saved conversation ids. Clear the in-memory prefix
cache as well when changing ownership or privacy scope.

The native Mac app enables the tier for ordinary, non-thinking conversations
inside that Home's disposable `.sevra/prefix-cache` directory. Home backups
exclude this directory. Incognito and conversations with a recorded thinking
phase do not use the disk tier; Home and privacy transitions clear held
in-memory state before encoding. An unavailable disk cache falls back to
ordinary inference.

## Diagnostics

`SlotstreamDiagnostics` returns structured `CheckReport` values that an app
can inspect or display:

```swift
import SlotstreamDiagnostics

let report = Diagnostics.prefillSchedule()
print(report.name, report.passed, report.items.count)
```

`Diagnostics.runtime()`, `.governorPolicy()`, `.pullIntegrity()`,
`.machinePlanning()`, `.httpFraming()`, `.httpRouting()` and
`Goldens.sampler(...)` all run without weights. See [TESTING.md](TESTING.md).

<a id="what-is-not-here-yet"></a>

## Optimization controls

`InferenceOptimizations()` retains the explicit reference configuration.
`try InferenceOptimizations.environment()` resolves the deployment defaults
and validated environment overrides. These are deliberately different entry
points: an existing caller that constructs reference controls keeps those
semantics. Ordinary engine construction resolves the environment defaults.

Automatic prompt-read grouping requires the request memory controller and a
compatible chronological schedule. `Engine.generate` creates a controller
when the caller does not supply one. The lower-level `Generator.generate`
overload without a controller keeps ordinary chronological processing.
Begin and pass the controller before preparation, as described below, to
include that work in the same request deadline and memory reservation.
Explicit controls, saved control sets and public initializers retain their
compatibility behavior; no numerical or capacity guarantee follows from
enabling an experimental control.

## API stability

`Engine.generate` currently uses callbacks. A typed delta stream, dedicated
executor, and `Conversation` API are planned but not available. For now,
`slotstream run` shows how the CLI calls the engine; the HTTP API is also
available for callers in another process.

## Context and request control

These additive APIs are available starting in Slotstream 0.2.14.

Construct an engine with the plan that prices its context. Shared
`ContextConfiguration` validates `maxContextTokens` and
`maxPrefillWaitMinutes` before loading. Attach it using
`MemoryPlan.withRequestPolicy`, then use `Engine(modelDir:plan:)`.
Call `beginRequest` before tokenization or images, pass that controller to
`encodeChatWithVision`/`encodeWithVision` and `generate`, and use its
`connected` callback for cancellation. The request clock includes the generation
queue; it stops counting the deadline at the first sampled token.

Legacy call signatures remain available. `GenStats.requestFailure` adds a
structured code, elapsed/limit/estimate or memory details when applicable;
`runtimeError` and `finishReason` continue to expose failure to older callers.
Check these before using generated tools. An invalid legacy assignment to
`maxContextTokens` keeps the advertised window unchanged and refuses subsequent
work until corrected; a larger allocation requires a newly planned engine.

`Planner.contextFeasibility` searches actual discrete windows under frozen
machine inputs and runtime allocation controls. It returns the requested plan,
largest fitting plan and refusal, independently of the time policy. Diagnostic
qualification is explicit; it never changes the public implementation or
MTP/vision limits advertised by ordinary serving.

Starting in Slotstream 0.2.17, `Planner.resolveContextWindow(.automatic,
request:on:mtpAvailable:visionAvailable:)` returns the plan the command-line
server uses: the automatic window for the machine's memory tier, lowered at
startup when available memory is short, with an `AutomaticContextWindow` that
lists every candidate and why it was or wasn't taken. `.tokens(n)` plans an
explicit window, and `Planner.automaticContextWindow` evaluates the tier
without planning against live memory. `Planner.plan(_:on:)` still plans the
request's own `maxContextTokens`, 32,768 unless set. A window above 32,768 now
retains one complete conversation when its plan can hold one; otherwise the
plan's notes say how much a follow-up reuses. `ContextPolicy.maxTokens` is the
model's 262,144 tokens, and requests with images stay within
`ContextPolicy.visionLimit`.

Automatic selection preserves cache above the measured decode range when a
larger window would remove slots. Its clamped speed estimate cannot establish
that those slots have no value. Such candidates omit `relative_request_cost`
and set `request_cost_calibrated` to false. The plan's legacy
`expected_peak_gb` is the planned full-workload envelope, not measured usage;
`memory_target_semantics`, `expected_peak_semantics` and
`non_cache_allowance_bytes` make the distinction explicit in JSON.
`planned_headroom_gb` is the remaining planned budget after that envelope,
not live free RAM. Near the minimum cache size it can be smaller than the
nominal planning margin; a fully resident model can leave more unassigned.


<!-- ===== docs/FX.md ===== -->

# Use fx with slotstream

[fx](https://fx.sh) is Vercel Labs' coding agent. You can point its gateway
client at slotstream to use a local model for file reads, edits, and tool
calls. The setup below uses a separate fx profile and asks you to approve
actions.

Automatic action reviews time out in this setup, and long-session compaction
is unreliable. See [Limitations](#limitations) before starting a long task.

## Setup

Start slotstream in one terminal and leave it running:

```sh
slotstream serve --port 11434
```

Save the following as `fxs` in your working directory:

```sh
#!/bin/sh
# Run fx with a separate profile for the local model.
FX_PROFILE_DIR="$HOME/.fx-slotstream"
mkdir -p "$FX_PROFILE_DIR/.fx"
[ -f "$FX_PROFILE_DIR/.fx/settings.json" ] || cat > "$FX_PROFILE_DIR/.fx/settings.json" <<'JSON'
{
  "provider": "gateway",
  "models": { "gateway": "slotstream/qwen3.8-flash-next:4bit" },
  "permission_mode": "ask",
  "auto_upgrade": false
}
JSON
HOME="$FX_PROFILE_DIR" \
FX_PERMISSION_MODE=ask \
FX_GATEWAY_BASE_URL=http://127.0.0.1:11434 \
FX_GATEWAY_CHAT_URL=http://127.0.0.1:11434/v3/ai/language-model \
AI_GATEWAY_API_KEY=local-dummy-key \
FX_DISABLE_KEYCHAIN=1 \
exec fx "$@"
```

In a second terminal, make it executable and check the model list:

```sh
chmod +x fxs
./fxs models
```

The list should include `slotstream/qwen3.8-flash-next:4bit`. Then run `./fxs`
where you would normally run `fx`. You need fx installed separately.

The wrapper keeps settings, sessions, and usage records under
`~/.fx-slotstream/.fx`. It sets `FX_PERMISSION_MODE=ask` on every launch
because fx can rewrite its settings file. The API key is a placeholder;
slotstream doesn't authenticate it.

**Keep both gateway URLs on loopback HTTP with an explicit port.** fx accepts
`127.0.0.1`, `localhost`, or `[::1]`, with no userinfo in the URL. An invalid
override is silently ignored by the tested fx client, which then connects to
Vercel. Check `./fxs models` after changing these settings.

<a id="what-works-and-what-does-not"></a>

## Supported features

The main agent loop supports multiple tool calls per turn, tool results,
file reads and writes, images, and follow-up turns. Image parts must contain
inline bytes; see [Images](API.md#images).

Reasoning is available through fx's effort setting and streams separately
from the answer. slotstream retains compatible conversation state, including
reasoning state, so follow-up turns process only the new material while that
state is cached.

The first turn processes fx's system prompt and tool schemas as well as your
question. That can take minutes on a small memory target. The main agent
step has no client-side deadline in the tested client.

## Limitations

- **Automatic permission reviews:** `permission_mode: auto` sends a separate
  review request with a different tool set. It misses the conversation cache
  and can exceed fx's 30-second review deadline, leaving the action on hold.
  Use the wrapper's `ask` setting to approve actions yourself.
- **Session compaction:** fx gives each summary chunk a 120-second budget.
  Generation can exceed it, even with the smaller window advertised for the
  compactor alias. Start a new session if compaction fails.
- **Structured output:** JSON-schema constrained output returns a typed
  400 error.
- **Provider tools:** tools executed by a hosted provider, such as fx's web
  search, aren't available locally. They are removed from the request before
  it reaches the model. Ordinary client-executed tools remain available.

These observations describe the tested fx integration. A client update can
change its accepted overrides, request fields, or deadlines.

## Troubleshooting

### The model list doesn't show slotstream

Check that the server is running and both URLs use loopback HTTP with an
explicit port. Use the model ID returned by `./fxs models` or
`GET /coding-agent/v1/models`.

### fx reports an invalid finish reason

Upgrade slotstream. An older server may not implement this gateway protocol;
a proxy that changes the response stream can also cause this error.

### The first turn takes minutes

Check the progress in the server terminal. A cold first turn processes fx's
full standing prompt. `slotstream doctor` estimates prompt-processing time
for your memory plan.

### Requests fail with `unsupported_field`

A client may be sending a field this server doesn't support. Open an issue
with the field name and both software versions.

<a id="what-slotstream-serves"></a>

## Protocol reference

slotstream implements the Vercel AI SDK Language Model Specification v4 over
AI Gateway protocol 0.0.1.

| Route | Purpose |
|---|---|
| `POST /v3/ai/language-model` | Model calls, streamed as Server-Sent Events |
| `POST /v1/ai/language-model` | The same handler under the `v1` path |
| `GET /coding-agent/v1/models` | Model IDs and capabilities |
| `GET /coding-agent/v1/credits` | A zero balance for `fx credits` |

The model emits tool calls in its native XML format. slotstream parses them
incrementally, converts arguments using the tool's JSON Schema, and returns
AI SDK `tool-call` parts.

<a id="how-it-is-tested"></a>

## Tests

`slotstream-checks --tier t0` tests validation, prompt mapping, the model
catalogue, stream frames, and tool-call parsing without weights or a GPU.

`Tools/fx_gates.sh` tests a live server: model discovery, rejected requests,
streaming, a complete tool loop, and conversation reuse. When fx is installed,
it also tests the client completing a task with the local model.


<!-- ===== docs/HERMES.md ===== -->

<a id="hermes-with-slotstream"></a>

# Use Hermes with Slotstream

Use Hermes with a model running on your Mac. Hermes handles the conversation
and tools, such as reading files; Slotstream runs the model.

## Before you start

- [Install Slotstream](GETTING-STARTED.md). Check `slotstream --version`:
  you need 0.2.21 or later for `slotstream launch`, used below, and 0.2.8 or
  later to [start Hermes yourself](#start-hermes-yourself). Rerun the
  installer to update.
- [Install Hermes](https://github.com/NousResearch/hermes-agent#quick-install).
  Return here once the `hermes` command is available.

Both programs run on the same Mac.

## Start Hermes

For your first session, create a practice folder and a small file in
Terminal:

```sh
mkdir -p ~/slotstream-demo
cd ~/slotstream-demo
printf 'The garden gate code is MAPLE.\n' > note.txt
```

This creates or replaces `note.txt` in the practice folder. Then start Hermes:

```sh
slotstream launch hermes
```

If Slotstream is not running, this starts it in the background first and
shows its start until the model answers. If the model hasn't been
downloaded, it asks to download it first. The server gets a context window of
at least 65,536 tokens, which gives Hermes room for its instructions, tools,
and conversation history, and uses more memory than ordinary chat. It keeps
running while Hermes runs and stops 30 minutes after the last session ends;
`slotstream stop` stops it sooner. If startup fails, see
[Troubleshooting](#troubleshooting) below.

The first time, this creates a separate Hermes folder,
`~/.hermes-slotstream`, with the [configuration below](#configure-hermes),
so your usual Hermes settings stay as they are. It then starts
`hermes chat --provider slotstream --model qwen3.8-flash-next:4bit` with that
folder selected. Later launches use the folder as it is, including any
changes you make to it.

No OpenAI account or key is needed. The first reply can take several
minutes; `tail -f ~/.slotstream/logs/serve.log` in another window shows its
progress.

Anything after `hermes` is passed to Hermes, for example
`slotstream launch hermes chat --yolo`. If the server runs on another port,
pass it before the tool name: `slotstream launch --port 8080 hermes`. To
use a different folder, set `HERMES_HOME` before the command.

<a id="verify-the-loop"></a>

## Try it

Ask Hermes:

> Read note.txt using your terminal tool and tell me what it says.

You should see a tool call to read the file, followed by its contents:
`The garden gate code is MAPLE.` If Hermes asks permission to read the file,
approve that action.

Then ask:

> What was the gate code? Answer from our conversation without reading the file again.

It should answer `MAPLE`.

<a id="configure-hermes"></a>

## The configuration

`slotstream launch hermes` writes this to `~/.hermes-slotstream/config.yaml`,
with the address and window of your server:

```yaml
model:
  provider: slotstream
  default: "qwen3.8-flash-next:4bit"
  context_length: 65536
providers:
  slotstream:
    base_url: http://127.0.0.1:11434/v1
    api_key: unused
    api_mode: chat_completions
    extra_body:
      max_tokens: 4096
agent:
  reasoning_effort: none
  local_stream_stale_timeout: 1800
auxiliary:
  vision:
    provider: main
    timeout: 1800
  compression:
    provider: main
    timeout: 1800
    extra_body:
      max_tokens: 4096
      temperature: 0.2
      presence_penalty: 0
  title_generation:
    provider: main
    timeout: 1800
    extra_body:
      max_tokens: 64
  approval: {provider: main}
  skills_hub: {provider: main}
  review: {provider: main}
  mcp: {provider: main}
  memory_query_rewrite: {provider: main}
  tts_audio_tags: {provider: main}
  triage_specifier: {provider: main}
  kanban_decomposer: {provider: main}
  profile_describer: {provider: main}
  goal_judge: {provider: main}
  curator: {provider: main}
  monitor: {provider: main}
  background_review: {provider: main}
  moa_reference: {provider: main}
  moa_aggregator: {provider: main}
compression:
  enabled: true
```

To change it, open it in a text editor, for example with
`nano ~/.hermes-slotstream/config.yaml`; press **Control+O**, then **Enter**
to save and **Control+X** to close.

The configuration sends replies to Slotstream, and every side task Hermes
runs besides them: conversation summaries, titles, the check that decides
whether a command needs your approval, and the rest of the `auxiliary` list.
A side task left out of that list uses Hermes's automatic choice, which tries
cloud providers such as OpenRouter when the server is busy or stopped, so
each one is set to `provider: main`. Hermes tools that use web services still
need their own connections.
Keep the provider name `slotstream` in both the configuration and launch command.
It keeps this connection separate from other providers named `custom`.
Leave `api_key: unused` as written. It is a placeholder.

The three `max_tokens` settings limit replies, summaries, and titles separately.
They are output ceilings, not required response lengths. You can adjust them
for your tasks. Keep replies and summaries within the server's output limit;
see [output budgets](HERMES-NOTES.md#output-budgets).

Reasoning is optional. The `reasoning_effort` setting above uses `none` to skip
the model's extra thinking before answering. Change it to `medium` to enable
thinking, or remove the setting to use Hermes's default. Tools work with either
setting.

<a id="run-the-server-yourself"></a>

## Run the server yourself

To watch the server, or to start Hermes without `slotstream launch`, start
the server in its own Terminal window first:

```sh
slotstream serve --max-context 65536
```

Wait until you see `slotstream listening on http://127.0.0.1:11434`, and
leave the window open; `slotstream launch hermes` then uses this server as it
is. Slotstream chooses its memory plan and whether to use speculative
decoding automatically. Without the flag the window depends on the Mac, for
example 32,768 at 48 GB; the flag keeps the window Hermes is configured for on
every Mac. Press
**Control+C** in the window, or run `slotstream stop`, to stop the server.

<a id="start-hermes-yourself"></a>

## Start Hermes yourself

Without `slotstream launch`, [run the server yourself](#run-the-server-yourself),
then create the folder and the configuration once:

```sh
mkdir -p ~/.hermes-slotstream
nano ~/.hermes-slotstream/config.yaml
```

Paste the configuration above, keeping the indentation, and save. If you've
followed this guide before, replace its previous configuration block with this
one; `slotstream launch hermes` names any side task an older block leaves
out. Preserve any unrelated settings you added; do not leave duplicate sections
or the old `model.base_url` and `model.max_tokens` entries. Then start Hermes
with the folder selected:

```sh
HERMES_HOME="$HOME/.hermes-slotstream" \
  hermes chat --provider slotstream --model qwen3.8-flash-next:4bit
```

## Use it again

Next time, open your working folder and run `slotstream launch hermes`. If
you start Hermes yourself, run the server first as above, then the Hermes
command including `HERMES_HOME`.

When finished, exit Hermes with `/exit`. A server `slotstream launch` started
stops 30 minutes later, or at once with `slotstream stop`. Stop a server you
started yourself with **Control+C** in its window.

Slotstream 0.2.14 defaults to 30 minutes from accepting a request to the first
sampled token, including queueing and preparation. If you change the server's
`--max-prefill-wait`, keep Hermes's client timeouts compatible with it. See
[server request deadlines](HERMES-NOTES.md#server-request-deadlines).

## Troubleshooting

| Problem | What to do |
|---|---|
| Command not found | Open a new Terminal window. If it still fails, check the program's installation guide. |
| `slotstream launch: no Slotstream server answered` | You ran it with `--no-start`. [Run the server yourself](#run-the-server-yourself), or run the launch without `--no-start`. |
| `another Slotstream model process is running` | A server on another port, `slotstream run` or a check holds the model, and only one fits in memory. Stop it, or pass that server's port with `--port`. |
| `Slotstream stopped while starting` | The lines above it, and `~/.slotstream/logs/serve.log`, say why. If memory is short, close memory-heavy apps or pass a smaller `--memory-gb`. |
| `Hermes needs a context window of at least 65536 tokens` | The running server has a smaller window. If you started it, stop it and run this guide's command. If another agent still uses the server `slotstream launch` started, run the launch again once that agent exits; it then restarts the server with the larger window. |
| `config.yaml has no slotstream provider` | The folder has a configuration from something else. Add the configuration above, or set `HERMES_HOME` to a new folder. |
| `does not point at port` | The configuration names another address. Correct `base_url`, or pass the matching `--port` to `slotstream launch`. |
| `sets context_length to` | The configuration was written for a server with another window. Set `context_length` to the window the message names, or restart the server with the `--max-context` it names. |
| `` does not set `provider: main` for these Hermes side tasks `` | The configuration comes from an older version of this guide. Replace its `auxiliary` block with the one above. |
| Hermes cannot connect | The server stopped, with `slotstream stop` or 30 minutes after the last session; run `slotstream launch hermes` again. If you configured Hermes yourself, keep the server running and copy the address and model name exactly as shown. |
| A local connection fails while using an HTTP proxy | Add `localhost` and `127.0.0.1` to the proxy exclusions in `NO_PROXY` and `no_proxy`, preserving any existing exclusions. Check this profile's `.env` as well as your shell settings. |
| An error mentions OpenRouter or asks for a cloud API key | Hermes selected a different connection. Replace the old guide configuration with the one above and start with `slotstream launch hermes`. |
| A log still labels the provider `custom` | Hermes also uses that internal label for named providers. Check the endpoint address to confirm which connection it selected. |
| Hermes cannot find provider `slotstream` | Check that the file is named `config.yaml`, the indentation matches, and `HERMES_HOME` selects this folder. See the diagnostic command below. |
| Another server is running | Stop your existing Slotstream server with `slotstream stop` or Control+C in its window, then run `slotstream launch hermes` again. For another app, see [port conflicts](TROUBLESHOOTING.md#the-server-cant-listen-on-port-11434). |
| The first reply is slow | `tail -f ~/.slotstream/logs/serve.log` shows its progress, or the Slotstream window if you started the server yourself. Long prompts and summaries can take several minutes. |
| Hermes says “reasoning…” with thinking off | Hermes uses that word in its loading animation even when model thinking is off. |
| Insufficient memory | Close memory-heavy apps. Run `slotstream doctor --max-context 65536` to check the plan without loading the model. See [memory help](TROUBLESHOOTING.md#the-whole-mac-is-slow). |
| Replies or summaries stop early | Check all three `extra_body.max_tokens` settings. A `max_tokens` entry directly under `model` does not set Hermes's chat output limit in the tested versions. |

To diagnose a connection, exit Hermes and run:

```sh
HERMES_HOME="$HOME/.hermes-slotstream" \
  hermes --profile default chat --cli --verbose \
  --provider slotstream --model qwen3.8-flash-next:4bit
```

`--profile default` selects the configuration in this folder even if you
previously selected another Hermes profile. Send a short greeting. Any reported
inference endpoint should be `localhost:11434` or `127.0.0.1:11434`. Keep the
error and the server's output, from its window or
`~/.slotstream/logs/serve.log`, when reporting a problem.

For other connection errors, see [connection troubleshooting](CLIENTS.md#troubleshooting).
Developers can find configuration explanations, image support, and integration
tests in the [Hermes engineering notes](HERMES-NOTES.md).


<!-- ===== docs/CODEX.md ===== -->

<a id="use-codex-with-slotstream"></a>

# Use Codex with Slotstream

Run OpenAI's Codex CLI on the model Slotstream serves. Codex edits files and
runs commands through its own tools; Slotstream provides the model over the
OpenAI Responses API on your Mac. No OpenAI account or API key is needed, and
the model runs offline; only the first start downloads a file, described
below.

## Before you start

- Slotstream 0.2.21 or later, installed with its model, from the
  [getting started guide](GETTING-STARTED.md). `slotstream --version` shows
  yours; the install command updates it.
- Codex CLI installed: `npm install -g @openai/codex`. The
  [Codex repository](https://github.com/openai/codex) lists other ways.

These steps were verified with Codex 0.148.0. Codex changes quickly; if a
newer version behaves differently, start with the troubleshooting table at
the end.

## Start Codex

In Terminal, go to your project directory and run:

```sh
slotstream launch codex
```

If Slotstream is not running, this starts it in the background first and shows
its start until the model answers. Codex then starts with
`provider: slotstream` and the model name in its header. The first reply takes
a while: Codex's instructions and tool descriptions are read once before the
first token, and later turns read only what is new. A new session reuses the
instructions an earlier one read.

The server keeps running while Codex runs, and stops 30 minutes after the last
session ends. `slotstream stop` stops it sooner, and
`tail -f ~/.slotstream/logs/serve.log` shows what it is doing. To run the
server yourself, in its own Terminal window, start `slotstream serve` before
`slotstream launch codex`; launch then uses it as it is.

`slotstream launch codex` does three things before starting Codex:

- It asks the server for its model and context window.
- It describes the model to Codex in a catalog file under
  `~/.slotstream/launch/codex/`. Codex needs this for a model it does not
  know: without it Codex assumes a far larger window than the server has,
  and the model's first file edit fails because the `apply_patch` tool is
  not declared. The catalog carries the base instructions your installed
  Codex version ships. The first start downloads them once from the Codex
  repository on GitHub; later starts use the saved copy.
- It passes the connection to Codex as `-c` settings.

Your `~/.codex/config.toml` is not changed, and plain `codex` keeps using
your usual provider.

Anything after `codex` goes to Codex:

```sh
slotstream launch codex exec "Summarize this repository"
slotstream launch codex -i picture.png
```

If the server runs on another port, pass it before the tool name:
`slotstream launch --port 8080 codex`. To see the command, settings and
files without starting anything, add `--dry-run`:
`slotstream launch --dry-run codex`.

## Try it

Ask for something that needs a file edit and a command:

```text
Create a file named hello.txt whose entire content is the single line: SLOTSTREAM OK. Then read the file back and tell me exactly what it contains.
```

Codex applies the edit through its `apply_patch` tool, runs a shell command
through `exec_command`, and reports the contents. Check that `hello.txt`
exists and holds the line.

Pictures work in both directions. Attach one with
`slotstream launch codex -i picture.png` and ask what it shows, or ask Codex
to look at an image file in the project and it uses its `view_image` tool;
either way the model reads the picture on your Mac.

Thinking is off by default, which is faster and is the mode the model's tool
calls were measured in. To turn it on for a session, run
`slotstream launch codex -c model_reasoning_effort="low"` (or `medium` or
`high`).

## Configure Codex yourself

`slotstream launch codex` is the simplest way to start. To start Codex
directly instead, for example from an editor integration, set the
connection up once by hand.

With the server running (`slotstream serve`, or one `slotstream launch` started), download the catalog script and run it:

```sh
curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/Tools/codex_catalog.py -o codex_catalog.py
python3 codex_catalog.py
```

In a Slotstream source checkout, `python3 Tools/codex_catalog.py` does the
same. It reads the served context window from the running server, downloads
the base instructions that ship with your installed Codex version, and
writes `~/.codex/slotstream-models.json`. Run it again after updating Codex
or changing `--max-context`.

Then create `~/.codex/slotstream.config.toml` with:

```toml
model = "qwen3.8-flash-next:4bit"
model_provider = "slotstream"
model_catalog_json = "/Users/YOUR-USER/.codex/slotstream-models.json"

[model_providers.slotstream]
name = "Slotstream"
base_url = "http://127.0.0.1:11434/v1"
wire_api = "responses"
stream_idle_timeout_ms = 1800000
```

Replace `YOUR-USER` with your macOS user name; the path must be absolute.
Keep `model_catalog_json` above the `[model_providers.slotstream]` header;
in TOML a key below a header belongs to that table. This file is a Codex
profile: start Codex with it using

```sh
codex --profile slotstream
```

To make Slotstream your default instead, put the same lines in
`~/.codex/config.toml`.

`stream_idle_timeout_ms` gives a long first prompt time to read: Codex's idle
limit counts events, and a cold prompt on this model is read for minutes
before the first token. Slotstream sends a progress event every ten seconds
during that read, so the default limit usually holds, but a large first
prompt on a small memory target can exceed it. `slotstream launch codex`
sets the same limit.

Codex cannot use Slotstream through `codex --oss`. That mode checks the
Ollama server version and refuses anything older than the release that
added Ollama's own Responses API, and Slotstream reports its own version
number there.

## Troubleshooting

| Problem | What to check |
|---|---|
| `slotstream launch: no Slotstream server answered on port 11434` | You ran it with `--no-start`. Start `slotstream serve` in another Terminal window, or run `slotstream launch codex` without `--no-start`. If the server uses another port, pass it with `--port`. |
| `another Slotstream model process is running` | A server on another port, `slotstream run` or a check holds the model, and only one fits in memory. Stop it, or pass that server's port with `--port`. |
| `Slotstream stopped while starting` | The lines above it, and `~/.slotstream/logs/serve.log`, say why. If memory is short, close memory-heavy apps or pass a smaller `--memory-gb`. |
| ``slotstream launch: `codex` is not on your PATH`` | Install Codex with `npm install -g @openai/codex`, or add the folder that holds `codex` to `PATH`. |
| `could not download Codex ...'s instructions` | The first start of each Codex version needs the internet once. Try again when online. A version that is not published on GitHub, such as one you built yourself, has no instructions to download. |
| `` `codex cloud` runs tasks on OpenAI's servers `` | Codex Cloud does not use this Mac, so the launch does not start it. Run `codex cloud` directly. |
| `` `--output-schema` asks for a reply that follows a JSON schema `` | Slotstream does not produce schema-constrained replies yet. Run the task without `--output-schema`. |
| `the server on port 11434 is busy` | Every connection the server takes is in use. Wait for running requests to finish and try again. |
| `Model metadata for ... not found. Defaulting to fallback metadata` | Codex is running without the catalog. Start it with `slotstream launch codex`, or check the manual setup: `model_catalog_json` above every `[...]` header, an absolute path, and an existing file. |
| `404` or `tool type 'custom' is not supported` | The server is older than 0.2.20. Update Slotstream with the install command. |
| `model called an undeclared tool: apply_patch` | Codex was started without the catalog, so `apply_patch` was never declared. Start it with `slotstream launch codex`. |
| `Reconnecting... waiting for network` | The server stopped, with `slotstream stop` or 30 minutes after the last session. Run `slotstream launch codex` again. |
| `stream disconnected before completion` or an idle timeout | The first prompt took longer than Codex waited. Raise `stream_idle_timeout_ms`, or give the server more memory so the prompt reads faster (`slotstream stop`, then `slotstream launch --memory-gb <gb> codex`). |
| `context_length_exceeded` | The prompt no longer fits the served window. Run the server yourself with a larger `--max-context`, then restart Codex with `slotstream launch codex`, so Codex knows the real window and compacts in time. |
| `No running Ollama server detected` or `Ollama ... is too old` | You used `codex --oss`. Use `slotstream launch codex` instead. |
| Connection refused | The server stopped, with `slotstream stop` or 30 minutes after the last session. Run `slotstream launch codex` again. Otherwise check that the port matches; see [port conflicts](TROUBLESHOOTING.md#the-server-cant-listen-on-port-11434). |

For details of the wire protocol, see the [API reference](API.md#v1responses).
[General troubleshooting](TROUBLESHOOTING.md) covers startup, memory, and
model files.


<!-- ===== docs/CLAUDE-CODE.md ===== -->

<a id="use-claude-code-with-slotstream"></a>

# Use Claude Code with Slotstream

Run Anthropic's Claude Code on the model Slotstream serves. Claude Code edits
files and runs commands through its own tools; Slotstream provides the model
over the Anthropic Messages API on your Mac. No Anthropic account or API key
is needed, and the model runs offline.

## Before you start

- Slotstream 0.2.21 or later, installed with its model, from the
  [getting started guide](GETTING-STARTED.md). `slotstream --version` shows
  yours; the install command updates it.
- Claude Code installed, from its
  [setup guide](https://code.claude.com/docs/en/setup).

These steps were verified with Claude Code 2.1.270. If a newer version
behaves differently, start with the troubleshooting table at the end.

## Start Claude Code

In Terminal, go to your project directory and run:

```sh
slotstream launch claude
```

If Slotstream is not running, this starts it in the background first and shows
its start until the model answers. The first reply then takes a while: Claude
Code's instructions and tool descriptions, about 15,000 tokens, are read once
before the first token. Later turns, and later sessions, read only what is
new.

The server keeps running while Claude Code runs, and stops 30 minutes after
the last session ends. `slotstream stop` stops it sooner, and
`tail -f ~/.slotstream/logs/serve.log` shows what it is doing. To start the
server yourself instead, see [run the server yourself](#run-the-server-yourself).

`slotstream launch claude` asks the server for its model and window, then
starts Claude Code with:

- the server as its API address, a placeholder token, and the Slotstream
  model for every model name Claude Code uses, so subagents and background
  tasks run on your Mac too;
- the served context window and reply limit, so Claude Code compacts the
  conversation before it outgrows the window;
- thinking off, request and stream timeouts long enough for a first
  prompt, and Claude Code's nonessential network traffic (telemetry, error
  reports, update checks) off;
- its `WebSearch` tool off, since it runs on Anthropic's servers.

The connection is passed with `--settings`. Claude Code applies those
settings over the `env` in your own settings files, so nothing there can send
this run elsewhere: Claude Code's switches for Bedrock, Vertex, Foundry and
its other cloud providers are turned off, and any `ANTHROPIC_API_KEY` or
`apiKeyHelper` is cleared, so no key of yours reaches the server. An exported
`CLAUDE_CODE_OAUTH_TOKEN` is not sent either; the placeholder token is. Your
settings files are not changed, and plain `claude` keeps using your
account. A managed settings file from your organization still takes
precedence over the launch.

Anything after `claude` goes to Claude Code:

```sh
slotstream launch claude --continue
slotstream launch claude -p "Summarize this repository"
```

If the server runs on another port, pass it before the tool name:
`slotstream launch --port 8080 claude`. To see the command and settings
without starting anything, add `--dry-run`.

## Try it

Ask for something that needs a file edit and a command:

```text
Create a file named hello.txt whose entire content is the single line: SLOTSTREAM OK. Then read the file back and tell me exactly what it contains.
```

Claude Code writes the file with its `Write` tool, reads it back, and reports
the contents. Check that `hello.txt` exists and holds the line.

Pictures you paste into Claude Code are read by the model on your Mac.

## Thinking

Thinking is off by default, which is faster. To turn it on, set
`MAX_THINKING_TOKENS` to any positive number; `CLAUDE_CODE_EFFORT_LEVEL`
(`low`, `medium` or `high`, the default) sets how much the model thinks:

```sh
MAX_THINKING_TOKENS=1 CLAUDE_CODE_EFFORT_LEVEL=low slotstream launch claude
```

Claude Code asks for its thinking to be hidden, so it shows that the model is
thinking but not the text.

## What differs from Claude

- **Cost.** Claude Code estimates a price for a model it does not know, so its
  cost figures are not zero. Nothing is billed; the model runs on your Mac.
- **Web search.** Claude Code's `WebSearch` tool runs on Anthropic's servers,
  so the launch turns it off.
- **Files.** Text files and pictures work. PDF documents and files uploaded
  through the Files API do not.
- **Prompt caching.** Claude Code's cache markers are ignored. Slotstream
  reuses prompts on its own: a conversation's next turn reads only what is
  new, and a new session starts from the instructions an earlier session
  read. See [shared prompts](#keep-shared-prompts-on-disk).
- **Model choice.** The model aliases `opus`, `sonnet` and `haiku`, in
  `/model`, `--model` and subagent settings, all run the Slotstream model. A
  full Claude model name, such as one pinned in a subagent file, is refused
  with a not-found error; use an alias there instead.

<a id="keep-shared-prompts-on-disk"></a>

## Keep shared prompts on disk

The server keeps a few recent conversations in memory. When other
conversations fill that space, a new Claude Code session would read the
instructions again. A server `slotstream launch` starts also keeps them on
disk, in `~/.slotstream/prefix-cache`, so new sessions start from the saved
instructions, also after the server restarts. `slotstream prefix-cache` lists
what is saved, and `slotstream prefix-cache --clear` removes it while no server
runs. [The `serve` reference](CLI.md#slotstream-serve) describes the directory
and its disk quota.

<a id="run-the-server-yourself"></a>

## Run the server yourself

To watch the server or choose its window, start it in its own Terminal window
before `slotstream launch claude`:

```sh
slotstream serve --max-context 65536 --prefix-cache-dir ~/.slotstream/prefix-cache
```

Wait for `slotstream listening on http://127.0.0.1:11434`. `slotstream launch
claude` then uses this server as it is, and it runs until you press
**Control+C** in its window or run `slotstream stop`. The larger window leaves
more room for the conversation. On a Mac with little memory it can leave too
little to keep a whole conversation for the next turn:
`slotstream doctor --max-context 65536` shows how much is kept at that size.
A server `slotstream launch` starts uses the window `slotstream doctor` picks.

## Configure Claude Code yourself

`slotstream launch claude` is the simplest way to start. To start `claude`
directly instead, set these variables first. Use the model name and window
that `curl -s http://127.0.0.1:11434/v1/models` shows:

```sh
export ANTHROPIC_BASE_URL=http://127.0.0.1:11434
export ANTHROPIC_AUTH_TOKEN=slotstream-local
export ANTHROPIC_MODEL=qwen3.8-flash-next:4bit
export ANTHROPIC_DEFAULT_OPUS_MODEL=qwen3.8-flash-next:4bit
export ANTHROPIC_DEFAULT_SONNET_MODEL=qwen3.8-flash-next:4bit
export ANTHROPIC_DEFAULT_HAIKU_MODEL=qwen3.8-flash-next:4bit
export ANTHROPIC_SMALL_FAST_MODEL=qwen3.8-flash-next:4bit
export CLAUDE_CODE_MAX_CONTEXT_TOKENS=65536
export CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
export MAX_THINKING_TOKENS=0
export API_TIMEOUT_MS=1800000
export CLAUDE_STREAM_IDLE_TIMEOUT_MS=1800000
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
unset ANTHROPIC_API_KEY CLAUDE_CODE_USE_BEDROCK CLAUDE_CODE_USE_VERTEX CLAUDE_CODE_USE_FOUNDRY
claude --disallowedTools WebSearch
```

An `env` block in a Claude Code settings file overrides these variables, so
first remove any `ANTHROPIC_` and `CLAUDE_CODE_USE_` entries, and any
`apiKeyHelper`, from `~/.claude/settings.json` and the project's
`.claude/settings.json`.

`CLAUDE_CODE_ATTRIBUTION_HEADER=0` removes a line that changes with every
conversation from the start of the instructions; Slotstream ignores that line
anyway.

## Troubleshooting

| Problem | What to check |
|---|---|
| `slotstream launch: no Slotstream server answered on port 11434` | You ran it with `--no-start`. Start `slotstream serve` in another Terminal window, or run `slotstream launch claude` without `--no-start`. If the server uses another port, pass it with `--port`. |
| `another Slotstream model process is running` | A server on another port, `slotstream run` or a check holds the model, and only one fits in memory. Stop it, or pass that server's port with `--port`. |
| `Slotstream stopped while starting` | The lines above it, and `~/.slotstream/logs/serve.log`, say why. If memory is short, close memory-heavy apps or pass a smaller `--memory-gb`. |
| ``slotstream launch: `claude` is not on your PATH`` | Install Claude Code from its setup guide, or add the folder that holds `claude` to `PATH`. |
| `does not have the Anthropic Messages API Claude Code needs` | The server is older than 0.2.21. Update Slotstream with the install command and restart the server. |
| Claude Code asks you to log in | It was started without the connection. Start it with `slotstream launch claude`. |
| `API Error: Request timed out` | The first prompt took longer than Claude Code waited. Give the server more memory so the prompt reads faster (`slotstream stop`, then `slotstream launch --memory-gb <gb> claude`), or export larger `API_TIMEOUT_MS` and `CLAUDE_STREAM_IDLE_TIMEOUT_MS` values before `slotstream launch claude`. |
| `the server on port 11434 is busy` | Every connection the server takes is in use. Wait for running requests to finish and try again. |
| `prompt is too long` | The conversation outgrew the window. Claude Code compacts it and continues; if it keeps happening, [run the server yourself](#run-the-server-yourself) with a larger `--max-context`. |
| `ignoring request field(s)` in the server's log | Claude Code sent options Slotstream does not use. The request still runs; the message appears once per field. |
| Every new session reads the instructions again | With a server you started yourself, add `--prefix-cache-dir`; see [keep shared prompts on disk](#keep-shared-prompts-on-disk). |
| Connection refused | The server stopped, with `slotstream stop` or 30 minutes after the last session. Run `slotstream launch claude` again. Otherwise check that the port matches; see [port conflicts](TROUBLESHOOTING.md#the-server-cant-listen-on-port-11434). |

For details of the wire protocol, see the [API reference](API.md#v1messages).
[General troubleshooting](TROUBLESHOOTING.md) covers startup, memory, and
model files.


<!-- ===== docs/CODING-AGENTS.md ===== -->

<a id="coding-agents"></a>

# Use coding agents with Slotstream

Coding agents such as Claude Code, Codex, Pi, opencode and Hermes run their
own loop: they read your files, run commands and ask a model what to do next.
Slotstream can be that model, running on your Mac. One command starts an
agent already connected to it:

```sh
slotstream launch <agent>
```

| Agent | Command | Guide |
|---|---|---|
| Claude Code | `slotstream launch claude` | [Claude Code](CLAUDE-CODE.md) |
| Codex | `slotstream launch codex` | [Codex](CODEX.md) |
| Pi | `slotstream launch pi` | [Pi](#pi) below |
| opencode | `slotstream launch opencode` | [opencode](#opencode) below |
| Hermes | `slotstream launch hermes` | [Hermes](HERMES.md) |

`slotstream launch` needs Slotstream 0.2.21 or later and the agent itself,
installed separately. These agent versions were verified: Claude Code
2.1.270, Codex 0.148.0, Pi 0.85.1, opencode 1.18.31 and Hermes 0.21.1.

## How it works

In your project's folder, run `slotstream launch` with the agent's name, or
without one to pick from the agents installed. It asks the server for its
model and context window, connects the agent to it, and starts the agent in
the same window. The agent then works as usual. Everything after the agent's
name is passed to the agent, for example
`slotstream launch pi -p "Summarize this repository"`.

When no server is running, `slotstream launch` starts one in the background
first and shows its start until the model answers:

- with the context window `slotstream doctor` picks for this Mac, or 65,536
  tokens for Hermes when that is smaller, since Hermes needs it;
- with prompt caches on disk in `~/.slotstream/prefix-cache`, so an agent's
  instructions are read once, also across restarts;
- with its log in `~/.slotstream/logs/serve.log`.

That server keeps running while any agent `slotstream launch` opened runs, so
the next session starts from the instructions already read. It stops 30
minutes after the last agent exits; `slotstream stop` stops it sooner.

To watch the server or choose its window, run it yourself in its own Terminal
window before `slotstream launch`:

```sh
slotstream serve --max-context 65536
```

The agents' instructions and tool descriptions are long, and the larger
window leaves room for the conversation. Hermes needs it; for the others it
helps on a Mac with memory to spare. `slotstream launch` uses a server you
started as it is, and never restarts it.

Options for `slotstream launch` go before the agent's name:

| Option | Meaning |
|---|---|
| `--port <n>` | The port the server listens on (default 11434). |
| `--memory-gb <gb>` | Memory target for a server it starts. Default: automatic. |
| `--idle-exit <minutes>` | How long a server it starts keeps running after the last agent exits (default 30); `0` keeps it running until `slotstream stop`. |
| `--no-start` | Use a running server only. |
| `--dry-run` | Print the server it would start, the command, variables and files, and start, download and write nothing. Keys and tokens are hidden. |

Each agent is connected in the way that leaves your own setup alone:

| Agent | How the connection is passed | What stays yours |
|---|---|---|
| Claude Code | Variables and `--settings` for this run | Your settings files and login |
| Codex | `-c` settings for this run, and a model description under `~/.slotstream/launch/codex/` | `~/.codex/config.toml` |
| Pi | A `slotstream` provider in `~/.pi/agent/models.json` | Everything else in that file, as written, and your default model |
| opencode | The `OPENCODE_CONFIG_CONTENT` variable for this run | Your opencode configuration files |
| Hermes | Its own folder, `~/.hermes-slotstream`, created once | Your usual `~/.hermes` |

The launch also closes the ways an agent could send your code to a hosted
model while you think it runs on your Mac. Claude Code's cloud provider
switches are turned off and its API key and key helper cleared for the run,
and its `WebSearch` tool, which runs on Anthropic's servers, is turned off.
opencode may use only the `slotstream` provider during the run. Hermes's
side tasks, such as the check that decides whether a command needs your
approval, use the same model. `codex cloud`, which runs tasks on OpenAI's
servers, is refused.

Thinking starts off, which is faster; each guide shows how to turn it on.
The first reply of a session takes a while, because the agent's instructions
are read before the first token; the server's log shows the progress, and
later turns read only what is new.

<a id="pi"></a>

## Pi

[Pi](https://github.com/earendil-works/pi) is a minimal terminal coding
agent. Install it with:

```sh
npm install -g @earendil-works/pi-coding-agent
```

Start Pi in your project:

```sh
slotstream launch pi
```

Pi reads custom providers from `~/.pi/agent/models.json`, so the launch adds
a `slotstream` provider there, or updates its address and window. Everything
else in the file stays as you wrote it, including other models you added to
that provider, and a file that is a link to your dotfiles stays a link. It
then starts Pi with `--provider slotstream --model qwen3.8-flash-next:4bit
--thinking off`. A `--model` or `--thinking` you pass yourself replaces the
launch's own. Pi uses `--provider` only together with `--model`, so
`--provider slotstream` alone gets the model added, and another provider
without a model is refused. Pi's own commands, such as `pi install` or
`pi update`, pass through unchanged. Your default model does not change:
plain `pi` starts as before, and the Slotstream model appears in Pi's model
list.

To think before answering, pass a level: `slotstream launch pi --thinking
low` (or `medium` or `high`).

### Configure Pi yourself

To add the provider by hand, put this in `~/.pi/agent/models.json`, merged
with any providers already there:

```json
{
  "providers": {
    "slotstream": {
      "baseUrl": "http://127.0.0.1:11434/v1",
      "api": "openai-completions",
      "apiKey": "slotstream-local",
      "compat": { "supportsStore": false, "supportsStrictMode": false },
      "models": [
        {
          "id": "qwen3.8-flash-next:4bit",
          "name": "Qwen3.8-Flash-Next (Slotstream)",
          "reasoning": true,
          "thinkingLevelMap": { "minimal": null, "xhigh": null, "max": null },
          "input": ["text", "image"],
          "contextWindow": 65536,
          "maxTokens": 8192,
          "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
        }
      ]
    }
  }
}
```

Set `contextWindow` to the window `slotstream serve` reports, and start the
server yourself as shown in [how it works](#how-it-works). Then run
`pi --provider slotstream --model qwen3.8-flash-next:4bit --thinking off`.
Pi asks for medium thinking unless told otherwise, so keep `--thinking off`
for the faster mode.

<a id="opencode"></a>

## opencode

[opencode](https://opencode.ai) is a terminal coding agent with its own
interface. Install it with:

```sh
npm install -g opencode-ai
```

Start it in your project:

```sh
slotstream launch opencode
```

The launch passes a `slotstream` provider, and the Slotstream model as the
main model, the small model opencode uses for titles, and the model of its
`build` and `plan` agents, in the `OPENCODE_CONFIG_CONTENT` variable. It also
enables only the `slotstream` provider for the run, so an agent or command
set to another provider fails instead of sending your code there. opencode
merges that with your own configuration files, which stay unchanged. If you
already export `OPENCODE_CONFIG_CONTENT`, the launch adds to it.

For a single task without the interface, use `slotstream launch opencode run
"Summarize this repository"`.

### Configure opencode yourself

To use Slotstream from plain `opencode`, add the provider to
`~/.config/opencode/opencode.json`:

```json
{
  "$schema": "https://opencode.ai/config.json",
  "model": "slotstream/qwen3.8-flash-next:4bit",
  "small_model": "slotstream/qwen3.8-flash-next:4bit",
  "provider": {
    "slotstream": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Slotstream",
      "options": {
        "baseURL": "http://127.0.0.1:11434/v1",
        "apiKey": "slotstream-local"
      },
      "models": {
        "qwen3.8-flash-next:4bit": {
          "name": "Qwen3.8-Flash-Next (Slotstream)",
          "tool_call": true,
          "attachment": true,
          "limit": { "context": 65536, "output": 8192 }
        }
      }
    }
  }
}
```

Set `limit.context` to the window `slotstream serve` reports, and start the
server yourself as shown in [how it works](#how-it-works).

## Troubleshooting

| Problem | What to check |
|---|---|
| `slotstream launch: no Slotstream server answered on port 11434` | You ran it with `--no-start`. Start `slotstream serve` in another Terminal window, or run it without `--no-start`. If the server uses another port, pass it with `--port`. |
| `another Slotstream model process is running` | A server on another port, `slotstream run` or a check holds the model, and only one fits in memory. Stop it, or pass that server's port with `--port`. |
| `Slotstream stopped while starting` | The lines above it, and `~/.slotstream/logs/serve.log`, say why. If memory is short, close memory-heavy apps or pass a smaller `--memory-gb`. |
| ``slotstream launch: `pi` is not on your PATH`` | Install the agent with the command in its section, or add the folder that holds it to `PATH`. |
| `the server on port 11434 is not Slotstream` | Another server, such as Ollama, uses the port. Stop it, or start Slotstream with `--port` and pass the same port to `slotstream launch`. |
| `needs a context window of at least` | You started the server yourself, or another agent is using it. Stop it with `slotstream stop` (or Control+C in its window) once it is free and run the launch again; it starts a server with that window. |
| `Pi's models file is not plain JSON` | The file has comments or another format the launch does not rewrite. Add the provider by hand as shown above. |
| Pi says `Unknown provider "slotstream"` | Pi could not read its models file. Its warning above names the entry to fix; Pi ignores the whole file while any entry is invalid. |
| `` Pi uses `--provider` only together with `--model` `` | Name a model with `--model`, or run `pi` directly to use another provider. |
| opencode fails with `UnknownError` | An agent or command in your opencode configuration names a model from another provider, which the launch does not enable. Set its model to `slotstream/qwen3.8-flash-next:4bit`, or run it with plain `opencode`. |
| `the server on port 11434 is busy` | Every connection the server takes is in use. Wait for running requests to finish and try again. |
| The first reply is slow | `tail -f ~/.slotstream/logs/serve.log` shows its progress, or the Slotstream window when you started the server yourself. Long first prompts take minutes on small memory targets. |
| A new session reads the instructions again | With a server you started yourself, add `--prefix-cache-dir` to keep shared instructions on disk, as in [the Claude Code guide](CLAUDE-CODE.md#keep-shared-prompts-on-disk). A server `slotstream launch` starts does this already. |
| Connection refused | The server stopped, with `slotstream stop` or 30 minutes after the last agent exited. Run `slotstream launch` again. |

[Connect apps and agents](CLIENTS.md) covers chat apps and other
OpenAI-compatible clients. For the wire protocols, see the
[API reference](API.md).


<!-- ===== docs/TESTING.md ===== -->

# Testing

Checks in `SlotstreamDiagnostics` return a `CheckReport`. The CLI, test
runner, and host apps use those same functions.

Choose a suite based on what you have installed:

```bash
make checks          # build, then run T0: no GPU, weights or network during the checks
make checks-all      # adds the MLX tier
make test            # Tools/verify.sh, the acceptance battery against real weights
make context-test    # isolated context policy + proxy fixtures; no MLX or weights
make coverage        # line coverage of the library
python3 Tools/process_memory_gate.py  # native CPU/GPU peak accounting, no model
python3 Tools/memory_override_gate.py # CLI override matrix on simulated Macs
```

The initial build can download Swift packages and the prebuilt Metal library.
The native process-memory regression compiles the production counter and uses
small Metal buffers. It checks that peaks survive buffer release, persistent
and temporary allocations remain distinguishable, concurrent reads retain the
high-water, and an older or invalid kernel reply cannot become a bogus peak.
The static gates run it automatically. Request samples remain separate from
the process-lifetime counter; the memory acceptance gate checks both when the
native lifetime observation is present.

## Memory acceptance and macOS paging

Correctness and process-memory acceptance do not require unchanged system-wide
swap counters. Those counters include every app on the Mac and cannot attribute
paging to Slotstream. The speculative, governor and context diagnostics record
`swap_clean` separately; the memory assessor reports `global_swap_deltas`.
Neither field overrides a completed correctness check or an observed process
footprint within its budget. Unavailable paging observations are not evidence
of a clean interval.

Real headroom checks, process-memory ceilings, OS pressure cancellation,
allocation safeguards and required output remain mandatory. Paging can still
indicate system contention, so retain the observations and investigate actual
pressure or loss of responsiveness. A passing functional run with paging does
not qualify clean benchmark timings. Performance studies keep their declared
exclusion rules; frozen historical results are not regraded under this policy.
See the [decision](../db/records/decisions/global-paging-is-diagnostic.md).

## Configurable context without a model

`make context-test` compiles the production planner, feasibility solver,
schedule and request controller with inert device observers. It uses the
existing allocation golden and process/transport fixtures. It needs Python
and a Swift compiler, but does not invoke SwiftPM, load MLX, touch weights,
start a server or simulate pressure on the host. The dedicated
`context-proxies` CI workflow runs this same command.

Every run writes a fresh report and raw logs. Set `CONTEXT_TEST_OUT` to choose
the directory. Reports bind the exact tested source and driver bytes, map
each acceptance case to its proxy scope, and list its deferred native checks.
Missing prerequisites or failing checks fail the command. Proxy success never
certifies tensor numerical parity, model capacity, speed, answer quality, a
real client installation or a release.

To check explicit windows against an already built candidate without a new
compile or model launch:

```bash
python3 -m unittest discover -s Tools -p context_window_matrix_test.py
python3 Tools/context_window_matrix.py \
  --binary /path/to/candidate/slotstream --out .build/context-window-matrix
```

The candidate needs its build identity, source archive and Metal library beside
it. This command invokes only `doctor` with simulated device metadata and
`prefill-schedule`. It checks the public ceiling, allocation ledger, cold and
continued scheduling, padded attention bounds and overflow refusals. Its report
identifies any source differences between the candidate and the current checkout;
it does not turn an older candidate's result into current-source build evidence.
These checks do not load the model or establish native capacity.

Create a portable source handoff without a binary or model:

```bash
python3 Tools/context_acceptance.py prepare --out .build/context-handoff
```

The handoff contains a source archive, its hashes, the acceptance inventory,
the original capacity profiles and explicit external dependencies. Preserve
the handoff manifest outside the extracted source. On the intended test Mac,
extract into a fresh directory, review the manifest, restore the pinned model,
and build that exact source with the normal `make build` procedure. The archive
contains the source and fixtures for the proxy/capacity workflow; the complete
repository brain/docs and independently installed clients remain dependencies
of the broader release battery.

Bind the new candidate and model on that target, then explicitly run native
capacity qualification there:

```bash
python3 Tools/context_acceptance.py bind \
  --handoff /path/to/context-handoff/handoff.json \
  --binary .build/release/slotstream --model /path/to/pinned-model \
  --out .build/context-binding
python3 Tools/context_acceptance.py run-capacity \
  --binding .build/context-binding/binding.json \
  --out .build/context-native --execute-on-target
```

Binding reads metadata and calls only `context-check --plan-only`. It checks
the candidate's source archive against the handoff; an older binary cannot
stand in for changed source. The native command runs the required governor
and combined draft/image resource checks, then the frozen incremental capacity
profiles in order. It preserves original retained warm-up lengths, verifies
model payloads, rechecks identities and real readiness, and stops the entire
campaign at the first failed stage. It never refreshes a failed baseline or
retries silently. Rebinding on a different host captures that host's model
metadata without changing the frozen workload.

Capacity success still leaves numerical, final API/client, installed release
and rollback acceptance separate. The catalog names those remaining checks;
the public context ceiling changes only after its native qualification. There
is no background model waiter and no automatic hardware fallback from the
software command.


## Building from source

To build from source, install Apple's Command Line Tools, then run:

```bash
git clone https://github.com/carloslfu/slotstream
cd slotstream
make build
make checks
```

`make checks` builds first, which can need network access, then runs T0
without weights, network access or a GPU. `make checks-all` adds the MLX tests.
`Tools/verify.sh` tests against the real model, including
reference comparisons, cache resizes, speculative decode, and server
regressions. [Testing](TESTING.md) explains the suites and coverage gaps;
[Contributing](../CONTRIBUTING.md) covers the development workflow.

Release builds come from tagged commits in GitHub Actions. After downloading
a release archive, you can verify its provenance with the GitHub CLI:

```bash
gh attestation verify slotstream-arm64.tar.gz --repo carloslfu/slotstream
```

## Why there is no `swift test`

The supported Command Line Tools setup lacks XCTest and Swift Testing.
The project uses a plain executable, `slotstream-checks`, so contributors
can run checks without installing Xcode. CI uses the same runner.

```bash
.build/release/slotstream-checks --list
.build/release/slotstream-checks --tier t0 --tier t1
.build/release/slotstream-checks --filter http --json
```

## Tiers

Tiers group checks by their dependencies. Choose the tiers your machine can
run; a T0 pass covers only T0.

| Tier | Needs | Runs |
|---|---|---|
| **T0** | Nothing. Pure Swift. | Every push |
| **T1** | MLX, and so the Metal library beside the runner | Every push |
| **T2** | The pinned tokenizer fixture | Not yet built |
| **T3** | A synthetic checkpoint | Not yet built |
| **T4** | The real 105 GB of weights | The dev Mac, per release |

**Run tiers above T0 sequentially.** Several checks in one process can still
allocate memory at the same time, despite the guard against multiple model
processes. Use the small explicit targets in `Tools/verify.sh` and check
available memory before a model test.

The full live-governor drill is a separate bounded exception: its normal
1 GB shrink and 2 GB grow deadbands require a starting arena larger than the
ordinary 10 GB tests. `verify.sh` uses `elastic-drill --slots 1000
--max-memory-gb 13`, after checking 16 GB reclaimable. The command independently
checks its derived total target plus 3 GB spare, each controlled poll and
generation and actual process-memory peaks, and records global paging separately. It preserves the real
cooldown and exact output checks. A skipped drill fails full acceptance.
Run this gate without other heavy work. Full model hashing holds the same
process exclusion lock as inference and must pass before native acceptance.

The battery also requires `elastic-drill --memory-limit-gb 10
--max-memory-gb 10 --mtp off`. This checks that a small cache recovers after
pressure even when the lost cache is below the normal growth threshold.
It preserves both cooldowns, the saved ceiling and exact output. The public
adaptive-server gate separately checks startup, the production timer,
status metadata and a completed request; its `--limit-gb` and `--no-elastic`
options cover fractional limits and explicitly pinned serving.

MTP diagnostics require and price the draft head before Engine allocation,
including when their `--mtp` option is left at `auto`; explicit `off` is
incompatible. The full `mtp-check` includes vision and uses an explicit 12 GB
target after a 15 GB reclaimable preflight. Its text-only leg can be selected
with `--vision off` under the ordinary 10 GB test target. A text-only pass does
not prove the combined image/MTP leg. `mtp-rowcheck` runs under the ordinary
10 GB target: it synthesizes one prompt below and one above the indexer budget
and requires that, in the exact mode, every row of a two-row and a three-row
verify pass reproduces the one-row pass at its position in the same mode bit
for bit, and that a three-row pass leaves the state three one-row passes leave
(checked through the next token's logits); the stock pass's deviation is
printed beside it. The shorter prompt is extended to just below 1,024 keys and
its positions advance one token at a time across that count, where the
backend switches attention kernels. The weights-free `verify-pass-rows` check
(T1) holds the kernels: every dense matmul shape, the exact mode's attention
at each key count where the backend changes kernels or block layout, its
indexer selection with tied blocks at the budget, and quantized products of
up to five rows. It also counts what the split and whole-pass paths change at
those points.

The full original vision-serving photographs need a separate profile:
`--memory-gb 14.5` with `SLOTSTREAM_PREFILL_CHUNK=3072`, MTP off, and a
20.5 GB real reclaimable preflight. The explicit workspace covers the larger
image's attention buffers while retaining the original photographs and
assertions. The old 10 GB profile correctly refuses that image before
dispatch. This override applies only to the full image server; ordinary
quality gates keep their smaller target. Successful and nonempty responses
are required before different-image answers count as content evidence.

### The Metal library

MLX finds its shaders beside the executable that is running, through `dladdr` on
its own code. `make build` puts `mlx.metallib` in `.build/release`, which the CLI
and the runner share, so T1 works there with no extra step. A test bundle would
need its own copy in `.xctest/Contents/MacOS/`. Without it the first MLX call
fails with `Failed to load the default metallib`.

The LCOV path also runs instrumented transport fixtures over real loopback
HTTP, including malformed responses, resumability, raw-source compatibility,
and sustained-memory bounds. Their line hits are combined with the catalogue;
network code is no longer represented only by weights-free catalogue coverage.

## What runs where

Each push and pull request runs the workflows for what it changes. `ci.yml`
builds and checks the engine. `sevra-mac.yml` checks the Mac app in
`apps/macos`, and also runs for engine changes because the app builds on the
engine. `context-proxies.yml` covers the context proxies, and `docs.yml` the
documentation and the brain.

| Suite | What it covers | Weights | Where |
|---|---|---|---|
| `slotstream-checks` (T0/T1) | prefill schedule, context policy, runtime and cache bounds, governor policy, pull integrity, machine planning, HTTP framing and routing, vision geometry, request shaping and the embedding splice, sampler behaviour, persistent prefix policy, state files, rows shared across turns, a shared head surviving the conversation's own checkpoint at its boundary, eviction and directory maintenance | no | CI + local |
| `Tools/static_gates.sh` | shell and python syntax, doc parity, fixture digests, manifest digests, planner gates, installer gates | no | CI |
| `Tools/sampler_gates.sh` | the sampler against a numpy reference, and the governor's branches | no | CI |
| `Tools/consumer_smoke.sh` | a package outside the repository can import and use the library | no | CI |
| `Tools/check_sevra_mac.sh` | the Mac app over scripted inference in disposable Homes: runtime, sources and the sandboxed document helper, reviewed changes, knowledge bases, skills, mini-apps, and offscreen checks of the production views | no | CI + local |
| `Tools/build_sevra_xcode.sh` | the Xcode project builds for Apple silicon, ad hoc signed, with the package versions the checks use and the helper, dbmd and Metal library in the bundle | no | CI |
| `Tools/verify.sh` | the acceptance battery: provenance, goldens, byte-equality across cache sizes and live resizes, MTP, the memory promise, long context | **yes** | dev Mac |
| `Tools/api_robustness.sh` | Serving regressions against a live server | **yes** | dev Mac |
| `Tools/issue21_gate.py` | Chat Completions caps with and without tools, incremental truncated scalar/nullable-string arguments, compatible branch selection and later-turn reuse when reasoning is omitted, terminal usage and process survival | **yes** | dev Mac, already-running server |
| `Tools/issue21_e2e.py` | issue-21 and OpenAI compatibility gates plus exact conversation replay after restart; saves requests, SSE, commands and cleanup receipts | **yes** | `Tools/verify.sh`, owns one bounded server at a time |
| `Tools/issue21_long_context.py` | long tool-enabled conversations, advancing disk reuse when reasoning is omitted, and identical prompt/output after a real restart; captures raw SSE and progress logs | **yes** | dev Mac, owns one bounded server at a time |
| `sevra-mac-checks --real-basics` | the Mac app's basic jobs with the real model: a PDF answer, a reviewed edit, a mini-app and an attached file | **yes** | dev Mac |
| `optimization-state-check --variant persistent-prefix[-mtp]` | a persisted state restores with the saved representation; a disk hit continues exactly like a memory hit; that continuation, written as reused plus new rows, restores exactly; a regenerated reply resumes the kept parent; a request that keeps its state off disk writes nothing; draft cache included | **yes** | dev Mac |
| `optimization-state-check --variant shared-prefix[-mtp]` | a prompt's system message is kept during its own prefill at the last 256-token pass end at or before its boundary, forked into memory and written to disk as a shared prefix; a second conversation reuses it from memory and a fresh cache restores it from disk, both continuing exactly like the cold prompt; a prompt sharing only a head with a kept state writes that head; a `sharedPrefixTokens` hint replaces the system boundary; a request kept off disk writes nothing; the shared prefix outlives later turns and is classed after conversations; draft cache included | **yes** | dev Mac |
| `Tools/shared_prefix_e2e.py` | shared prefixes through `serve`: the first conversation writes its system prompt during prefill; a second one in the same process reuses it from memory; a restarted server restores it from disk; a conversation whose system prompt shares only a head writes that head and its own system prompt; another restarted server reuses those; later turns leave the shared prefixes in place and `prefix-cache` lists them; reused conversations' output ids equal a server without a prefix cache exactly | **yes** | dev Mac |
| `Tools/persistent_prefix_e2e.py` | the disk prefix cache through `serve` over three turns: turn 1 keeps the system prompt as a shared prefix and later turns write only their new rows; a restarted server must restore the turn-2 state from its segments and match the first server's turn-3 prompt and output ids exactly; another restarted server regenerating turn 3 must restore the kept parent and match again; `prefix-cache` lists and clears copies; a cold server shows the prompt cost it saves | **yes** | dev Mac |
| `Tools/vision_ref.py` | the vision tower against an independent float32 implementation of the reference | tower only (0.9 GB) | dev Mac |
| `Tools/vision_serving.py` | every dialect with a real picture, against a live server | **yes** | dev Mac |
| `Tools/e2e_release.sh` | the installed release, end to end | **yes** | dev Mac, per release |

<a id="why-vision-needs-two-of-those"></a>

### Vision checks

A faulty image encoder can produce embeddings with the correct shape while
losing the image content. `Tools/vision_ref.py` compares the encoder with an
independent implementation. `Tools/vision_serving.py` checks the full request
path by requiring the model to identify the photograph's content.

The encoder comparison allows numerical variation from bfloat16 arithmetic.
The tolerance comes from comparing the reference at float32 and bfloat16;
slotstream must stay within that band. The two independent float32
implementations agree to 0.99996. These checks test implementation correctness,
not general vision accuracy.

## OpenAI agent integration

The `openai-conversation`, `openai-tool-output`, and `openai-context-budget`
catalogue checks cover request/history semantics, complete-call publication,
stream equivalence, and the separate default/maximum context budgets. The
`responses-request`, `responses-events`, `responses-codex-tools`, and
`responses-codex-fixture` checks cover the Responses API that Codex uses:
item and tool parsing, the event stream and its response object, Codex's
real tool schemas, and a captured Codex first-turn request. The
`anthropic-request` and `anthropic-events` checks cover the Messages API that
Claude Code uses, with fixtures in the shapes Claude Code 2.1.270 sends: the
system prompt and its attribution line, tool loops with replayed thinking
signatures, images, documents and errors, and the streamed events and
non-streamed message. `think-split-stream` checks that streamed reasoning and
answer split exactly as a whole reply does, wherever deltas break.
`serving-edges` covers tool arguments the model writes out of range, stop
sequences after reasoning, and the request line of refused requests.
`launch-plans` builds every `slotstream launch` plan without a server, tool or
file system: the connection each agent receives, the routes to cloud
providers each plan closes, Codex's `-c` placement under its subcommands, the
Pi models file edited in place with its order and numbers kept, the opencode
configuration, Hermes's folder, profile and side-task pins, dry-run redaction,
and the refusals. `launch-server` covers the server `slotstream launch`
starts: its command line, paths and messages, when a running server is
restarted or left alone, the agent picker, the `/slotstream/status` body, and
the idle policy with a fake clock and fake processes, including a process id
reused by another program and a child that exited but was not collected.

`slotstream launch` itself is accepted against a real model with every agent
installed:

```sh
AGENT_PATH=/path/with/claude/codex/pi/opencode/hermes/and/node \
  Tools/coding_agents_gate.sh .build/arm64-apple-macosx/release/slotstream /tmp/coding-agents
```

It starts one server at `MEMORY_GB` (default 12) behind
`Tools/token_usage_proxy.py`, which records each request's prompt and reused
tokens. Each agent, in a throwaway home, creates `hello.txt` with the line
`SLOTSTREAM OK` and reads it back in one session, then reads a note in a
second session that must start from the instructions the first one read;
Claude Code also reads a picture, and runs again across a server restart
with `--prefix-cache-dir`. The Anthropic Python SDK's stream accumulator, a
tool result and a token count run against the same server when
`ANTHROPIC_SDK_PYTHON` names a Python that has the SDK. Name phases after the
output folder to run only some of them; `FAKE=1` swaps the model for
`Tools/launch_fake_server.py` to check the script's own plumbing. Follow the
repository's model-process and memory rules before running it. Its launches
pass `--no-start`, so they use only the server the script measures.

The server `slotstream launch` starts on its own is accepted with Claude Code,
Pi and Hermes installed:

```sh
AGENT_PATH=/path/with/claude/pi/hermes/and/node \
  Tools/launch_start_gate.sh .build/arm64-apple-macosx/release/slotstream /tmp/launch-start
```

It uses port 11531 and a throwaway home whose model folder links to the real
one. Its phases, in order: `--no-start` refuses and starts nothing; a dry run
describes the server and starts nothing; a terminal is asked which agent to
start; a Hermes folder the launch cannot use is refused before any server
starts; the first launch starts the server in its own session and answers; a
second launch reuses it; Hermes restarts it with a 65,536-token window;
`slotstream stop` stops it; two Pi launches at once start one server; a
server with a short `--idle-exit` stops by itself only after its agent exits;
a server started by hand is never restarted; Control-C during a start stops
that server; and `slotstream stop` during a start stops it too. Name phases
after the output folder to run only some of them. Each phase that loads the
model first waits until no other model process runs.

Against an already-running server, run `python3 Tools/openai_tools_gate.py
--output /tmp/openai-tools.jsonl`. This exercises the real model and HTTP/SSE
wire contract and saves every request and response. It supplies fixed tool
results after validating calls, without executing model-authored commands.
The [Hermes guide](HERMES.md) covers the real-client configuration and fixture
read. Run the ordinary API, gateway, and image gates when changing shared
serving code.

For an installed Hermes source checkout with its own environment:

```sh
/path/to/hermes/.venv/bin/python Tools/hermes_config_gate.py \
  /path/to/hermes /tmp/hermes-config-check
/path/to/hermes/.venv/bin/python Tools/hermes_integration_gate.py \
  /path/to/hermes /tmp/hermes-slotstream-check --compress --long-output --contaminated
```

Both gates read the configuration from the guide and use Hermes's CLI agent
initialization. The configuration gate uses synthetic HTTP responses with real
network access disabled. It checks output limits, optional reasoning, stale
custom-provider settings, an explicit profile override, missing/disabled providers, unavailable endpoints,
title fallback, auxiliary timeouts, and preservation after failed summaries.
It also checks that the guide pins every side task Hermes defines to the main
model, and that with the server failing or stopped the command-approval check
asks the user instead of sending the command to another provider.
Run it separately against each supported Hermes checkout.

The integration gate uses a real model and requires the larger context in the
Hermes guide. To check the guide's automatic planning, start the server with
`slotstream serve --max-context 65536`, following the repository's model-process
and memory-safety rules. The gate records the running server's memory plan and
checks its context. Add `--expect-mtp on` or `--expect-mtp off` to require the
selected state when qualifying that path; a planning-only `doctor` result does
not prove which path an integration run exercised. It creates an isolated
Hermes home, denies non-loopback Python network connections, permits only the
fixture's `cat` command through the actual Hermes tool dispatcher, and checks
the real agent, title fallback, compaction, and recall. `--long-output` also
requires a complete reply beyond the ordinary server default, with tools disabled
for that probe. `--contaminated` adds stale generic-provider settings.
`--cli` checks the actual CLI entry point instead of the multi-turn scenario;
run it separately. Raw HTTP and result records stay in the output directory.
Add `--image Tools/assets/vision_test/secret1.jpg` to check Hermes's own vision
discovery and an actual image turn. The server must have enough memory for both
the configured context and the vision tower. The OpenAI gate's `--vision` option
also checks image tool calls and the advertised capability.

## Coverage

```bash
Tools/coverage.sh t0 t1 --lcov coverage.info
python3 Tools/coverage_ratchet.py coverage.info
```

`swift test --enable-code-coverage` is not available here, so the runner is
built with the profiling instrumentation directly and `llvm-cov` reads what it
wrote; the CLT ships `llvm-profdata` and `llvm-cov`, just not the test modules.

Coverage is a review aid. Historical per-file percentages do not block CI.
The job still fails on a build error, a failed instrumented check, or a missing
or invalid report. It uploads LCOV and puts per-file changes in the job summary.
`Tools/coverage-floor.json` is the retained comparison snapshot; its historical
name is kept for compatibility. `--update` deliberately refreshes that snapshot,
and is not needed to make CI pass.

Inspect uncovered behavior in each change. Memory limits, recovery, cache
integrity, cancellation, download verification and public API compatibility
require meaningful boundary and failure checks. Regression checks should fail
when the bug is reintroduced. All existing correctness suites remain required.
Context-proxy, CLI and real-model checks run separately and are not included
in this coverage report. A narrower coverage gate needs reliable measurement
and a specific risk justification before adoption. See the
[coverage policy](../db/records/decisions/coverage-as-review-feedback.md).

<a id="where-the-coverage-is-not"></a>

### Initial coverage snapshot

The table below records the initial weights-free suite: 21.73% of 7,138
library lines, from 121 assertions. It is a historical snapshot, not a current
coverage report. Run the commands above for the current checkout.

| File | Lines | Covered | Why the rest is not |
|---|---|---|---|
| `Server.swift` | 1,132 | 6% | The socket loop and the request handlers. The framing, routing and CORS rules are split out and covered; the handlers still need an engine to answer with. |
| `WeightDownload.swift` | 642 | 0% | Historical coverage snapshot. The dedicated `Tools/slotpack/checks.py` gate now exercises real HTTP multi-chunk raw and compressed pulls, resume, corruption, fallback, cancellation, optional-file races, file safety, and manifest/codec bounds. |
| `Layers.swift`, `ExpertStore.swift`, `Engine.swift`, `Checkpoint.swift`, `Model.swift`, `NgramStore.swift`, `GatedDelta.swift` | ~2,900 | 0–3% | The model. These need a checkpoint. On the dev Mac they are covered by parity against the Python reference and by the byte-equality gates; a synthetic checkpoint would bring that to CI. |
| `Generate.swift` | 388 | 19% | The sampler is covered; the prefill and decode loops, and the sweep's admission and cache-cap hooks, run only with the model loaded. Gated by `sweep-check` and `Tools/verify.sh`. |
| `Governor.swift` | 213 | 27% | The policy is fully covered as a pure function. The live loop — poll, decide, lock, resize — still needs an engine to resize. |

The snapshot covers the weights-free runner. Tests against the real model,
such as `verify.sh` and `api_robustness.sh`, exercise additional paths locally;
those runs aren't included in this coverage percentage.

## Download transport gates

`Tools/static_gates.sh` runs `python3 Tools/slotpack/checks.py` without the
model. Its native harness compiles exact production sources into an immutable
per-run executable. The gate includes AddressSanitizer/UndefinedBehaviorSanitizer
codec checks, manifest identity and coverage, real HTTP fault injection, and
legacy raw multi-chunk compatibility. Receipts include source and binary hashes.

A new package also requires a full original-hash-checked offline build, an
independent public CDN reconstruction through the actual CLI default, and a
model-load smoke test before release. See [DOWNLOAD-FORMAT.md](DOWNLOAD-FORMAT.md)
for the producer, full-pull, and libFuzzer tools. Full transfer timings are
diagnostic unless the machine and network conditions qualify as a benchmark.

## Configurable context gates

`Tools/context_gates.py --report result.json` compares frozen default allocation
fields and validates CLI bounds, metadata, complete schedules and strict tool
termination without loading weights. The frozen allocation is pinned at an
explicit 32,768-token window, and the automatic window for the frozen tiers is
checked against `Tools/fixtures/context-automatic-v1.json`.
`Tools/planner_gates.sh` also checks each tier's automatic window, quiet and
busy starts, fixed caches and explicit windows. `Tools/consumer_smoke.sh` compiles the
original public function signatures as an external package.

The native `optimization-state-check --variant context-serving --json` injects
memory and monotonic-clock failures through real HTTP handlers and a single
floor-sized model, including queued requests and subsequent recovery. Variants
`context-small-projections-64` and `context-small-projections-128` exercise the
complete bounded arithmetic family with a prospectively fixed rechunking
control, repeated state checks and rollback/continuation checks. The matching
`partial`, `prefix`, `shorttail` and `sparse-prefix` variants cover boundaries
and reused state. These are correctness gates; their synthetic prompts do not
demonstrate answer quality.

`Tools/context_qualification.py` accepts a frozen binary/model-window protocol
and advances through strictly increasing prompt lengths only when the preceding
rung completes its required output within its planned memory and independent
wall-clock limits. Global paging remains diagnostic. The protocol binds the driver files,
reconstructible build, pinned model manifest and model directory observations;
the runner fully verifies model payload hashes before inference. It checks
actual physical query rows and padded key extents as well as prompt and
delivered output IDs. The retained protocol additionally requires completed,
interleaved warm conversations and exact observed cache ownership.
`Tools/context_qualification_checks.py` verifies refusal of incomplete,
over-budget and malformed evidence, including stopping after a failed rung.
It preserves the first counterexample and
never retries or changes the protocol. Full-window capacity, numerical parity,
latency calibration, advertised MTP/vision combinations and real clients remain
separate acceptance requirements in the engineering plan.

`Tools/gateway_client_gate.mjs <sdk-root> <output-dir> <port> <context>` uses
the separately installed, published `ai` and `@ai-sdk/gateway` packages. It
records their versions and checks discovery, streaming and a complete tool
round trip through an allowlisted fixture read. Requests stay on the selected
loopback server. This successful-client check does not replace the strict
receiving-side terminal and authority gates.

## Measure your Mac

Allow about ten minutes once the weights are downloaded. For comparable speed
results, reduce competing load and exclude timing intervals affected by paging.
Functional checks may run with apps open when real memory safeguards permit
them. Run one model process at a time.

1. Install or upgrade, then record the version:

   ```bash
   curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh
   slotstream --version
   ```

2. Print the plan. Copy the whole `slotstream memory plan` block; it carries
   the device line, the target, and the cache size:

   ```bash
   slotstream doctor
   ```

3. One cold generation. This offers the download on first use. When it
   finishes, `run` prints `--` lines to stderr: prefill, decode, and the
   expert-cache line that ends with the peak. Copy all of them.

   ```bash
   slotstream run --greedy --max-tokens 128 --prompt "Explain how a hash map works, in about 200 words."
   ```

4. Warm decode. Start the server in one terminal:

   ```bash
   slotstream serve
   ```

   In another, send the same request three times and keep all three
   results. The third is the warm number. If you would rather not run the
   Python one-liner, the JSON carries `eval_count` and `eval_duration` in
   nanoseconds; decode tok/s is the first divided by the second, times a
   billion. Only decode is printed: the second and third requests reuse the
   whole prompt, so their prompt timings measure no prefill. Step 5
   measures prefill.

   ```bash
   for i in 1 2 3; do
     curl -s localhost:11434/api/generate -d '{
       "model": "qwen3.8-flash-next:4bit",
       "prompt": "Explain how a hash map works, in about 200 words.",
       "stream": false,
       "options": {"temperature": 0, "num_predict": 128}
     }' | python3 -c 'import json,sys; d=json.load(sys.stdin); print("decode %.2f tok/s" % (d["eval_count"]/d["eval_duration"]*1e9))'
   done
   ```

   Press **Ctrl+C** in the server terminal before the next step.

5. Measure a long prompt. It reports time, speed, and peak memory, checking
   available memory between passes. Use 4096 tokens on a small Mac.

   ```bash
   slotstream context-check --tokens 8192
   ```

6. Open a [measurement report](https://github.com/carloslfu/slotstream/issues/new?template=measurement-report.yml)
   and paste the raw output from steps 1 to 5, plus the Mac model, the SSD,
   the macOS version, what else was open, and whether the fans ran or the
   machine throttled.

Single runs vary by 15% or more on a loaded machine. If two runs disagree by
that much, say so rather than picking the better one.

## Prompt-speed qualification

The prompt-speed diagnostics use one bounded model process at a time. Check
reclaimable memory and wait for other model runs and builds to finish first.
`prompt-checkpoint` checks exact interior reuse, disk arithmetic provenance,
and admission of the larger automatic scope. The phase checks cover pending
token ownership, later cold-equivalent reuse, cancellation and private state:

```bash
.build/release/slotstream optimization-state-check --variant prompt-checkpoint --json
.build/release/slotstream optimization-state-check --variant generation-phase --json
.build/release/slotstream optimization-state-check --variant generation-phase-mtp --json
```

`prompt-scopes-bench` runs warmups and alternating pairs in one loaded engine.
It excludes paging-contaminated pairs from its timing decision and checks
identical output, unchanged compute shapes, read bytes and physical memory.
The separate `scope-larger-family --tokens 8192` numerical gate uses
`SLOTSTREAM_OPT_WORKSPACE_TILE=1024 SLOTSTREAM_OPT_SCOPE_FRONTIER=1`.

After `Tools/check_sevra_mac.sh`, place the pinned Metal library beside
`apps/macos/.build/release/sevra-mac-checks`. Its `--real-cache --home <new-dir>`
check uses synthetic inventory text to test save, unload, reload, incognito
isolation and thinking-state exclusion. The existing `--real-thinking` and
`--real-metrics` checks exercise the app's phase transition and recorded
statistics. Each requires its own new disposable Home; none is a clean
throughput benchmark.


The fused prefill integration adds a scalar Double reference over the actual
BF16 inputs to the MLX check tier. It exercises causal and sparse masks,
strided queries, grouped heads and partial tiles, and checks exact fallback
for decode and unsupported dtypes. The real-model checkpoint diagnostic
requires fusion to execute and preserves exact warm/cold cache equivalence:

```bash
.build/release/slotstream optimization-state-check --variant fused-prefill-component --json
.build/release/slotstream optimization-state-check --variant fused-prefill-checkpoint --json
```

Run the phase, app-cache and ordinary acceptance gates against the deployed
settings too. The arithmetic-preserving `integrated` diagnostic remains a
separate reference test. A kernel upgrade is evaluated against numerical and
task-quality evidence; changing an old greedy token alone does not establish
an error. Never regenerate the historical parity goldens with the upgraded
backend. For timing, retain the old executable and its matching Metal library,
use alternating paired runs, and separately compare the new executable with
`SLOTSTREAM_OPT_FUSED_PREFILL=0`. Exclude intervals with paging from speed
claims while preserving their functional results.

The qualified profile automatically couples fused-workspace accounting with
larger expert-read groups. Test the default with no optimization environment
overrides using the `fused-workspace-component`,
`prefill-opportunity-equality`, `prefill-followup-lifecycle`,
`prefill-followup-checkpoint`, `prefill-followup-mtp-equality` and
`prefill-followup-mtp-vision` variants. Equality accepts `--tokens 16387` and
checks raw logits, every retained state tensor and teacher-forced continuation
against the original grouping. Run one model process at a time with the
documented memory preflight. The separate `prefill-opportunity-compute` probe
requires `SLOTSTREAM_OPPORTUNITY_PROMPT_FILE` and `SLOTSTREAM_PREFILL_CHUNK`;
it tests physical feasibility at a fixed floor-sized pool, not normal planner
acceptance or an output-quality guarantee. The catalogue includes the pure
policy's hardware, dtype, image, shape, key-boundary and explicit-disable
fallbacks, plus MTP phase lifetimes, shifted draft positions and retained output
accounting and the tradeoff between write barriers and larger read groups.
Run `optimization-state-check --variant prefill-followup-mtp-equality --tokens 16387` to require that
MTP actually exercises larger automatic groups while preserving prompt logits,
retained state, speculative output and continuation exactly.
Use `SLOTSTREAM_OPT_FUSED_WORKSPACE=0` for the original main-attention accounting
and automatic group cap; explicit group controls remain diagnostic tools.
Preserve excluded timing cells and the fixed trial limit in the
[initial automatic-policy validation](../db/records/measurements/automatic-prefill-policy-2026-09-21.md)
and [MTP qualification](../db/records/measurements/mtp-prefill-policy-2026-09-21.md).

`Tools/verify.sh` requires a Python environment with `mlx==0.32.2` and
`mlx-lm` for the independent current-backend model comparisons. It defaults
to `.venv/bin/python`; `SLOTSTREAM_REFERENCE_PYTHON` selects another interpreter.
`Tools/current_backend_reference.py` writes into a new verification directory,
holds the model lock and enforces a physical-memory ceiling. Its main-layer
reference reads only requested n-gram rows from the original weights.

Historical MLX 0.31 layer goldens still run with `parity --row-invariant`,
which selects the original one-row projection arithmetic. The ordinary
production projections are checked separately against the current Python
reference. Both comparisons retain the existing numerical tolerance. The old
draft-head golden is also run and its cross-backend differences are reported
explicitly as diagnostics; the current-backend draft-head comparison remains
required. Neither historical fixture is regenerated or relaxed.

`prefix-check` requires live reply equality, reuse and invalidation behavior.
It also prints the historical experiment that compared arbitrary batch
schedules. `--legacy-rechunk-bounds` reinstates that experiment's old numerical
bounds and depth heuristic when reproducing its original protocol. Those
different arithmetic schedules are not a cache-corruption oracle.
`prefix-exact-check` remains a required, separate gate for bit-identical raw
logits and tokens under the actual cache schedule.


<!-- ===== docs/TROUBLESHOOTING.md ===== -->

# Troubleshooting

Start with `slotstream doctor`. It checks your Mac's memory and disk space
without loading the model. Find the problem that matches what you're seeing below.

## `slotstream: command not found` after installing

Open a new Terminal window and try again. If that doesn't help, run the
[installer](GETTING-STARTED.md#install) again and read its final message.
It tells you how the `slotstream` command was added to your shell.

## The server can't listen on port 11434

Another app is using the address Slotstream needs. Ollama uses the same
port by default, and `slotstream launch` may have started a Slotstream server
there in the background; `slotstream stop` stops that one. Stop the other
server, or start Slotstream on a different port:

```sh
slotstream serve --port 11500
```

Use port 11500 in your chat app's connection settings too. Slotstream checks
this before loading the model.

## Another model process is already running

Slotstream runs one model process at a time to limit memory use. Find the
Terminal window where you started it and press **Control+C** before retrying.
`slotstream launch` starts its server in the background, with no window;
`slotstream stop` stops it, and `slotstream stop --port <n>` a server on
another port. If you're unsure what's running, this command lists Slotstream
processes:

```sh
pgrep -fl slotstream
```

## The whole Mac is slow

Close memory-heavy apps and check `slotstream doctor` for memory warnings.
Stop Slotstream with **Control+C** and restart it without custom memory
settings so it can choose a size that fits.

An 8 GB Mac can't run the model: even the minimum setting needs more memory
than it has, so Slotstream refuses to start. See
[hardware and speed](HARDWARE.md) for the limits of smaller Macs.

For manual adjustments, read the [memory settings](CLI.md#memory-options).
Forcing a larger setting can make the whole Mac slower.

## Generation is slower than the estimate

The estimates don't account for every chip, SSD, or temperature. Compare
your Mac with the [measured results](HARDWARE.md#results), especially if it
has less memory or a slower disk.

Other apps can leave less memory for Slotstream. Close them and let the
server adjust, or restart it. Avoid forcing a larger memory setting just
to match an estimate.

Replies can also be slower just after starting the model. Later requests
can reuse parts already loaded into memory.

## The first token takes a long time

Slotstream has to process your question and conversation history before it
starts replying. The terminal shows progress during long requests.
`slotstream doctor` estimates waits for different prompt lengths.

For a full conversation at the default limit, the estimated wait is about
3.0 min for the 48 GB M5 Pro plan and 6.4 min for the 16 GB plan. These
estimates come from the M5 Pro; slower SSDs can take longer. Follow-up turns
reuse unchanged history while it stays in memory. To keep long conversations
across server restarts as well, start the server with
`--prefix-cache-dir <folder>`; see the [command reference](CLI.md#slotstream-serve).
A server `slotstream launch` starts does this already, in
`~/.slotstream/prefix-cache`.

If Hermes or fx gives up before Slotstream replies, check the timeout
settings in the [Hermes guide](HERMES.md#troubleshooting) or
[fx guide](FX.md#the-first-turn-takes-minutes).

## A download was interrupted or may be damaged

Run this again to resume:

```sh
slotstream pull
```

Keep the partially downloaded files; Slotstream reuses its saved progress.
To check existing files without downloading anything, run:

```sh
slotstream pull --verify
```

It checks for corruption and names any damaged files. The optional file
used for speculative decoding is skipped if absent.

For transport options and how verification works, see the
[download reference](DOWNLOAD-FORMAT.md#download-behavior-and-measured-checks).

## Store the weights on another disk

The model files are also called *weights*. Replace the path below with a
folder on your external SSD. Keep the quotes if the path contains spaces:

```sh
slotstream pull --dir "/Volumes/My SSD/slotstream-model"
slotstream serve --model "/Volumes/My SSD/slotstream-model"
```

If the model files are already there, only the second command is needed.
An external disk may be slower than the internal SSD.

## Reclaim disk space or uninstall

Stop Slotstream first. In Finder, choose **Go → Go to Folder** and enter
`~/.slotstream` to find its files.

Delete the `models` folder there to remove the downloaded model and keep
the program. This frees about 105 GB after a full download.

If you started the server with `--prefix-cache-dir`, that folder holds saved
conversations. Run `slotstream prefix-cache --dir <folder> --clear` to empty
it, or delete the folder.

To remove both, delete `~/.slotstream`. Also remove the
`/usr/local/bin/slotstream` wrapper or the PATH entry the installer added to
your shell profile. If you chose a custom install or model folder, use
that location instead. The [installation reference](CLI.md#file-locations)
lists the default paths.

## Problems with older versions

[Run the installer again](GETTING-STARTED.md#update-or-get-help) to update.
Then stop and restart any running server. `slotstream --version` shows your
installed version; the [changelog](../CHANGELOG.md) lists the fixes in each release.

## Problems on macOS 14 or 15

The installer is tested on these versions; the runtime still needs testing.
[Open an issue](https://github.com/carloslfu/slotstream/issues/new) with the
error and your `slotstream doctor` output.

## The context window is smaller than `doctor` showed

Auto picks the window from your Mac's memory, then plans it against the
memory available when the server starts. If other apps hold too much then,
Slotstream starts with a smaller window and says so in its startup lines,
instead of turning speculative decoding off. The window stays for that
server's lifetime. Close memory-heavy apps and restart the server to get the
larger window, or pass `--max-context N` for a specific size; a window that
doesn't fit is refused with the largest one that does.

## A long request is refused or interrupted

The feasibility report and request deadlines below are available starting
in Slotstream 0.2.14. Check `slotstream --version` and the installed command's `--help`.

Inspect `slotstream doctor --max-context N --json` with the same memory options
as the server. `context_feasibility` reports what fits in memory; an estimate
of `null` means that schedule has no qualified throughput estimate.

For `prefill_wait_exceeded` or `prefill_deadline_exceeded`, send less missing
history, reuse an exact valid prefix, or deliberately raise
`--max-prefill-wait`. The default is 30 minutes from accepted request to first
model token, including queueing and images. `0` disables only time. Client read
and stale-stream timeouts must allow the selected server budget.

For `insufficient_memory`, free memory or reduce the memory/context target.
A large image can exceed the attention-workspace allowance even after its
resident tower fits; resize the image. Neither `--no-elastic` nor a fixed pool
turns off memory checks. After an interruption, retry a short request; an
unavailable engine returns an explicit error instead of resuming partial state.

For other unresolved problems, include your Mac model, memory, Slotstream
version, the command you ran, and the error message. Remove credentials
and private file contents before posting.


<!-- ===== docs/DOWNLOAD-FORMAT.md ===== -->

# Model download and Slotpack v1

Fresh `slotstream pull` downloads use immutable, losslessly compressed objects
from the public Hugging Face mirror. No download token is needed. The installed
model retains the original safetensors, tokenizer, configuration, and draft-head bytes.
Inference does not load the transport format or require a decoder at runtime.

The compressed package has its own `-Slotpack` repository. The original
repository remains usable by ordinary Hugging Face clients without downloading
both representations. Use Slotstream to reconstruct the compressed package;
its objects are not directly loadable model files.

The complete package is **88,295,438,048 bytes**, including its manifest,
representing **105,264,463,248 original bytes**: **16.12% fewer bytes**.
The manifest is embedded in the executable, so a normal pull transfers only
the objects. This is a measured byte reduction, not a promised reduction in
total installation time. Decoding and writing overlap downloads; a slow CPU
or disk can limit a very fast connection.

## Download behavior and measured checks

The installed model has 25 files, including the optional 1.5 GB draft head.
A missing draft head still allows inference with speculative decode off.
Since 0.2.19 `pull` also fetches one optional sidecar outside the compressed
objects, after the weights: `lookahead/tap-correction-attention-rank128-v1.safetensors`
(37,540,708 bytes), the decode forecast correction, pinned by size, SHA-256 and
mirror commit and written atomically. It is not part of the manifest, so every
existing pull and manifest revision is untouched; a missing or failed sidecar is
reported and never fails a pull, and the engine then runs the previous forecast.
Decoding and disk writes overlap the transfer. The client starts at eight
independent connections and increases concurrency only when measured
throughput improves. `--connections` fixes the count; `--transport raw`
selects the original file-based download. Existing raw partial downloads
keep their progress automatically.

Each compressed object, reconstructed chunk, and final file is hash-checked.
Unavailable objects fall back to the pinned Hugging Face files.

For historical context, the raw downloader measured 112 MB/s for a complete
install on a 1 Gbit/s datacenter link. A full `slotstream pull --verify`
measured 8 s on the development Mac. These measurements do not predict a
new user's download or verification time. See the [download measurements](../db/records/measurements/lossless-model-download-2026-09-05.md)
and [current hosting acceptance](../db/records/measurements/hugging-face-lossless-download-2026-09-06.md).

## Integrity and publication

`PinnedModel.swift` remains the authority for original paths, lengths, SHA-256
digests, and optional-file status. `PinnedTransportManifest.swift` embeds a
manifest and its SHA-256. Loading the manifest checks its own digest and exact
agreement with every original pin before any network request.

The current manifest is
`cf5461819d34044fcfaff7be230ea6c9ba595247a3f3018033c8dc85bc363a38`.
Its public prefix is:

```text
https://huggingface.co/carloslfu/Qwen3.8-Flash-Next-MLX-4bit-Slotpack/resolve/<pinned-revision>/slotpack/v1/<manifest-sha256>/
```

The manifest names original files and an ordered array of objects. Every
object records its compressed length and SHA-256, reconstructed length and
SHA-256, and one to three original file ranges. Ranges concatenate in that
order inside the decoded object. Complete, non-overlapping coverage of every
original file is mandatory. No object may mix required and optional files.

Objects live at `objects/<first-two-hash-characters>/<sha256>.bin`. Each is at
most 40 MiB plus a 32-byte header. `PinnedTransport.swift` selects the exact
Hugging Face commit, and the manifest pins every object's bytes independently
of its host. Hugging Face supplies large-file delivery and caching. The
compressed representation does not depend on a Cloudflare product.

A new model or representation gets a new immutable prefix. Do not overwrite
an existing package with different bytes. Upload every object, verify public
reads and a complete client reconstruction, then release the embedded pin.
Cache hits can improve the route and origin load; they do not reduce the
object's byte count. Server-requested throttle waits remain cancellable.

### Hosting cost and older releases

Public Hugging Face hosting avoids publisher charges per model download under
its current best-effort public-repository policy. Storage and rate limits still
apply; no paid plan or billing fallback is enabled by the downloader. See
[Hugging Face storage policy](https://huggingface.co/docs/hub/en/storage-limits)
and [request limits](https://huggingface.co/docs/hub/en/rate-limits).

The earlier `weights.sevra.page/slotpack/v1/` URLs are compatibility redirects
to the same package on Hugging Face. `Tools/slotpack/legacy-redirect/` deploys
only static assets and `_redirects`, with no Worker script, R2 binding, or
proxy. Cloudflare documents these static asset requests and storage as
[free and unlimited](https://developers.cloudflare.com/workers/static-assets/billing-and-limitations/).
Keep the hostname excluded from the application Worker's wildcard route.
New clients contact Hugging Face directly.

Upgrade older releases for the new server-throttle handling. An earlier client
on a very fast or shared connection may need to rerun its resumable pull after
a Hugging Face request-limit window resets.

Cloudflare analytics now describe only older clients' redirect traffic, not
the model bytes delivered by Hugging Face. Hugging Face's default model counter
counts selected query files; this client fetches hash-named objects and embeds
the manifest, so that counter does not measure compressed pulls or completed
installs. GitHub release-asset counts remain an acquisition proxy. No telemetry
is added. See [Hugging Face's counting rules](https://huggingface.co/docs/hub/en/models-download-stats).

Before retiring an old hosted copy, qualify a complete anonymous pull through
the new default against every original file hash, and verify that an earlier
released client follows the legacy redirects and reconstructs missing files.
Preserve historical publication evidence when retiring storage.

## Codec

`Sources/CSlotpack` implements the bounded codec in C, with no external codec
dependency. Its four byte-renormalized rANS states use the standard
[rANS equations](https://github.com/rygorous/ryg_rans). Encoding happens offline.

All wire integers are little endian. The fixed header is:

| Offset | Field |
|---|---|
| 0 | Eight bytes: `SLTPK001` |
| 8 | Reconstructed byte length, uint32 |
| 12 | Representation kind, uint32 |
| 16 | Packed weight byte length, uint32; zero except kind 3 |
| 20 | Quantization group size, uint32; zero except kind 3 |
| 24, 28 | Reserved uint32 values, both zero |

Kind 0 copies raw bytes. Kind 1 codes arbitrary bytes. Kind 2 codes exact
16-bit words. Kind 3 codes a concatenation of packed four-bit weights, BF16
scales, and BF16 biases. The encoder uses raw kind 0 when compression would
increase the framed size. Original safetensors headers and all small files
are included; no data is inferred from a different model version.

Generic sections begin with a uint32 count of used symbols, followed by
ascending sparse `(symbol delta, frequency)` pairs in canonical unsigned
base-128 varints. The first delta is from zero; subsequent deltas are from
the preceding symbol plus one. Frequencies sum exactly to 4096 for bytes or
65536 for 16-bit words. A uint32 stream length precedes four uint32 rANS
states and the renormalization bytes. The lower state bound is `1 << 23`;
decoded symbols alternate across the four states. The decoder requires exact
stream consumption and all terminal states equal to that bound.

For kind 3, the number of groups is twice the packed weight length divided
by the group size, which must be 32 or 64. Each group contributes two scale
bytes and two bias bytes. The encoder stores exact scales, a byte center
derived from round-to-nearest-even `-bias / scale` clamped to 0–255, and an
exact signed modular residual from the predicted BF16 bias. The prediction
rounds `-scale * center` to BF16 with integer arithmetic. Non-finite cases
use a defined zero prediction and retain their exact residuals. Signed
residuals are zigzag-encoded as 16-bit words; no precision is discarded.

Those three generic sections are followed by a uint32 count of quantization
contexts. Each context has a uint32 key and sixteen varint frequencies
summing to 4096. The key is
`center * 1024 + ((scale_bits & 32767) >> 5)`. At most 2048 distinct contexts
are permitted. The final rANS section codes low then high weight nibbles
using the context of their quantization group. Contexts, frequencies, lengths,
arithmetic bounds, terminal states, and all buffer boundaries are checked.

## Download lifecycle

The downloader uses a persistent URLSession per network worker. Automatic
mode starts with eight workers and tries sixteen, then thirty-two only while
measured throughput improves. Retries, a decoder backlog, or a throughput
plateau stop expansion. An explicit `--connections` value fixes the count.
Decoder concurrency, queued buffers, and completed operation lifetimes are bounded independently.

Every object is checked against its compressed SHA-256 before decoding.
Decoded bytes must match the reconstructed SHA-256 before they reach their
disjoint original ranges. Workers sync affected partial files before marking
an object complete in `.slotpack-state.json`. Final files replace
`*.slotpack.part` only after their original whole-file SHA-256 passes. A
damaged resume reconstructs the affected file's chunks and verifies again.

Retries do not advance verified progress twice. SIGINT/SIGTERM cancel active
requests, drain in-flight writes, and preserve completed chunks. A directory
lock prevents simultaneous writers. Partial files reject symlinks, hard
links, and non-regular files; valid final-file symlinks remain reusable.
Small resume metadata is bounded before allocation. Disk admission charges
the remaining reconstructed allocation plus a margin, not compressed size.

If a compressed object is missing or corrupt, the same original ranges are
retrieved from pinned Hugging Face sources and checked against the decoded
object digest. Signed redirects are cached in memory and refreshed after
expiry. An unavailable optional draft head may be skipped after its writers
drain; a required-file failure cannot report success.

`--transport raw`, explicit `PullOptions.sources`, or
`SLOTSTREAM_WEIGHTS_SOURCES` retain raw-file mirror compatibility. Automatic
mode also preserves an existing legacy `.part` download and its `.partmap`.
`--transport compressed` explicitly selects the new layout. Switching formats
reuses completed original files; partial progress belongs to its own format.

## Reproduce and qualify

Building and embedding require Python 3.9 or later and a C compiler;
Hugging Face publication additionally requires `huggingface_hub` (qualified
with version 1.29.0). Publication uses an existing `HF_TOKEN` or
`--token-stdin`, writes no credential, preserves existing repository files,
and resumes from the actual committed inventory. A conflicting object at an
existing immutable package path is rejected.

```sh
python3 Tools/slotpack/pack.py --model /path/to/original-model --output /path/to/package --workers 8
python3 Tools/slotpack/publish_hf.py --package /path/to/package --repo OWNER/MODEL --receipt /path/to/upload.json
python3 Tools/slotpack/embed.py --package /path/to/package
python3 Tools/slotpack/checks.py
```

Pin the resulting Hugging Face commit in `PinnedTransport.swift` and the
legacy redirect, then complete public qualification before release.
`publish_r2.py` remains a historical optional publisher; it is not the default.

The builder verifies every original file, every compressed round trip, and
complete coverage before writing its manifest and build receipt. The gates
exercise exhaustive finite BF16 prediction cases, randomized centers,
malformed/truncated frames under sanitizers, manifest corruption, real HTTP
fallbacks, cancellation/resume, damaged resumes, file safety, and legacy raw
multi-chunk downloads. `codec_fuzz.c` additionally supports libFuzzer with
AddressSanitizer and UndefinedBehaviorSanitizer on a toolchain providing it.

Use `full_pull.py` for complete loopback or public-CDN qualification. A CLI
run with `--binary /path/to/slotstream --base-url PUBLIC_PREFIX
--default-source` clears transport overrides and tests the actual default.
Its receipt records the binary and source hashes, fresh/resumed status,
CDN/connection observations, zero required raw fallback, and independently
computed SHA-256 hashes of every installed original file. A full public
default qualification must pass before releasing a new package pin.

`Tools/coverage.sh --lcov FILE` also instruments the real manifest, HTTP,
legacy raw, and sustained-memory fixtures. It unions their line hits with
the catalogue, mapping immutable source snapshots back to their exact source
files. Transport files use the union of physical source-line hits; unrelated
files retain their original LLVM summaries. Function and branch denominators
from separate executables are not combined.


<!-- ===== docs/HARDWARE.md ===== -->

<a id="measured-on-real-macs"></a>

# Hardware and speed

## What you need

- An Apple Silicon Mac, with macOS 14 or later.
- About 110 GB of free SSD space for the model.

Choose Apple menu → About This Mac to check your chip and memory. The
installer has been tested on macOS 14 and 15; model runs have been tested
on macOS 26. Windows, Linux, and Intel Macs are not supported by this engine.

Slotstream is built for Macs with 16 to 64 GB of memory, where the model
cannot fit. It runs on 96 GB and larger Macs too, where the model fits in
memory, but it is not optimized for them; see
[Who it's for](../README.md#who-its-for).

**Compatibility-tier support is coming soon.** An 8 GB Mac can't run the
current model: even the smallest memory plan needs more
memory than it has, so Slotstream refuses to start instead of swapping. On
other Macs, close memory-heavy apps before running the model.

To check your own Mac without downloading or loading anything, run:

```sh
slotstream doctor
```

## Understanding speed

A **token** is a small piece of text, often part of a word. `tok/s` means
tokens per second. The speeds below describe a reply after the model's
cache has warmed up. Prompt-processing measurements appear separately below.
The first reply also needs time to load the model and
process your question. Long conversations take longer to process.

Your chip, SSD, and other running apps affect speed. A memory size alone
isn't enough to predict it.

<a id="rows"></a>

## Results

These reply-generation results were measured on real Macs, using different
releases and settings. The leading result remains the latest qualified warm
decode benchmark. The [0.2.23 calibration attempt](../db/records/measurements/release-speed-calibration-2026-09-22.md)
has not yet qualified a replacement full-answer baseline:

| Mac | Memory | Reply speed |
|---|---|---|
| **MacBook Pro, M5 Pro (our development Mac), 0.2.19 at a 22 GB target** | **48 GB** | **15.86 tok/s** |
| Same M5 Pro, 0.2.16 configuration at a 20 GB target | 48 GB | 13.47 tok/s |
| Same M5 Pro, historical 0.2.3 result | 48 GB | ~12 tok/s |
| Mac mini, M2 (base storage) | 16 GB | 1.41 tok/s |
| MacBook Pro, M4 Pro | 24 GB | 5.41 tok/s |
| MacBook Air, M5 | 32 GB | 6.22 tok/s |
| MacBook Pro, M4 Max | 36 GB | 8.41 tok/s |
| MacBook Pro, M3 Max | 64 GB | 12.38 tok/s |
| MacBook Pro, M4 Max | 64 GB | 15.93 tok/s |
| Same M4 Max, model on a 10 Gb/s external SSD | 64 GB | 2.98 tok/s |
| MacBook Pro, M5 Max, 0.2.3, auto (34.6 GB target) | 128 GB | ~21–22 tok/s |
| Same M5 Max, 0.2.3, 48 GB target | 128 GB | ~26.9 tok/s |
| Same M5 Max, 0.2.3, 73 GB target | 128 GB | ~31.5 tok/s |

The historical leading M5 Pro result is the 0.2.19 release benchmark of the
shipping forecast against the 0.2.18 forecast (1.10x faster, 14.38 to 15.86 tok/s,
identical output). Both arms used smaller prompt passes and disabled prefix
caching, so more of the same budget held experts. These are controlled
benchmark settings, not today's automatic configuration; the 0.2.16 row is the pre-release benchmark of that release's
configuration. The M5 Pro results are from the author; the
others are community reports. The M5 Max rows are outside the target range:
the model fits in memory on that Mac, and engines that keep it resident
report faster replies there.
The 18 GB size still needs reports, and 8 GB Macs don't run the
model. Open the details below for versions, settings, and credits.

<details>
<summary>Full results and test conditions</summary>

| Mac | Memory | SSD | macOS | slotstream | Plan | Warm decode | Long prompt | Reported memory | Reported by |
|---|---|---|---|---|---|---|---|---|---|
| MacBook Pro, M5 Pro | 48 GB | internal, 2 TB | 26.6.2 | 0.2.19 | 22 GB target, two drafts, corrected decode forecast, ~100 experts/layer | 14.38 to 15.86 tok/s with the corrected forecast, arm medians over counted cells from 24 held-out pairs | not measured | not recorded | [@carloslfu](https://github.com/carloslfu), 2026-09-16 |
| MacBook Pro, M5 Pro | 48 GB | internal, 2 TB | 26.6.2 | 0.2.16 candidate | 20 GB target, two drafts, decode lookahead, ~88 experts/layer | 11.79 to 13.47 tok/s with the lookahead, arm medians over eligible runs from 34 held-out pairs | not measured | not recorded | [@carloslfu](https://github.com/carloslfu), 2026-09-13 |
| MacBook Pro, M5 Pro | 48 GB | internal, 2 TB | 26.6 | 0.2.3 | auto: 33 GB target, ~152 experts/layer | ~12 tok/s; 12.8 with `--mtp` at a 28 GB memory target | ~220 tok/s at a 4096-token pass (est.) | 32 GB (estimate) | [@carloslfu](https://github.com/carloslfu), 2026-09-02 |
| Mac mini, M2 | 16 GB | internal, 256 GB | 26.6.2 | 0.2.2 | auto: 10.2 GB target, ~21 experts/layer | **1.41 tok/s** | not measured; `context-check` postdates 0.2.2 | 6.1 GB | [@flol's report](https://github.com/carloslfu/slotstream/issues/5), 2026-09-02 |
| Same Mac mini, M2 | 16 GB | internal, 256 GB | 26.6.2 | 0.2.3 | auto: 10.7 GB target, ~25 experts/layer | **1.48 tok/s** | 11 tok/s for 8192 tokens (12.1 min) | 8.1 GB RSS on the long prompt | [@flol's re-run](https://github.com/carloslfu/slotstream/issues/5#issuecomment-5525389653), 2026-09-03 |
| MacBook Pro, M4 Pro | 24 GB | 512 GB; location not specified | 26.6.2 | 0.2.24 | auto: 15.9 GB target, ~53 experts/layer; no draft head or lookahead at this size before 0.2.25 | **3.57 tok/s**; 3.85 to 3.97 in a later round | 93 tok/s for 8192 tokens (18.0 GB target) | 16.6 GB process peak on the long prompt | [@davidcavazos's report](https://github.com/carloslfu/slotstream/issues/41), 2026-09-25 |
| Same M4 Pro | 24 GB | 512 GB; location not specified | 26.6.2 | 0.2.25 | auto: 17.4 GB target, ~58 experts/layer; draft head and decode lookahead on | **5.41 tok/s**; 5.58 and 4.92 in the same round | 86 tok/s for 8192 tokens (18.0 GB target) | 16.7 GB process peak on the long prompt | [@davidcavazos's re-run](https://github.com/carloslfu/slotstream/issues/41#issuecomment-5850068364), 2026-09-26 |
| MacBook Air, M5 | 32 GB | 1 TB; location not specified | 26.6.2 | 0.2.11 | 22 GB target, ~75 experts/layer planned | **6.22 tok/s** | 126.28 tok/s for 8192 tokens, 2048-token passes | 17.75 GB RSS on the long prompt | [@arczhi's report](https://github.com/carloslfu/slotstream/issues/12), 2026-09-07 |
| MacBook Pro, M4 Max | 36 GB | internal, 1 TB | 26.0.1 | 0.2.22 | auto: 27.1 GB target, ~90 experts/layer | **8.41 tok/s** | 166 tok/s for 8192 tokens (27.1 GB target) | 24.8 GB process peak on the long prompt | [@JohnClarkson's report](https://github.com/carloslfu/slotstream/issues/26), 2026-09-20 |
| MacBook Pro 14", M3 Max | 64 GB | 512 GB; location not specified | 27.0 | 0.2.18 | auto: 48.1 GB target, ~119 experts/layer | **12.38 tok/s** | 213 tok/s for 8192 tokens (34.6 GB target) | 30.1 GB process peak on the long prompt | [@merken's report](https://github.com/carloslfu/slotstream/issues/20), 2026-09-16 |
| MacBook Pro 16", M4 Max | 64 GB | internal, 1 TB | 27.0 | 0.2.22 | auto: 48.1 GB target, ~119 experts/layer | **15.93 tok/s** | 270 tok/s for 8192 tokens (34.6 GB target) | 30.1 GB process peak on the long prompt | [@YenHub's report](https://github.com/carloslfu/slotstream/issues/22), 2026-09-19 |
| Same M4 Max, external SSD | 64 GB | external, 1 TB, USB 3.2 Gen 2 (10 Gb/s) | 27.0 | 0.2.22 | auto: 48.1 GB target, ~119 experts/layer | **2.98 tok/s** | 51 tok/s for 8192 tokens (34.6 GB target) | 30.2 GB process peak on the long prompt | [@YenHub's report](https://github.com/carloslfu/slotstream/issues/23), 2026-09-19 |
| MacBook Pro 16", M5 Max | 128 GB | internal, 2 TB | 26.6.2 | 0.2.3 | auto: 34.6 GB target, ~152 experts/layer | ~21–22 tok/s with speculative decoding | not measured | not measured; server path only | [@waterliu1981's update](https://github.com/carloslfu/slotstream/issues/6#issuecomment-5520489176), 2026-09-03 |
| Same M5 Max | 128 GB | internal, 2 TB | 26.6.2 | 0.2.3 | manual: 48 GB target, ~253 experts/layer | ~26.9 tok/s with speculative decoding | not measured | not measured | same report |
| Same M5 Max | 128 GB | internal, 2 TB | 26.6.2 | 0.2.3 | manual: 73 GB target, ~401–441 experts/layer as reported | ~31.5 tok/s with speculative decoding | not measured | not measured | same report |

The historical memory values retain their original measurement limits. The
M5 Pro figure is a planner estimate, and older reported values do not establish
the kernel lifetime footprint peak added by the reporting correction. These
hardware configurations have not been requalified with the new counter.

The 16 GB M2, 24 GB M4 Pro and 32 GB M5 Air results are below the planner's
estimates, the M4 Pro's 5.41 tok/s on 0.2.25 against its ~8, and the 36 GB
M4 Max's 8.41 tok/s is just under its ~9. The 64 GB M3 Max and
the 64 GB M4 Max on its internal SSD are above their ~11 estimate, and the
128 GB M5 Max result is above its estimate. The planner uses the M5 Pro curve
and doesn't model these differences.

The 64 GB M4 Max separates disk speed from everything else: with the same
plan and release, it decoded 15.93 tok/s from its internal SSD and 2.98 tok/s
from a 10 Gb/s USB drive that read 0.9 GB/s. A 16 GB Mac with a fast SSD would
still help separate disk speed from memory capacity at the small end.

The three 0.2.22 reports ran without the decode-forecast file added in 0.2.19.
Their logs print `no correction at lookahead/tap-correction-attention-rank128-v1.safetensors`.
`slotstream pull` fetches the 37.5 MB file; through 0.2.24, the download
`slotstream run`, `serve` and `launch` offer on first use did not. The 24 GB
M4 Pro's 0.2.25 re-run had it.

The Air's long-prompt test explicitly used a 22 GB target with vision and
speculative decoding off. Its full warm-server command and system load were
not supplied. The M5 Max row uses the reporter's updated results after
moving from 0.2.1 to 0.2.3. Community results have not been independently
rerun by the author.

Full methods, raw reports, and limits are in [MEASUREMENTS.md](../MEASUREMENTS.md):
the M5 Pro throughout, the M2 in C1, the M5 Max in C2, the M5 Air in C3, the
M3 Max in C4, the 64 GB M4 Max in C5, the 36 GB M4 Max in C6 and the 24 GB
M4 Pro in C7.

### What the columns mean

- **Plan**: the target and cache size `slotstream doctor` prints with nothing
  else running. Auto sizes down while other apps hold memory, so say what was
  open.
- **Warm decode**: tokens per second on the third identical request to a
  running server, once the expert cache has warmed up. The first generation
  in a fresh process is colder and slower; report it too.
- **Long prompt**: prefill tokens per second from `context-check`, which
  reads a synthetic prompt through the real engine with process-budget and
  real-headroom safeguards. Keep paging observations with any timing result.
- **Peak**: the process-memory bound reported by `run` and `context-check`,
  combining native lifetime physical-footprint and RSS peaks with current
  usage; request samples are separate observations. This is measured separately
  from the plan's estimate.

</details>

## Recent prompt-processing results

Changes shipped in 0.2.23 have separate prompt-processing measurements on the
48 GB M5 Pro at a 10 GB target. Each row has three clean matched pairs and
identical generated token IDs within every pair. Times are medians within
each arm; the percentage is the median of paired reductions, so it need not
equal the percentage calculated from the two displayed medians.

| Workload | Matched control | Median times, control → enabled | Median paired time reduction |
|---|---|---|---|
| 16K inventory prompt, MTP on | Larger-read workspace policy off | 155.22 s → 53.94 s prefill | 65.40% |
| 2K prose follow-up, MTP off | Prefix checkpoints disabled | 30.73 s → 4.42 s request | 85.62% |

The inventory fixture reads 16,387 synthetic tokens with two MTP drafts and
emits 11 tokens before its stop token. Both arms use fused attention; the
control disables fused-workspace accounting. It qualifies the automatic
read policy before release, not the complete change from the prior release.
See the [policy qualification](../db/records/measurements/mtp-prefill-policy-2026-09-21.md).

The prose fixture tests the installed release. It reads 2,090 tokens on its
first request, then 2,092 on the follow-up, reusing 2,048 and emitting the
same 16 capped output tokens in both arms. The table measures only the
follow-up. Only one complete two-request pair passes the timing gates, so the
percentage is not a repeated full-session result. Earlier releases already
had prefix caching; this is its benefit against disabled checkpoints, not an
incremental release gain.

The [published-release audit](../db/records/measurements/published-prompt-speed-audit-2026-09-22.md)
also covers other prompt types, lengths, budgets, MTP settings and disk reuse.
Short requests show no consistent speedup. Paging-affected long-request
comparisons stay excluded from qualified timing claims; the larger-memory
release comparison and kernel-only attribution have too few clean pairs for
a repeated claim. None of these percentages updates the warm reply-speed
ranges or establishes a speedup on another Mac.

### Fresh installed-release first reads and exact repeats

The same development Mac and memory target were measured with ordinary
caching and planner-owned settings. The table separates prompt processing
from the full repeated request, which includes a capped reply:

| Prompt | Eligible first reads / repeats | First-read prefill range | Median repeated request |
|---|---:|---:|---:|
| 2K code | 4 / 3 | 13.86–28.11 s | 2.79 s |
| 2K prose | 3 / 3 | 14.91–26.51 s | 3.22 s |

The prospective desktop load and process-page-in screen passed for the
included observations. Several runs had system swap-ins; the stricter global
no-swap subset is insufficient for a repeated first-read claim. Request
history changed read batching despite an unchanged memory plan. The linked
[measurement](../db/records/measurements/release-prefill-2k-2026-09-22.md)
keeps server-first prompts and later misses separate and preserves every
excluded request. These observations do not establish a new decode headline,
a general ETA correction, or a whole-release speedup.

## Does more memory help?

Within Slotstream, yes: a larger expert cache reduces SSD reads and improves
reply speed. From 96 GB the model fits in memory; Slotstream runs there and
benefits from a larger cache, but that is not the case it is optimized for,
and engines that keep the model resident report faster replies there. See
[Who it's for](../README.md#who-its-for).
The clearest community evidence is
[@waterliu1981's cache sweep](https://github.com/carloslfu/slotstream/issues/6#issuecomment-5520489176)
on the same M5 Max, using Slotstream 0.2.3 with speculative decoding enabled:

| Total-process memory target | Reported warm reply speed |
|---|---|
| 34.6 GB (auto) | ~21–22 tok/s |
| 48 GB (manual) | ~26.9 tok/s |
| 73 GB (manual) | ~31.5 tok/s |

All three runs used the same Mac with 128 GB installed memory. The targets
are decimal GB budgets, not measured process peaks or installed-memory
requirements. This comparison supports a gain from allocating more memory
on that machine; comparing its auto result with the M5 Pro alone would not
isolate the effect of memory.

The manual rows are the reporter's warm-speed summaries, without the repeated
per-run timings supplied for auto. They have not been independently rerun or
remeasured on 0.2.16. The report does not establish a universal scaling curve,
a best automatic target, or a larger qualified context window. See
[memory defaults and overrides](../README.md#why-doesnt-slotstream-use-all-of-my-ram)
to try a larger target while leaving room for macOS and other apps.

## Speed estimates

### Planning ranges

The README's estimates combine the real reports above with the development
Mac's measured configurations and planner curve. They are rough expectations
across hardware and settings, not a fitted scaling model or statistical
confidence intervals. Endpoints are rounded outward to whole tok/s.

| Installed RAM | Estimated warm reply speed | Basis and main inference |
|---|---|---|
| 16–<24 GB | ~1–6 tok/s | The M2 mini reported 1.41 tok/s; the M5 Pro-based 16/18 GB simulations estimate about 3.5 to 5 tok/s before the decode lookahead's gain. The upper end has not been measured on a real Mac in this band. |
| 24–<48 GB | ~5–16 tok/s | The 24 GB M4 Pro reported 5.41 tok/s on 0.2.25, rounded outward to 5; the M5 Pro measured 15.86 tok/s on 0.2.19 at a 22 GB process target, rounded outward to 16. That benchmark used different prompt-workspace and cache settings from today's automatic plan, and the upper end assumes a comparable chip and SSD. The 36 GB M4 Max reported 8.41 tok/s on 0.2.22 and the 32 GB M5 Air 6.22 tok/s on 0.2.11. The M4 Pro's first report, 3.57 tok/s on 0.2.24, ran before 0.2.25 turned on its draft head and decode lookahead; its SSD read cold experts through the engine at 3.7 GB/s, about a third of the development Mac's rate, and other apps held memory during both runs. |
| 48–<96 GB | ~15–27 tok/s | The lower reference rounds down from the 48 GB M5 Pro's 15.86 tok/s on 0.2.19 at a 22 GB target, below its own 33.6 GB automatic target, whose larger cache has not been timed; the 0.2.16 result at a 20 GB target was 13.47 tok/s, and the older ~12 tok/s result remains historical evidence. A 64 GB M4 Max reported 15.93 tok/s on 0.2.22. A 64 GB M3 Max reported 12.38 tok/s on 0.2.18, below this range; a rerun on the current release is pending. The upper end transfers the M5 Max's 26.9 tok/s at a 48 GB process target to a comparable Mac with enough available memory. That run used a 128 GB Mac; it was not a measurement of a 48 GB Mac. |
| 96 GB+ | ~20–32 tok/s | The 128 GB M5 Max reported about 21 to 22 tok/s in auto and 31.5 tok/s at a 73 GB process target. Applying this range to other Macs in the band is an estimate. This row is outside Slotstream's target range: the model fits in memory from 96 GB. |

The 96 GB+ row's lower endpoint allows for the same reporter's roughly 20 tok/s
warm auto runs on 0.2.1; the main results table uses the updated 0.2.3 report.
The upper ends of High and the 96 GB+ row assume an M5 Max-class chip, fast
internal SSD, speculative decoding and manual targets that leave room for macOS and
other apps. A 48 GB process target cannot consume all of a Mac's installed
48 GB; it needs a larger machine. These ranges mix releases, so they are not
predictions for a single current build. No release-speedup multiplier was
applied to community reports.

A slow SSD, older chip, different prompt, draft acceptance or memory pressure
can produce results outside the ranges. The 64 GB M4 Max decoded 2.98 tok/s
with the model on a 10 Gb/s external drive, far below its band. More RAM
helps only when the engine can use it to reduce a bottleneck; the band labels do not establish a causal
speed ranking. In particular, there is no measured performance boundary at
96 GB. The shared context recommendation reflects the current planning
guidance, independently of reply speed.

### Automatic memory plans

The columns were checked against a build of `main` after 0.2.24, which
streams the draft head's experts on smaller caches and runs the decode
lookahead in plain decode. They describe the plans in auto mode, which picks the
context window along with the target and speculative decoding. The draft file
is available and no other apps hold memory. Simulated RAM is in decimal GB; a
Mac's marketed memory capacity can produce a different decimal-GB device
reading and target. Auto picks 32,768 tokens through 32 GB of simulated RAM,
65,536 at 36 GB, 32,768 at 48 GB, 131,072 at 64 GB and 262,144 from 96 GB.

| Simulated RAM (decimal GB) | Automatic memory target | Speculative decoding | Decode lookahead | Automatic context window |
|---|---|---|---|---|
| 8 GB | No plan fits | Not applicable | Not applicable | Not applicable |
| 16 GB | 10 GB | Off | On | 32,768 |
| 18 GB | 11.5 GB | Off | On | 32,768 |
| 24 GB | 16 GB | On, experts streamed | On | 32,768 |
| 32 GB | 22 GB | On | On | 32,768 |
| 36 GB | 25 GB | On | On | 65,536 |
| 48 GB | 33.6 GB | On | On | 32,768 |
| 64 GB | 43.2 GB | On | On | 131,072 |
| 96 or 128 GB | 54.7 GB | On | On | 262,144 |

These are allocation plans, not measured performance tiers. The planner's
M5 Pro-based warm-decode estimates for the rows without speculative decoding
are ~3.5 tok/s at 16 GB and ~5 tok/s at 18 GB of simulated RAM. They leave
out the decode lookahead, which made plain decode 1.05x to 1.11x faster on
the development Mac. At 24 GB the draft head now runs with its experts
streamed. The historical 22 GB benchmark measured 15.86 tok/s on 0.2.19 with two
drafts at about 100 experts per layer (14.38 with the 0.2.18 forecast).
Although its total budget matches the 32 GB simulation, its smaller prompt
passes and disabled prefix cache leave a different expert pool. It does not
measure the current automatic plan. The earlier estimate of about 10 tok/s
came from a two-draft measurement at 76 experts per layer on 0.2.14; the real
M5 Air result above was slower. A shared memory budget does not establish
matching runtime settings or speed.

The 15.86 tok/s result with the corrected forecast at about 100 experts per
layer, and the 13.47 tok/s result of 0.2.16 at about 88, are measured references,
not predictions for larger caches. Larger caches have not been timed with
0.2.19 yet; the M5 Max sweep above
demonstrates gains beyond auto with an earlier release. Do not apply the
development Mac's release speedup to those community figures.

**Auto mode picks the memory target, cache size, speculative decoding and
context window.** It takes the largest window of 32,768, 65,536, 131,072 or
262,144 tokens that keeps speculative decoding, the draft head's resident
experts and the decode lookahead as the 32,768-token plan has them, keeps
one complete conversation of that length for follow-up turns, and adds at
most 10% to the planner's estimate for a typical request of 2,000 prompt
tokens and a 400-token reply. `slotstream doctor
--sim-ram <GB>` shows every candidate and its reason. At 24 GB a 65,536-token
window would add 22%, and at 32 GB it would stream the draft head's experts,
whose reads the estimate does not price.
At 36 GB it adds 9%, as the cache drops from 96 to 75 experts per layer.
At 48 GB, auto keeps the original cache because its size exceeds the measured
decode range: the estimate cannot price the loss, even when it reports little
or no change. At 64 GB, 262,144 would add 18%.
From 64 GB the larger window's memory comes from room the 32,768-token plan
leaves unused, so the cache keeps its size and the target rises above that
plan's 34.6 GB, to 43.2 GB at 64 GB and 54.7 GB from 96 GB. Real available
memory and Metal limits can change these decisions; a busy start applies the
same cache and speed rules before choosing its window. A larger Mac can
therefore receive a smaller window when widening it would sacrifice cache
whose performance benefit is unmeasured. `--max-context N` fixes any
window up to 262,144; on a 32 GB Mac, `--max-context 65536` gives the larger
window with the draft head's experts streamed.

For prompts near 32,768 tokens, the planner estimates about 3 minutes of
prefill from 24 GB and 6.4 minutes at 16 GB; near 65,536 it estimates
about 9 minutes at 24 GB, where the draft head's experts stream and the
prefill pass is smaller, and about 8 minutes from 32 GB. These estimates use the M5 Pro's prefill curve,
not measurements on those memory sizes. The planner's historical prefill
curve has not been recalibrated for the new read policy; the bounded results
above cannot supply a multiplier for every pass size, prompt and context.
Windows above 128,256 tokens have no
calibrated estimate yet. On the development Mac, a full 131,072-token prompt
took 38 minutes to read at a 16 GB target, and its passes slowed as the prompt
grew: the planner's estimates, which ignore position, came within a few
percent of the measured time at 65,536 tokens and fell about a third short of
it past that. Leave room for the reply in the
configured window. Startup, queueing, images and reasoning before visible
answer text add to the user's wait.

The repeated target from 96 GB is the intentional conservative default: the
33 GB base ceiling plus the draft head and the full window's charge. Auto does
not increase its ceiling for the M5 Max's demonstrated larger-cache gains. These simulated plans
describe allocation policy; they do not measure speed. See
[memory defaults and overrides](../README.md#why-doesnt-slotstream-use-all-of-my-ram).

## How to measure

Context is a startup choice. Since 0.2.17 auto picks it for each Mac and
`--max-context` fixes it; 0.2.14 added the feasibility report and request-wait
controls described here.
Use `doctor --json` with the intended `--max-context` and memory policy to inspect the feasible
window before loading. A memory-feasible window does not promise a short wait:
the request-to-first-token budget defaults to 30 minutes, including preparation
and queueing. Setting `--max-prefill-wait 0` disables only that time policy.

Keep the configured window, prompt count and required reply count with each
result. Capacity evidence needs a complete prompt and reply and process memory
within budget, including sampled footprint and lifetime peaks. Record global
paging separately; it does not identify which application caused it. Report MTP and vision separately;
a text-only capacity result does not qualify those modes or answer quality.

Allow about ten minutes once the weights are downloaded. Close other
memory-heavy apps if you want clean speed measurements, and exclude timing
intervals affected by paging. Functional checks can run with other apps open
when the memory and pressure safeguards permit it. Run one model
process at a time.

To share your Mac's results, follow the [measurement steps](TESTING.md#measure-your-mac),
then open a [measurement report](https://github.com/carloslfu/slotstream/issues/new?template=measurement-report.yml).
Allow about ten minutes once the model is downloaded. Reports are credited
to their authors.

The full [engineering notes](ENGINEERING.md#speed) explain prompt-processing
time, memory use, and the methods behind the performance claims.


<!-- ===== CHANGELOG.md ===== -->

# Changelog

What each release changed, newest first. `curl | sh` installs the latest
release; anything under **Unreleased** is on `main` only.
Version headings can be prepared before publication. The
[Releases page](https://github.com/carloslfu/slotstream/releases/latest)
determines which version the installer downloads.

## 0.2.26 - 2026-09-27

- The prefix cache's disk tier now writes states from 1,024 tokens instead
  of 2,048 (`--prefix-cache-min-tokens`). On 0.2.18, after a restart, a
  1,919-token conversation re-read its whole prompt in 45.5 s under the old
  default and resumed with 6.0 s of prefill under the new one. Since 0.2.22
  a restart resumes at the previous prompt's last prefill pass boundary, so
  it also re-reads the previous reply and up to one pass of that prompt. The
  servers `slotstream launch` starts and the development Mac app use this
  default, so Pi's opening prompt of about 1,600 tokens now reaches the
  disk. Each turn of a conversation between the two lengths now writes its
  state, about 120 to 170 MB, within the same quota. Measured by
  [@jasen215](https://github.com/jasen215) in
  [#17](https://github.com/carloslfu/slotstream/issues/17).
- A continued conversation writes one state per turn again. Since 0.2.22 the
  end of the previous reply also counted as a prefix other conversations
  share once the reply crossed a prefill pass boundary. When the new message
  crossed another boundary, that point was written as a second state;
  otherwise 0.2.25's fix for colliding shared prefixes rewrote the turn's
  own state as shared, which made it the first state removed when the quota
  needed room. Each of these writes added about 116 MB, kept until the quota
  removed it. Only the end of a system prompt, or the point where a prompt
  parts from another kept prompt, is saved as a shared prefix now.
  `Tools/prefix_turn_writes_e2e.py` checks it live.
- `slotstream parity --compare` refuses a NaN or infinite value instead of
  printing `PARITY PASS`. Swift's `max` drops a NaN, so such a dump could
  pass. By [@Pybsama](https://github.com/Pybsama) in
  [#31](https://github.com/carloslfu/slotstream/pull/31).
- `/v1/responses` accepts its own output replayed after parallel tool calls.
  The model writes a newline between calls, which streams as a message item
  between them, and a client replaying that output, as Codex does, got a
  400. By [@Pybsama](https://github.com/Pybsama) in
  [#32](https://github.com/carloslfu/slotstream/pull/32).
- Raw downloads (`--transport raw`, source overrides and resumed legacy
  downloads) honor a server's `Retry-After` and `RateLimit` headers, as
  compressed downloads have since 0.2.11, instead of retrying after 2 to 8
  seconds. Like compressed downloads, they wait 5 minutes after a 429 that
  carries neither header, and each wait is capped at 10 minutes. By
  [@Pybsama](https://github.com/Pybsama) in
  [#33](https://github.com/carloslfu/slotstream/pull/33).
- `slotstream prefix-cache --clear` names every file it could not remove and
  exits with an error, instead of reporting only the files it removed. By
  [@Pybsama](https://github.com/Pybsama) in
  [#34](https://github.com/carloslfu/slotstream/pull/34).
- A tool argument declared through a local `$ref` in its tool's schema keeps
  the declared type on `/v1/chat/completions`, `/v1/responses`,
  `/v1/messages` and the AI SDK gateway. A string argument such as `00123`
  reached the client as the number `123`, and the text `false` as a Boolean,
  although the same declaration written inline kept the string; a referenced
  decimal could arrive as a string. By
  [@Pybsama](https://github.com/Pybsama) in
  [#35](https://github.com/carloslfu/slotstream/pull/35).
- Control+C while `slotstream pull`, or the download `run`, `serve` and
  `launch` offer on first use, fetches the decode-forecast file now stops
  the command with exit code 130. The fetch caught the cancellation, so
  `pull` reported ready and the others went on to load the model. By
  [@Pybsama](https://github.com/Pybsama) in
  [#36](https://github.com/carloslfu/slotstream/pull/36).
- Opening the prefix cache directory keeps a file the system refuses to
  read, because of its permissions or an I/O error, instead of deleting it
  and the states that depend on it; damaged files are still removed.
  `PersistentPrefixCache(configuration:identity:)` and
  `Engine.enablePersistentPrefixCache(_:)` then throw
  `PersistentPrefixCache.InaccessibleFile`, which names the file. `serve`
  runs without the disk cache and says so, and the development Mac app runs
  without it until the file can be read. By
  [@Pybsama](https://github.com/Pybsama) in
  [#37](https://github.com/carloslfu/slotstream/pull/37).
- The AI SDK gateway matches tool results to their calls by `toolCallId` and
  puts them back in call order, as `/v1/chat/completions` does. The model's
  template pairs results with calls by position, so results sent in another
  order, as direct gateway requests and stored histories can, reached the
  model paired with the wrong calls. A missing, duplicate or unmatched result
  or a mismatched tool name is refused; a call's ID may recur in a later
  turn. The gateway also accepts an inline image in the `{type: "data"}` form
  the published `@ai-sdk/gateway` package sends, which it refused. By
  [@Pybsama](https://github.com/Pybsama) in
  [#38](https://github.com/carloslfu/slotstream/pull/38).
- `/v1/messages` joins consecutive messages of one role into one turn, as
  the Messages API does. Tool results split across adjacent user messages
  are accepted, and a turn's tool calls may continue in the next assistant
  message. Adjacent plain user messages, which rendered as separate turns,
  now render as one, with the turn's pictures before its text as in a single
  message. By [@Pybsama](https://github.com/Pybsama) in
  [#40](https://github.com/carloslfu/slotstream/pull/40).
- The server answers 431 for any request whose headers exceed 64 KiB.
  Headers of up to 128 KiB were served when their closing blank line arrived
  in the read that crossed the limit. By
  [@Pybsama](https://github.com/Pybsama) in
  [#42](https://github.com/carloslfu/slotstream/pull/42).
- A prefix cache file whose header gives an impossible array length or row
  range is removed as damaged when the directory opens, instead of crashing
  the server, the development Mac app or `slotstream prefix-cache` every time
  they open it. By [@Pybsama](https://github.com/Pybsama) in
  [#43](https://github.com/carloslfu/slotstream/pull/43).
- `slotstream launch` exits with code 130, as `run` and `serve` do, when
  Control+C interrupts the download it offers on first use; it reported a
  generic failure. On `/v1/messages`, an oversized request header is typed
  `request_too_large` instead of `api_error`, and the missing-result error
  lists only the calls still unanswered. When `serve` runs without its disk
  cache, the `prefix-cache --clear` command it suggests names that
  directory; without `--dir` it cleared launch's directory instead.
  Restoring a state from disk keeps a file the system refuses to open, as
  opening the directory does, instead of deleting it. A state rewritten for
  another draft mode or prefill pass size keeps its shared flag and the
  conversation ids recorded with it. The development Mac app's setup says
  when the decode-forecast file is missing.

## 0.2.25 - 2026-09-24

- The GPU stays awake while a request generates. Streamed decode leaves the
  GPU idle between short bursts of work, and an idle GPU lowers its clock and
  starts the next burst late; a one-thread kernel on its own command queue now
  keeps it busy, with outputs unchanged. It costs power, so
  `--gpu-keepalive auto`, the default, runs it only on AC power outside Low
  Power Mode; `on` and `off` override it, and `SLOTSTREAM_GPU_KEEPALIVE` sets
  the default. Saved statistics report `gpuKeptAwake`.
- Cache misses are read into host memory and copied straight into their cache
  slots instead of passing through staging arrays and a GPU scatter.
  `SLOTSTREAM_OPT_DIRECT_DEMAND=0` restores the previous path.
- Together, on the development Mac with identical output, the two made decode
  1.28x faster at a 10 GB target without the draft head and 1.22x faster at
  22 GB with it. `decode-overlap-check` compares both against the previous
  paths on a cold cache and covers the direct reads' failure recovery.
- The draft head can stream its experts. On a cache below 76 experts per
  layer after the head's full 1.6 GB charge, its 512 experts now stay on the
  SSD and pass through a 64-expert cache of their own, charged 0.4 GB, so the
  main cache keeps the other 1.2 GB; the output is the same. Automatic mode
  turns the head on from 28 experts per layer instead of 76, a 12 GB target,
  so 24 GB Macs now run speculative decoding. At a 12 GB target the head
  decoded 1.23x faster than plain decode with the lookahead.
  `SLOTSTREAM_MTP_EXPERTS=resident|streamed` forces a placement and
  `doctor --json` reports `mtp_streamed_experts`. The automatic context window
  never trades a resident head for a streamed one, so the draft-head tiers
  keep their windows; `--max-context 65536` on a 32 GB Mac now keeps
  speculative decoding with the head's experts streamed, and `--mtp on` fits
  from an 8.5 GB target.
- Without the draft head, the decode lookahead now runs in plain decode from
  20 experts per layer before its charge, including `--mtp off` and installs
  without the head's file. It made plain decode 1.11x faster at a 10 GB
  target with identical output. `SLOTSTREAM_OPT_EXPERT_PREFETCH=0` turns it
  off. Its 373 MiB moves a 48 GB Mac without the head from 152 to 149 experts
  per layer, inside the measured decode range, so that Mac's automatic window
  becomes 131,072 tokens.
- `draft-stream-check` compares a streamed head with a resident one and plain
  decode with and without the lookahead, and injects a failed draft read.

- The development Mac app attaches folders without a whole-tree scan or a
  file-count cap. Live directory browsing, filename search and scoped content
  search discover new and renamed files. Bounded continuations retain matches
  within a file, detect directory changes and expire under resource pressure.
  Reads can use a returned ID or a relative path inside the selected folder.
- Edits require a fresh read after an external file change. Attachment errors
  no longer offer the unrelated Home-reconciliation action, and the assistant
  is told how local folder access is granted.

- A shared system prompt kept on disk now survives when its save lands on
  the conversation's own checkpoint, which happens when the system prompt
  and the end of the prompt fall in the same prefill pass. The checkpoint
  was written first, the shared save found it and stopped, and a later turn
  removed it, so other conversations and restarts read the system prompt
  again. The head is now upgraded to shared, and a shared save that is
  skipped or fails is logged. Found and fixed by
  [@jasen215](https://github.com/jasen215) in
  [#27](https://github.com/carloslfu/slotstream/pull/27), for
  [#18](https://github.com/carloslfu/slotstream/issues/18).

- The development Mac app uses automatic speculative decoding when the draft
  head is available and fits the selected memory budget. Short conversations
  create earlier reusable checkpoints; longer prompts keep the engine's
  throughput schedule and all checkpoint provenance checks.
- Automatic readiness retains a loaded model while you read or compose in
  the foreground. Background inactivity, memory pressure, power saving and
  sleep still release it. Reloads within an app session reuse successful pinned
  verification only when APFS file identity and change timestamps still match.
  New app launches and changed files require fresh verification. Immediate
  reloads also wait for macOS's memory-statistics refresh before replanning,
  preventing stale readings from unnecessarily shrinking the cache.
- Growing the expert cache preserves slot positions and avoids gathering a
  second copy of the occupied cache. The live governor also checks temporary
  replacement memory against the process target and available system memory.
  When growth cannot fit, it keeps the current warm cache and retries later.
- The download `slotstream run`, `serve` and `launch` offer on first use and
  the development Mac app's model download now also fetch the 37.5 MB
  decode-forecast file 0.2.19
  added. Only `slotstream pull` did, so models downloaded the other ways
  decoded with the earlier, slower forecast. A model already downloaded
  without the file still needs one `slotstream pull`: `slotstream doctor` now
  says when the file is missing, and the engine's startup line names the
  command. Three community reports on 0.2.22 ran without the file. The
  library's `WeightStore.download(_:log:)` still fetches the weights only;
  the [library guide](docs/LIBRARY.md#check-and-download-weights) shows how
  to fetch the file with `TapCorrectionSidecar.ensure`.

## 0.2.24 - 2026-09-23

- Nullable string tool parameters declared with a JSON Schema type array now
  stream incrementally and preserve numeric-looking strings, like the equivalent
  scalar and `anyOf` forms. Truncated arguments retain the ordinary length
  termination behavior.
- Conversation splicing checks compatible branches in memory and on disk before
  choosing a retained transcript. A longer incompatible branch no longer hides
  a shorter matching conversation. Numerical checkpoint validation is unchanged.
- The native acceptance battery now runs the issue-21 streaming, branch-reuse
  and restart suite, including OpenAI compatibility checks.

## 0.2.23 - 2026-09-22

- Long prompts share expert reads across more existing compute passes, reducing
  repeated reads without changing those passes. The scheduler preserves
  checkpoint boundaries and chooses smaller groups when memory is tight.
  See the [read-group and reuse qualification](db/records/measurements/prompt-speed-qualification-2026-09-21.md).
- The Mac app can reuse compatible prompt checkpoints after a restart for
  ordinary conversations. Checkpoints record the producing pass size, and
  read groups stop at the checkpoint actually selected. Thinking and incognito
  conversations keep their inference state off disk; an unavailable cache
  falls back to ordinary inference.
- Thinking and answering continue one live generation session, including
  **Answer now**, natural completion and a reached thinking budget. The
  engine reads only the transition suffix instead of reprocessing the thought.
  [`Engine.generatePhased`](docs/LIBRARY.md#a-thinking-phase-followed-by-an-answer)
  exposes this behavior with separate sampling and output budgets for each phase.
- Automatically pair fused-workspace accounting with larger expert-read groups
  on the qualified M5 Pro text-prefill path, including MTP. Main and draft
  phases account for their overlapping hidden states. When smaller expert-buffer
  writes allow substantially larger groups, the engine selects them automatically.
  Groups adapt to the live memory budget and keep chronological compute passes;
  other execution paths retain their existing policy. Applications need no new
  setting. The [MTP measurements](db/records/measurements/mtp-prefill-policy-2026-09-21.md)
  record the paired gain, tested configuration, memory peaks and remaining limits.
- The engine and Mac app use the same pinned MLX backend and matching Metal
  libraries. The measured M5 Pro profile enables upstream fused D256 prefill
  attention with causal and sparse masks. Other profiles keep backend dispatch;
  `SLOTSTREAM_OPT_FUSED_PREFILL=0` restores its normal selection on the measured
  profile too. Disk checkpoints include the backend's arithmetic identity.
  The [integration measurements](db/records/measurements/fused-prefill-integration-2026-09-21.md)
  separate the complete backend upgrade from the fusion-only comparison;
  percentages from different studies must not be combined into a total speedup.
- Backend qualification keeps historical golden differences visible and adds
  independent current-backend model comparisons and scalar attention oracles.
  Exact speculative verification retains one-row projection arithmetic when
  selected. The installer matches older macOS shaders to the downloaded release,
  including releases made before this upgrade.
- Closing response details from the Mac app completes immediately while a
  thinking response is updating.

- Chat Completions streams long string tool arguments as they are generated,
  and parsing no longer rescans the whole growing argument at every token.
  Token-limit truncation now returns `finish_reason: length`, requested usage
  and the stream terminator, including incomplete required tool calls.
  Partial arguments are not completed calls and must not be executed.
- Conversation splicing recognizes earlier assistant replies inside a longer
  cached descendant, preserving generated reasoning and tool syntax across
  later turns. The disk cache retains generated IDs alongside its aligned
  checkpoint so restarting does not drop reasoning from the reconstructed
  prompt. Preparation checks retain the request's cancellation and
  deadline between history turns.
- Serving logs cache reuse decisions, periodic request phases and socket
  output failures. Prefill progress follows elapsed time and estimates the
  remaining wait from recent throughput. Ollama tool refusals name the
  supported OpenAI route.
- Serving diagnostics follow aligned cache boundaries and typed memory
  failures. The pressure test interrupts an actual scope spanning several
  prefill passes, then checks admission refusal and bounded recovery before
  retrying inference.

- Custom memory limits in the Mac app can exceed the automatic default within
  the Mac's supported range, with pressure protection and cache resizing still
  enabled. First switching to Custom keeps the current budget; later switches
  remember the last custom limit. Settings show current usage and the budget
  available now separately.
- `--memory-limit-gb` adds an adaptive process ceiling to the CLI. Existing
  fixed-cache flags retain their behavior. Diagnostics and budgeted model
  startup now share the same feasibility check at every context size, and
  cache resizing updates the reported current budget while retaining the
  selected ceiling.
- Adaptive limits survive the server's context assignment and are also
  available through `launch`. Fixed-profile diagnostics reject the option
  instead of silently ignoring it. Saved app limits outside the current Mac's
  range remain visible with a correction prompt.
- Fractional memory limits retain their precision in launch arguments and
  reported targets. Response details distinguish the budget used from the
  saved custom limit, and busy-machine guidance respects the hardware bound.
- Small caches recover after memory pressure or a busy startup even when the
  missing amount falls below the normal growth threshold. Recovery still waits
  for available memory and the existing cooldowns.
- Swift memory-planning APIs retain their original callable signatures.
  Directly constructed adaptive plans reject conflicting sources, missing
  targets and targets above the saved limit before model allocation.

- `slotstream optimization-state-check --variant complete-prompt` passes
  again. Tiling vision queries changes the rows an image produces, so the
  cache keys an image prompt on that setting too. When tiling joined the
  deployed family, the check kept looking up retained image states with the
  untiled key. It found nothing and stopped at its first image case, so its
  remaining cases, including every speculative-decoding case, did not run.
  Generation and the checks now take the key from one function. With them
  running again, the speculative variant asserts the resume rule's handoff: a
  state built without the draft head is not continued by a request that uses
  one. That request reads its whole prompt and answers what a cold read does.
  The previous handoff is still checked with `SLOTSTREAM_OPT_ALIGNED_RESUME=0`.
  No engine result changed.

## 0.2.22 - 2026-09-18

- Automatic context selection no longer trades away expert cache above the
  measured decode range on the assumption that its flat speed estimate means
  no performance cost. This fixes large explicit memory budgets silently
  becoming long-context reservations instead of useful cache. Busy starts use
  the same decision rule. Explicit `--max-context` choices remain available;
  startup and JSON reports distinguish the process budget, allocated expert
  cache and runtime/context allowances from measured memory usage.
- A continued conversation now computes what a cold one computes. The engine's
  arithmetic depends on how tokens were grouped into passes and on whether each
  one was read or generated, so the state a turn left behind, its prompt read
  in passes and its reply decoded a token at a time, was not what reading those
  same ids computes. The difference sat in the same band as re-chunking a
  prefill, 3.7% to 5.9% of the logit spread, and it crossed a token: on a
  1,430-token agent turn a fresh read scored `>` at 0.9576 and `]` at 0.0421
  for one position of tool-call syntax, the cached turn inverted them, and the
  model's first `file.edit` call came back malformed. A request now resumes
  only a state whose length is one of its own prefill pass boundaries and whose
  every token was read in those passes, and re-reads the rest, so the same
  conversation gives the same tokens and bit-identical prompt logits either
  way. The Sevra basics run that produced that malformed call now writes the
  same edit with no correction round. A follow-up turn pays one partial pass
  for it: measured at 961 slots on a three-turn chat, follow-up prefill 2.47 s
  against 8.56 s, both well under the 26.3 s of reading the conversation cold.
  New gate `slotstream prefix-exact-check`, and the weights-free
  `aligned-prefix-resume` catalogue check holds the policy in CI;
  `SLOTSTREAM_OPT_ALIGNED_RESUME=0` restores the previous reuse.

## 0.2.21 - 2026-09-18

- Faster speculative decode on long prompts. The three-row verify pass of
  draft depth 2 no longer falls to the dense attention kernel, whose cost
  grows with the context: from 6,144 tokens it runs two rows at a time through
  the vector kernel, the kernel plain decode's attention uses. The gain grows
  with the context, because it is the dense kernel's growth that it removes.
  On a quiet machine at a 22 GB target, speculative decode measured 11.82
  against 11.67 tok/s (x1.013) with a 16,356-token prompt, x1.085 on a second
  16,356-token prompt and x1.29 with a 32,740-token prompt, and the fetch-free
  verify pass is 32% cheaper at 32,740 tokens and 45% at 65,508. The split changes which drafts are accepted, so the 16k gain follows the prompt, 1% and 8% in the two measured; at 32k both arms accept alike and the ratio is the pass saving alone. `SLOTSTREAM_OPT_VERIFY_SPLIT=0` restores the previous pass.
- Exact mode, off by default: `SLOTSTREAM_OPT_ROW_INVARIANT=1` with
  `SLOTSTREAM_OPT_VERIFY_SPLIT_CONTEXT=0` makes a speculative run's output
  identical to a plain run's in the same mode for draft depths up to 4 (128 of
  128 tokens in the 16k comparison). The model's small dense matmuls run
  through one kernel at every row count, which changes plain decode's rounding
  as well, and each verify row attends in its own call over the keys plain
  decode reads, so the backend picks the same attention kernel.
- New gate `mtp-rowcheck` in `Tools/verify.sh`: in exact mode, every row of a
  two-row and a three-row verify pass, and the state a three-row pass leaves,
  must equal the one-row passes bit for bit, on a prompt whose positions cross
  1,024 keys, where the attention kernel changes, and one above the indexer
  budget. The weights-free `verify-pass-rows` catalogue check holds the
  kernels in CI, at every key count where the backend changes attention
  kernels. `mtp-bench --arms` compares plain, shipped, split and exact decode
  on one warm engine; `mtp-passcost` gains `--prompt-file`, `--max-context`
  and `--attention-modes` (stock, split, exact).
- Start a coding agent already connected to Slotstream with
  `slotstream launch claude`, `codex`, `pi`, `opencode` or `hermes`, or
  `slotstream launch` to pick from the agents installed. When no server is
  running, the command starts one in the background first and shows its start
  until the model answers: with the automatic window, or the one the agent
  needs when that is larger, shared prompts kept on disk in
  `~/.slotstream/prefix-cache`, and its log in
  `~/.slotstream/logs/serve.log`. The server keeps running while any agent
  the command opened runs, and stops 30 minutes after the last one exits
  (`--idle-exit`); Control-C while it starts stops it. The agent's settings
  are checked before the server starts, the model is never downloaded without
  asking, a server you started yourself is used as it is, and a server the
  command started is restarted only when its window is too small and no agent
  uses it. Two launches at the same time start one server between them.
  `--memory-gb` sets the new server's memory target, `--no-start` uses a
  running server only, and `--port` names the server's port, whose log is
  `serve-<port>.log`. Thinking starts off, and the agents get timeouts long
  enough for a first prompt: Claude Code runs with `MAX_THINKING_TOKENS=0`,
  30-minute request and stream timeouts and its nonessential network traffic
  off.
  The command reads the served model, window and reply limit, connects the
  agent for that run without changing its own configuration (Pi's models file gains
  a `slotstream` provider; Hermes gets its own `~/.hermes-slotstream`
  folder), and replaces itself with the agent. Codex gets a model catalog
  with its own version's base instructions, downloaded once; Claude Code gets
  its connection through `--settings`, so its settings files cannot redirect
  it. No prompt leaves the Mac by a side route: Claude Code's cloud provider
  switches are turned off and its API key and key helper cleared, the hosted
  WebSearch tool is denied, opencode enables only the `slotstream` provider
  through `OPENCODE_CONFIG_CONTENT` (an exported one is merged, not replaced)
  and points its build and plan agents at the model, Hermes pins every side
  task (including the command-approval check) to the model, and `codex
  cloud` is refused. Pi's models file keeps every other entry as written,
  down to key order and number formatting, and a symlinked file stays a
  symlink. `--dry-run` prints the plan with keys hidden and downloads
  nothing.
  New guides: [Claude Code](docs/CLAUDE-CODE.md) and
  [coding agents](docs/CODING-AGENTS.md) for Pi and opencode; the Codex and
  Hermes guides start with the command.
- `slotstream stop` stops the Slotstream server on a port (`--port`), whoever
  started it, including one `slotstream launch` is still starting, and says
  so when nothing runs there. The refusal when another model process holds
  the lock names it too.
- `slotstream serve --idle-exit <minutes>` stops the server after that long
  with no request and no registered process still running. Once it decides,
  new requests get a 503 instead of starting. `GET /slotstream/status`
  reports the server's process, window, requests and idle time, and
  `POST /slotstream/clients` registers a process to wait for; see
  [server status](docs/API.md#server-status), including what sized its
  memory plan. The server logs a line when `SIGTERM` stops it, also while the
  model loads, and a port already in use names `slotstream stop`.
- `slotstream prefix-cache` lists `~/.slotstream/prefix-cache` when no
  `--dir` is given, the directory servers started by `slotstream launch` use.
- Serve the Anthropic Messages API at `POST /v1/messages` and
  `POST /v1/messages/count_tokens`, the protocol Claude Code and the Anthropic
  SDKs use. Streaming follows Anthropic's event order with a `ping` every 10
  seconds during a long prompt read; tools, `tool_choice`, images, plain-text
  documents, thinking with replayable signatures, stop sequences and usage
  with the reused prompt as `cache_read_input_tokens` are supported. Unknown
  top-level fields are ignored and named in `X-Slotstream-Ignored-Fields`,
  Claude Code's per-conversation attribution line is dropped, and an
  oversized prompt fails with the `prompt is too long` message Claude Code
  compacts on. Anthropic's hosted server tools (`web_search_*`, `web_fetch_*`,
  `code_execution_*`, `tool_search_tool_*`, `mcp_toolset`, `advisor_*`) run on
  Anthropic's servers, so they are dropped from a request rather than
  answered, and naming one in `tool_choice` is an error; `container` and
  `mcp_servers` are refused with what they ask for.
- `/v1/chat/completions` accepts `store: false`, `metadata`,
  `prompt_cache_key`, `prompt_cache_retention`, `safety_identifier` and
  `service_tier` without effect, applies `min_p`, and takes the
  `effect_disposition`, `display_kind` and `display_metadata` bookkeeping
  Hermes puts on a message. Pi and the OpenAI SDKs
  send `store: false`, and it was refused
  ([#19](https://github.com/carloslfu/slotstream/issues/19)). `store: true`
  still returns 400, since no completion is kept to fetch.
- `/v1/chat/completions` without `max_tokens` lets a reply use a quarter of
  the served window, up to 8,192 tokens, as `/v1/responses` already did; the
  512-token default cut agents' edits short. The Ollama endpoints keep 512.
- A request that waits behind others until its estimated prefill no longer
  fits the wait budget now fails with the retryable 503
  `prefill_deadline_exceeded` instead of a 400.
- A request body over the 32 MiB limit is read and discarded before the 413
  answer, up to 256 MiB, so the client reads the error instead of losing the
  connection. A client that declares a large body and then stops sending gets
  its 413 as soon as the next piece fails to arrive, and the discarding never
  takes longer than 10 seconds in total.
- Library: an app that embeds Slotstream can count a chat request's prompt
  tokens (`Engine.countChatTokens`), find where a prompt's shared head ends
  (`Engine.sharedPrefixBoundary`), name a request's shared prefix and how long
  it is kept (`RequestControl.sharedPrefixTokens`, `.sharedPrefixRetention`,
  `SharedPrefixRetention`), store a checkpoint with that retention
  (`PrefixCache.storeReusableCheckpoint(…retention:)`), read the shared states
  on disk (`PersistentPrefixCache.storedSharedStates`), and see the save
  points and the stop sequence a generation used (`GenStats.sharedPrefix…`,
  `GenStats.stopSequence`). A server can report and bound its own use:
  `Server.activity`, `Server.idleExit`, `ServerActivity` and
  `ProcessIdentity`. Existing signatures are unchanged, and disk-tier
  statistics written by 0.2.18 to 0.2.20 still decode.
- Once the first image loads the vision tower, a server that retained whole
  conversations keeps the largest retention that fits beside it, instead of
  the much smaller budget share. At `--memory-gb 12` with a 65,536-token
  window, the share left coding agents' conversations too long to keep, so
  every later turn read the whole prompt again.
- A stop sequence is no longer matched inside the reasoning that precedes an
  answer.
- A tool call whose integer or number argument the model writes outside the
  range of a 64-bit integer, such as `1e20`, or as `inf`, no longer stops the
  server; the value is kept as a number, or as text when JSON has none.
- Streamed answers after reasoning no longer start with the blank lines the
  model writes after `</think>`, matching non-streamed answers.
- Reuse a shared system prompt across conversations
  ([#18](https://github.com/carloslfu/slotstream/issues/18)). While a prompt
  is processed, the head other conversations will start with is kept as a
  shared prefix: the system message when it ends 512 tokens or more in, and
  the longest head the prompt shares with a state already kept. The save point
  is the last prefill pass end at or before that boundary (the 256-token grid
  by default), so no pass is reshaped and outputs are unchanged. The state is
  forked into the in-memory prefix cache and, with `--prefix-cache-dir` and
  `--prefix-cache-min-tokens` or more tokens, written to disk. The next
  conversation with the same system prompt resumes from it instead of
  processing it again, in the same process or after a restart. Shared prefixes
  are kept once, are never replaced by the conversations that extend them, go
  after conversations when the disk quota needs room, and `slotstream
  prefix-cache` lists them, with their own row, a count in its summary and a
  `shared` field in `--json`, and `optimization-state-check shared-prefix`
  and `shared-prefix-mtp` hold the behavior on the model. Library callers name
  the shared head with `request.sharedPrefixTokens`; `GenStats` reports the
  save points. A request
  that declares tools, as coding agents' requests do, keeps its shared prefix
  in memory like a conversation (`request.sharedPrefixRetention`), and a
  shared prefix stays as recent as the conversation that continues from it,
  so other conversations no longer displace an agent's instructions before
  its next session starts.

## 0.2.20 - 2026-09-16

- Serve the OpenAI Responses API at `POST /v1/responses`, the protocol Codex
  requires since it dropped Chat Completions. Function tools, Codex's
  `namespace` bundles (flattened to `namespace.name` and split again on the
  way back) and freeform `custom` tools such as `apply_patch` render through
  the model's native tool grammar; replies stream as the item and delta
  events Codex reads, with a progress event every ten seconds during a long
  prompt read because Codex's idle timeout counts events. Reasoning streams
  as summary text and is accepted back as replayed history; pictures arrive
  as `input_image` parts in messages and in `function_call_output` items.
  `Tools/codex_catalog.py` writes the model catalog that tells Codex the
  served window and declares `apply_patch`. Guide: `docs/CODEX.md`; wire
  details: `docs/API.md`. Verified with `codex exec` creating a file through
  `apply_patch`, reading it back through `exec_command`, describing an
  attached picture, and reading one through `view_image`.
- Library: an app that embeds Slotstream can return allocator-held buffers
  after releasing its engine (`Engine.releaseUnusedMemory()`), drain the
  memory governor before that release (`MemoryGovernor.stopAndWait()`), and
  stop weight verification from a Stop button or at shutdown
  (`WeightStore.status(shouldContinue:)` and
  `WeightStore.sha256(of:shouldContinue:)`). Existing signatures are
  unchanged. The Sevra for Mac development app in `apps/macos` uses them; it
  is not part of the release download.
- Say who Slotstream is for. It is built for Macs that cannot hold the model,
  16 to 64 GB; 96 GB and larger Macs run it but are not the optimization
  target. The README, hardware guide, getting-started guide, engineering
  notes, `llms.txt`, agent instructions and the plan records now state it, and
  the 96 GB+ estimate row is labeled as the case where the model fits in
  memory. No number changed. Decision:
  `db/records/decisions/target-range-macs-that-cannot-hold-the-model.md`.
- Say that Slotstream is native end to end. The README's new Built native
  section, the engineering notes' native-stack table, the Swift library
  guide, the Sevra Mac notes and `llms.txt` now describe the Swift, MLX and
  Metal stack, the run-time-compiled Metal kernels, the C download decoder
  and the role of the Python tooling, and keep speed attributed to the
  measured mechanisms. No number changed. Decision:
  `db/records/decisions/native-stack-stated-on-every-surface.md`.

## 0.2.19 - 2026-09-16

- Generate replies 1.10x faster with a more accurate expert forecast at
  the same lead time. The decode lookahead now reads the model's state right
  after the previous layer's attention step, applies the next layer's router to
  it and corrects the result with a small learned table fitted for the
  checkpoint, so it reads about a fifth fewer expert records from the SSD during
  decode and wastes about three quarters fewer speculative bytes. Output is
  unchanged. `slotstream pull` fetches the 37.5 MB correction file
  (`lookahead/tap-correction-attention-rank128-v1.safetensors`) next to the
  weights and verifies it; if you installed the model with an earlier release,
  run `slotstream pull` once more, and until then the engine runs the 0.2.16
  forecast. The memory plan charges 409 MiB for the lookahead with the file
  (373 MiB without). `SLOTSTREAM_EXPERT_PREFETCH_TAP=boundary` keeps the 0.2.16
  forecast with the file present. Levers measured and left out on their
  registered gates: a co-routing prior, a lower read-issue threshold and
  computing the next layer's attention early, which reads more accurately but
  costs more GPU time than the reads it saves; see the
  [expert lookahead guide](docs/EXPERT-LOOKAHEAD.md).

## 0.2.18 - 2026-09-14

- Keep long conversations on disk with `serve --prefix-cache-dir <dir>`. A
  restarted server, or a conversation longer than the in-memory prefix cache
  holds, restores its last committed state instead of processing the whole
  prompt again, and continues exactly as it would have from memory. Each later
  turn writes only the tokens it added, and the previous turn's state stays so a
  reply can be regenerated after a restart. Off by default;
  `--prefix-cache-disk-gb`, `--prefix-cache-min-tokens` and
  `--prefix-cache-max-age-days` bound what is kept, and states nobody continued
  are removed before conversations. Files are checksummed, used only by the
  binary, model files and settings that wrote them, and hold conversation token
  ids: `slotstream prefix-cache --dir <dir> --clear` erases them. Library
  callers use `Engine.enablePersistentPrefixCache(_:)`, keep a request off disk
  with `RequestController.persistsPrefixState = false`, and remove a deleted
  conversation's states with `removeStates(overlapping:)`.

## 0.2.17 - 2026-09-13

- Pick the context window for each Mac. Auto now takes the largest of 32,768,
  65,536, 131,072 and 262,144 tokens that keeps speculative decoding, keeps one
  complete conversation ready for follow-up turns and adds at most a tenth to
  the planner's estimate for a typical request. In decimal-GB simulations that
  is 32,768 tokens through 32 GB, 65,536 from 36 GB, 131,072 at 64 GB and
  262,144 from 96 GB. A Mac that is busy at startup gets a smaller window
  rather than losing speculative decoding, and `doctor` lists every candidate
  with its reason.
- Accept `--max-context` up to 262,144, the model's full window, or `auto`.
  Requests with images still use at most 65,536 tokens. A window above 32,768
  keeps one complete conversation for follow-ups when the plan can hold it,
  and says how much a follow-up reuses when it can't.
- Raise auto's memory target on 64 GB and larger Macs by the chosen window's
  own charge, to 43.2 GB at 64 GB and 54.7 GB from 96 GB, so the expert cache
  keeps its size. `--max-context 32768` restores the former plan.
- With an explicit `--memory-gb`, auto also picks the window inside that
  target, trading some cache for a larger window within the 10% limit. Add
  `--max-context 32768` to keep an earlier plan. `--experts-per-layer` and
  `--pool-gb` keep 32,768 tokens.
- Show why `doctor` rejects a memory target in its tier table: too small for
  the window, above the Metal working set, or more than is reclaimable now.
- Qualify a full 131,072-token window on the development Mac: the prompt and
  reply fit their memory plans, and speculative decoding produced exactly the
  same reply. The full 262,144-token window has not run natively yet.

## 0.2.16 - 2026-09-13

- Generate replies 1.11x faster with speculative decoding. The new decode
  lookahead runs the router of the layer two ahead on the current hidden state,
  reads the experts it picks from the SSD straight into cache slots, keeps FP32
  router weights and drains the GPU every four layers instead of every layer.
  Output is unchanged. On twelve held-out prompts at a 20 GB target the
  development Mac went from 11.79 to 13.47 tok/s median. The memory plan charges
  its 373 MiB, and `SLOTSTREAM_OPT_EXPERT_PREFETCH=0` turns it off.
- Use speculative decoding from 32 GB Macs. Auto now turns the draft head on
  when the cache keeps 76 experts per layer, down from 120, a 21 GB target.
  Two drafts measured faster than plain decode at that size.
- Drain the GPU after every layer whenever a pass could not keep several layers
  of experts pinned, and let the memory governor re-plan with the running
  engine's lookahead decision and memory.
- Correct the benchmark report's medians, which took the upper middle value
  over an even count. Recorded cohorts score slightly lower, and every verdict
  stands.
- Document speed estimates and recommended context windows for each memory
  tier, and correct the 8 GB guidance: the smallest plan doesn't fit, so
  Slotstream refuses to start.

## 0.2.15 - 2026-09-11

- Stop treating system-wide macOS paging as a failed correctness or memory-budget
  check. Preserve paging diagnostics and the real process-memory, headroom,
  cancellation and completed-output safeguards. Clean performance measurements
  remain a separate qualification.
- Use two draft tokens by default when speculative decoding is enabled. Valid
  `SLOTSTREAM_DRAFT_DEPTH` overrides remain supported. The choice follows the
  mixed-workload comparison; it is not a universal throughput improvement.
- Fix peak-memory reporting after GPU buffers are freed by reading macOS's
  lifetime physical-footprint high-water. Keep current usage and sampled
  request peaks distinct, and report memory on early failures and cancellations.
- Add native memory-counter and explicit-budget regression checks, preserve
  older saved-statistics decoding, and clarify memory budgets and historical
  estimates in the documentation.
- Include the draft-depth studies, decode bottleneck evidence, and the reviewed
  Expert Lookahead plan. Learned expert prediction and expert prefetching remain
  planned work; this release does not implement them.

## 0.2.14 - 2026-09-10

- Faster repeated and continued prompts through committed prompt checkpoints,
  bounded prefill read grouping, and reuse of completed prompt state.
- Lower runtime memory through compact state and n-gram storage, bounded output
  buffering, and reduced temporary allocations. Expert reads and transfers
  retain checked bounds, exact ownership, and recovery after failure.
- Qualified rotation and projection paths reduce avoidable work while keeping
  reference fallbacks for unsupported shapes and platforms. Vision attention
  workspace and memory reservations are bounded independently.
- More responsive memory handling, cancellation, and serving recovery. Preserve
  the independent CLI, Swift library, and OpenAI/Ollama serving contracts.
- Publish the complete optimization evidence, including rejected experiments,
  performance limits, sustained generation comparisons, and practical serving
  results. [Integrated measurements](MEASUREMENTS.md#final-integrated-optimization-results)
  compare selected and reference paths in the same build; they are not a
  direct comparison with the previous public release. Prompt reuse improved
  responsiveness, while sustained decode was flat or slightly slower in the
  measured profiles. There is no universal tokens-per-second speedup claim.

- Shared context and request-wait configuration across run, serve, doctor and
  the Swift library. Exact allocation accounting includes retention and loaded
  components; discovery separates the configured window from model/mode limits.
- Cancellable queueing and request-to-first-token deadlines, with memory checks
  before bounded work, typed errors in every serving dialect and safe cleanup.
- Checked context diagnostics preserve reply room and enforce bounded late
  prefill passes. Larger public windows remain gated on full qualification.

The v0.2.12 and v0.2.13 tags stopped at prepublication test-harness failures
and have no release archives. This release corrects the isolated fixture and
startup-memory error checks, while preserving those tags and their evidence.
The release now signs and publishes the exact archive from successful main
CI, with checksum, source-identity and version verification.

The public artifact is installed and serving locally. Local acceptance completed
on September 11: all 25 original model gates and all 31 installed-release
checks passed against the same published binary. The final unchanged governor
and long-prompt memory tests completed with zero swap activity. Earlier failed
intervals and the passing reruns remain preserved in the
[release qualification record](db/records/measurements/release-qualification-0-2-14.md).

## 0.2.11 — 2026-09-06

- Compressed model downloads now come directly from the public Hugging Face
  mirror, avoiding publisher charges per download. The package, original
  hashes, compression saving, and resumable installation remain the same.
- Respect Hugging Face rate-limit reset headers and longer server-requested
  waits, with prompt cancellation. Add real HTTP retry and cancellation gates.
- Preserve older download URLs with free static redirects to Hugging Face.
  Add a resumable publisher that checks immutable package contents and
  preserves the existing model repository.

## 0.2.10 — 2026-09-06

- New model downloads use a lossless, quantization-aware package from
  Cloudflare R2 and its edge CDN: **16.12% fewer bytes**, reconstructing the
  exact original model. Independent chunks allow download, decoding, and
  writes to overlap while bounding memory.
- Embedded package and original-file hashes verify each stage. Unavailable
  objects fall back to pinned Hugging Face ranges. Verified progress resumes
  after interruption; damaged partial files repair automatically.
- Connection count adapts to measured throughput; explicit connection counts
  and raw mirrors remain available. Existing raw downloads keep their progress.
- Fixed retry progress accounting, duplicate verification, cancellation,
  simultaneous writers, malformed resume maps, unsafe partial-file types,
  and optional-file writes during fallback. Added native sanitizer and real
  HTTP gates for both transports.

- Interruption tests wait for durable chunk progress and explicit writer
  readiness, including delayed-start fixtures, so slower CI runners exercise
  the same resume state.

## 0.2.9 — not published

The release gate exposed a timing assumption in the interruption fixture.
No release asset was published; 0.2.10 includes the corrected qualification.

## 0.2.8 — 2026-09-05

- The OpenAI Chat Completions endpoint now supports function tools, streamed
  calls, tool results, and reasoning history. Tool IDs survive the full agent
  loop, including results returned out of order. Malformed calls fail
  explicitly. This adds the Hermes integration requested in
  [#11](https://github.com/carloslfu/slotstream/issues/11).
- An explicit `--max-context 65536` supports Hermes while ordinary serving
  keeps its existing context default. Long-context state and transient memory
  are charged before allocating the expert cache.
- Model discovery reports the runtime context window and available vision
  capability, so Hermes no longer misidentifies an image-enabled server as
  text-only. OpenAI requests accept
  Hermes's reasoning flags and bounded `options.num_ctx`. Unsupported
  structured output has an explicit rejection compatible with Hermes's
  fallback handling.
- Added a [connection guide](docs/CLIENTS.md) and a
  [Hermes setup guide](docs/HERMES.md), including local auxiliary routing,
  output budgets, and troubleshooting.

## 0.2.7 — 2026-09-04

- **The model reads images.** The checkpoint has always carried a vision tower
  — 333 `vision_tower.*` tensors the weight loader skipped by name — and the
  chat template has always rendered an image part to `<|image_pad|>`. Now the
  tower runs and its rows are spliced under those placeholders, on every
  dialect: Ollama's `images` array on `/api/chat` and `/api/generate`, OpenAI
  `image_url` parts, AI-SDK `file` parts on the fx gateway (whose catalogue now
  advertises the `vision` tag), and `slotstream run --image`.

  Based on [#10](https://github.com/carloslfu/slotstream/pull/10) by
  [@msx98](https://github.com/msx98), whose port of the tower and, in
  particular, whose prefix-cache design — keying each placeholder run on a
  digest of the image's bytes, because every image expands to a run of the same
  token id — are the load-bearing parts of this.

  Reworked before landing: the tower's attention goes through
  `MLXFast.scaledDotProductAttention` with a per-block `eval` (written out, it
  materialized a float32 `[16, N, N]` score matrix twice per block — 5.4 GB
  each at the largest image); the splice is a concatenation of contiguous spans
  on the GPU rather than a scalar loop over a CPU copy of the hidden state, and
  can no longer disagree with itself about how many rows it was given; image
  sources are inline bytes only, where the previous fallback to
  `Data(contentsOf:)` would fetch an arbitrary host or read a local file
  through `file://`; the tower loads under the generation lock and only when
  the machine can spare it, and `serve` announces its 0.9 GB rather than taking
  it silently against a printed plan that did not include it.

  Gated: `vision-check` (75 weights-free assertions — the reference
  processor's geometry, the source policy, run clipping, request shaping),
  `Tools/vision_ref.py` (the tower against an independent float32
  implementation of the reference, inside the band bfloat16 itself spans),
  `Tools/vision_serving.py` (19 assertions over every dialect, against a real
  server, requiring the model to name what is in the photograph), plus a
  vision leg in `mtp-check` and six image cases in `api_robustness.sh`.

- **A photograph's EXIF orientation is applied.** A phone stores its sensor's
  pixels plus a tag saying which way is up; every viewer turns the picture
  before showing it, and so does the reference processor
  (`ImageOps.exif_transpose`). slotstream did not, so every portrait photograph
  reached the model on its side — and the token count with it, since four of
  the eight orientations swap the axes. All eight are now checked corner by
  corner, weights-free, against EXIF's own table.

- **An image that ends mid-file is refused.** ImageIO is lenient by design:
  half a PNG decodes to the rows it has plus blank space, and reports itself
  complete, so an upload cut short by a dropped connection came back as a
  confident description of a mostly empty picture. The container's end marker
  is checked instead — `IEND` for PNG, the end-of-image marker near a JPEG's
  tail (editors append after it), `;` for GIF — and anything else is left to
  ImageIO rather than guessed at.

- **A transparent PNG is composited onto white, not onto black.** Found by
  putting a set of images with known content through the finished path: the
  decoder's context is premultiplied, so drawing over fresh memory made every
  transparent pixel black. Photographs have no alpha and never showed it;
  logos, charts, diagrams and screenshots exported with transparency do, and
  black text on a transparent background reached the model as black on black —
  it answered "the image is entirely black, with no discernible features or
  content". It now reads the text. Opaque images are byte-identical either way.

- **The request body cap is 32 MiB**, up from 4 MiB, so a base64 picture fits;
  the largest image accepted is 24 MiB decoded. The robustness suite's oversize
  probe moved with it — at 9,999,999 bytes it had silently stopped testing
  anything.

## 0.2.6 — 2026-09-03

- **An optional string in a tool schema is no longer read as a number.** fx
  writes an optional parameter as `anyOf: [{"type":"string"},{"type":"null"}]`,
  and three of the five required fields on its `terminal` tool are declared that
  way. Typed as an unknown union, the coercion took a numeric-looking value at
  face value, so a command or working directory like `2024` would have been sent
  as the integer 2024 and rejected. A union of exactly one real type plus null
  now resolves to that type; a genuine two-type union stays conservative. Found
  by capturing fx's real schemas off the wire rather than from a fixture.

## 0.2.5 — 2026-09-03

- **Fixed: a JSON null anywhere in an fx request failed the whole turn.** The
  chat-template bridge throws on `NSNull` and maps Swift `nil` to null, and the
  gateway dialect handed it `NSNull`, so a single `"default": null` inside one
  of fx's tool schemas — or a null for an unset optional argument in a replayed
  tool call — returned `400 template_error: Cannot convert value of type NSNull
  to Jinja Value` with no output at all. fx sends both routinely; the failure
  showed up on the first real multi-step task, one turn after a `terminal` call.
  Nulls now cross as an empty Optional, which the bridge degrades to null, so
  they render as `null` and array positions are preserved rather than dropped.
  Gated three ways: a T0 check that no path bridges an `NSNull`, and three live
  scenarios covering a null in a tool schema, in tool-call arguments, and in a
  JSON tool result.

## 0.2.4 — 2026-09-03

- **Decode is about 10% faster at small cache sizes, and the output is
  byte-identical.** `run` now prints a decode split beside the prefill one, and
  it found two things. The pool scatter was 20% of decode time and running at
  about 18 GB/s against a microbenchmark that writes slots at 49 to 75, because
  `SlotPool.ensure` ended every batch with a full GPU sync — 48 per token — that
  the gather in the same layer did not need. And the pool path was reading on
  the sweep's 12 lanes, tuned for long contiguous runs, where a layer's handful
  of nine-piece misses is latency-bound and wants more. Five interleaved rounds
  at 30 experts per layer: 6.93 to 7.63 tok/s, peak 7.5 to 7.8 GB, identical
  text. `SLOTSTREAM_SCATTER_MODE` and `SLOTSTREAM_POOL_QUEUE_DEPTH` are the A/B
  knobs; the sweep keeps its own `SLOTSTREAM_IO_QUEUE_DEPTH` of 12, because
  raising that one measured slower.
- **A pass can be read in query blocks, which bounds the attention transient
  without changing a number.** MLX 0.31.1 runs head dim 256 on its unfused
  attention path, materialising the whole `[24, pass, context]` score matrix;
  splitting the queries bounds it and is bit-identical at blocks of 256 and up.
  It is **off at every size the planner produces today** and engages only above
  the largest query-by-key product any prefill measurement covers — where the
  schedule already shrinks the pass — because measured end to end it lowers peak
  memory by 0.00 GB. The phase trace behind `SLOTSTREAM_MEM_TRACE=1` says why:
  attention, the PLE layer and the MoE sweep peak within 0.6 GB of each other,
  so bounding one alone can never lower the process. Recorded as a null result
  with its mechanism, not as an improvement.
- **M1 is closed, and the answer is that the eviction policy is not the lever.**
  The expert-locality study has been open since the first week: the simulator
  existed, the trace never did, because taking one needs a bounded forward pass.
  `SLOTSTREAM_ROUTER_TRACE` now records every routing decision and
  `Tools/trace_convert.py` feeds `Tools/cachesim.py`. On 220 decode steps at 30
  experts per layer, the shipped CLOCK measured 0.557 against LRU 0.568,
  LFU-decay 0.480, and an offline hot-set bound of 0.603 — so CLOCK stays. The
  same trace fixes the compulsory-miss ceiling for that workload at 0.906 and
  shows 10% of records serving 71% of accesses, which points the remaining work
  at capacity and a warm start rather than at eviction.
- The prefill sweep gathers each staging group's rows as it needs them instead
  of building one replicated copy of the whole pass up front (105 MB at a
  2048-token pass, 210 at 4096). Bit-identical; `sweep-check` reads the same
  3.320% of logit spread against the same control.
- **fx runs against slotstream.** A second HTTP dialect speaks the Vercel AI SDK
  Language Model Specification v4 over AI Gateway protocol 0.0.1, which is the
  wire [fx](https://fx.sh) uses. fx ships no plugin point, but its gateway client
  honours `FX_GATEWAY_CHAT_URL` and `FX_GATEWAY_BASE_URL` when they name an
  `http://` loopback address, so pointing it at a local server needs no fork and
  no patched binary: `POST /v3/ai/language-model` streams the turn,
  `GET /coding-agent/v1/models` is the catalogue, `GET /coding-agent/v1/credits`
  answers `fx credits`. Setup, limits and troubleshooting are in
  [docs/FX.md](docs/FX.md).
- **Native tool calling.** The model does not emit JSON tool calls; its template
  teaches it an XML form. That form is now parsed as it streams — several calls
  per turn, prose before and after, each argument typed against the tool's own
  JSON Schema — and a tag split across two token deltas can never leak into the
  user's transcript as text. Tool calls, tool results and reasoning also render
  back into a conversation, so an agent loop replays correctly.
- The catalogue derives every window from the running server's context cap
  rather than advertising a fixed one. fx reserves room for a reply only when
  the advertised reply budget is strictly smaller than the window; a fixed
  budget would, at small caps, tell fx it could fill the whole context with
  input and leave nothing to answer in.
- Agent turns get their own sampling defaults. The instruct default penalises
  repeated tokens, and the call format is obliged to repeat `</parameter>` and
  `</function>`, so the penalty pushed the model off the grammar exactly where
  it had to stay on it.
- The generated files cannot be committed stale. `llms-full.txt` comes from
  the docs and `MEASUREMENTS.md` / `PLAN.md` from the brain's records; a
  README edit that skipped the regenerate turned the docs job red, so
  `make hooks` now installs a pre-commit hook that regenerates them with
  the commit that moves their sources. The staleness check prints the drift
  it found instead of a bare verdict.

## 0.2.3 — 2026-09-02

- Reading a prompt is about twice as fast. A prefill pass of 256 tokens or
  more now sweeps each layer's experts through staging groups and MLX's
  grouped GEMM instead of gathering one matvec per token over the slot pool,
  reads consecutive experts as one contiguous `pread` per piece instead of
  nine ~307 KB pieces per record, and never writes the pool, so a long prompt
  no longer flushes what decode was using; the last pass admits the prompt's
  hottest experts so decode starts warm. Measured on the dev Mac, interleaved
  against 0.2.2's code: an 8k prompt at a 16 GB target 91 → 184 tok/s, ordinary
  prose 66 → 140, the 8.1 GB floor 51 → 93, `context-check --tokens 8192` 64 →
  152; at a matched 60-experts-per-layer pool a 4096-token pass reads 222
  tok/s (was 103). The n-gram rows a pass needs are now read in parallel
  rather than one at a time, which is where prose was paying ~35 s per 10k
  tokens. Peak memory is unchanged within 0.3 GB at 16 GB and 1.5 GB lower at the 8.1 GB floor, where MLX's buffer cache is now capped while a prompt is read.
  New gate `sweep-check`; `SLOTSTREAM_SWEEP=0`, `SLOTSTREAM_SWEEP_ADMIT=0`,
  `SLOTSTREAM_SWEEP_TRACE=1`, and `SLOTSTREAM_PREFILL_CACHE_MB` for A/B work.
  The planner's prefill estimates and the full-context waits on the README
  and in `doctor` moved with the measurements.
- slotstream is a Swift package as well as a binary. `Package.swift` declared
  no products, so nothing outside the repository could import it even by path:
  SwiftPM refused at graph resolution. There are now two library products —
  `Slotstream` (weights, planning, generation, serving) and
  `SlotstreamDiagnostics` (checks, goldens, benches) — beside the unchanged
  `slotstream` executable. `docs/LIBRARY.md` is the guide, including the part
  nobody guesses: MLX looks for its Metal shaders beside whichever executable
  is running.
- The weights are an addressable thing. `WeightStore.status()` answers ready /
  missing / incomplete / corrupt with the bytes still needed and the free disk
  where they land, so an app can ask before it tries to load, and a
  complete-looking copy is still hashed because size cannot see same-size
  corruption. `PinnedModel` is a public value; the download engine moved with
  it and no longer throws ArgumentParser's errors.
- The machine is a value, and a simulated one cannot allocate. `Machine`
  carries RAM, working set, availability and whether any of it was invented;
  a plan made for a simulated machine is marked and `Engine.load` refuses it.
  The global `Planner.availabilityOverride` is gone from the planning path.
  (Named `Machine`, not `Device`, because MLX exports its own `Device`.)
- Checks are library functions, not subcommand bodies. `runtime-check`,
  `governor-check`, `pull-check` and `sampler-golden` are `Diagnostics` and
  `Goldens` calls that return a report; the subcommands render it and print
  exactly what they always printed. New `slotstream-checks` runs the whole
  catalogue — 121 assertions across 9 checks, none needing weights — and CI
  runs it on every push alongside a coverage ratchet that holds a per-file
  floor.
- The serving layer's framing and routing rules are testable without a server:
  head parsing, the 411/413/431/400 decisions, absolute-form and query-string
  routing, and the loopback-only CORS policy. All of it previously needed a
  live server with 105 GB loaded.

- The repository carries its brain. `db/` is a public db.md store: every
  MEASUREMENTS.md and PLAN.md section is a record, every number on the README
  and the docs is a claim naming the measurement behind it and the surfaces
  it appears on, and decisions record what would reverse them. Both long
  documents are now generated from the records by `Tools/projections.py`,
  and `Tools/brain_gates.sh` (store validation, generated-document parity,
  the claims gate) runs in CI on every push.
- Docs: a Related projects section that names the peer engines and what each
  does differently, a Support section, `docs/HARDWARE.md` for rows measured
  on other Macs with an issue template to submit one, SECURITY.md, and
  CONTRIBUTING.md.
- CI runs the full build only when something other than prose changes; a
  docs-only push runs a twenty-second `docs` job (the llms-full.txt staleness
  check) instead.
- The serving layer answers while it is working. `/api/tags` and `/api/ps` read
  pool numbers through the *generation* lock, so both blocked for the length of
  a running request; with the accept loop also waiting on the connection
  semaphore, enough blocked metadata calls stopped the server answering
  anything at all, and a client polling either one saw a working server as a
  dead one. Pool numbers are published at each resize and read from a snapshot,
  and the accept loop never waits: a full pool answers 503.
- Reasoning no longer leaks into the answer. `think: true` returns the model's
  reasoning in `message.thinking` (`thinking` on `/api/generate`) and the reply
  in `content`; it used to hand clients the reasoning, a stray `</think>`, and
  the answer in one string.
- Deltas arrive per token. The incremental decoder waited for eight tokens
  before its first flush and held four back after it, so a client saw one delta
  per four tokens and nothing at all for a reply shorter than eight.
- An unseeded request is genuinely random. The sampler's default seed is a
  constant, so an unseeded request replayed the same text after every restart
  while the API documented the opposite. The seed is drawn at the HTTP boundary,
  leaving every offline gate deterministic.
- Stock clients work unchanged. JSON `null` means "not set" (the OpenAI client
  sends it for an unset `max_tokens`); `n: 1`, `frequency_penalty: 0`,
  `logprobs: false`, `logit_bias: {}`, `tools: []`, `response_format` text, and
  `user` are accepted at the value this server already implements and still
  refused at any other; `ollama show`'s empty `model` falls back to its `name`;
  and an untagged or `:latest` model name resolves to the only model. Knobs
  that would change the reply, such as `num_ctx` and `repeat_penalty`, are
  still refused rather than dropped.
- `ollama ps` reads correctly. It reported 104 GB of weights against a small
  pool and rendered "98% CPU" for a model running on the GPU; it now reports
  resident memory.
- HTTP framing is honest: 411 for a chunked body instead of reading it as
  empty, 413 for an oversized one instead of a bare connection reset, 431 for
  huge headers, 400 for a malformed `Content-Length`. A query string no longer
  404s the route, `HEAD` answers for the path actually asked for instead of a
  blanket 200, `/v1/models` carries `created`, and the first SSE delta carries
  the role. All of it is gated in `Tools/api_robustness.sh` (68 checks).

- Context length is documented and priced, and the cap is named for what it
  is. The 400 for a long prompt used to say "raise it with --max-context",
  a flag that could not go past the ceiling the server was already at; it
  now says the cap is the largest context measured so far (not a memory
  limit; context state is ~27 KiB per token) and what reading that prompt
  would have cost in time. `--max-context` above the ceiling is refused with
  the same explanation, on `serve` and `doctor`. The memory plan has a
  `context:` line and `/api/show` carries `max_context_tokens` and
  `est_prefill_s_at_max_context`; `doctor` ends with the wait before the
  first token by prompt length, and its tier table has a full-context column.
- Long prompts report progress: `run` and `serve` print the wait to expect
  and then one line per quarter for any prompt over 2k tokens.
- The prefill pass shrinks as the context grows (4096, 2048, 1024, 512 at
  about 4k, 14k, and 31k tokens), so a pass's query-by-key product never
  exceeds the largest one measured (a 4096-token pass finishing an 8,016-token
  prompt). Output is byte-identical at every pass size; the cost is some
  speed on the tail of a long prompt, and the plan's wait estimates include
  it. The never-measured 8192 pass is no longer a candidate.
- New `context-check` reads an N-token synthetic prompt through the real
  engine, reports seconds, tok/s, and peak memory against the plan, and stops
  before the machine swaps. New weights-free `prefill-schedule` prints the
  pass ladder and wait for any pass size. Gated in `planner_gates.sh`
  (bounded, floored, monotone, and equal to the doctor's wait) and one 2k rung
  in `verify.sh`.
- `pull`'s connection report counts the connections in use at once, one per
  session, instead of every distinct connection since the start; 0.2.1 could
  print "10 connections in use" for eight workers after two reconnects.
- `Tools/e2e_release.sh` expects the Ollama load acknowledgment for a chat
  with no messages, the 0.2.1 behaviour, instead of the 400 it asserted
  before; it was the one failing check of 31 against the installed 0.2.1.

## 0.2.2 — 2026-09-02

- Speculative decode pays, and ships. The draft head, `mtp.safetensors`
  (1.47 GB, sha256-pinned), is hosted on the weights mirror and pulled with
  everything else, so `--mtp auto` works out of the box on a large Mac. It is
  the manifest's one optional file: a source without it leaves the pull green
  with a notice and speculative decode off, `pull --verify` skips it when
  absent, and the startup check never asks to repair it. The weights are
  105.3 GB in 25 files.
- A rejected draft rolls back instead of re-running. The verify pass records
  the recurrent state after every position (the GDN recurrence stepped one
  token at a time, bit-identical to the fused kernel; conv windows sliced),
  so a rejection costs no model compute. Measured where auto enables the head
  (122 experts per layer, a quiet 48 GB Mac): ×1.24 decode with one draft
  (10.3 → 12.8 tok/s; ×1.33 on a code prompt, ×1.19 on a list, ×1.18 with the
  server's default sampling), up from ×1.17; ×1.20 at 57 per layer, up from
  ×1.12.
- One draft by default (was four). Four drafts lose at every size measured,
  ×0.88 even where auto turns the head on; one is best or tied everywhere and
  wastes the least on a rejection. `SLOTSTREAM_DRAFT_DEPTH` still overrides.
- The numbers are measured, not projected. The ×1.5–1.9 the 0.2.0 docs gave
  large caches assumed a five-token verify pass costs one token's pass; new
  hidden `mtp-passcost` measured 1.65 (a sixth of a pass per extra token),
  and auto's threshold reads 28 GB, not ~26. `mtp-check` bounds the reused
  speculative state's logits by the plain re-chunking band instead of
  comparing liveness, proves the recording pass exact against the batched
  one, and checks a rollback state by state against the plain path.
  `mtp-bench --sample` measures the sampled case.

## 0.2.1 — 2026-09-01

- `pull` opens the connections it claimed. Each of its eight connections is
  now its own URLSession: HTTP/2 multiplexes every request in a session over
  one TCP connection and ignores `httpMaximumConnectionsPerHost`, so every
  pull through 0.2.0 ran at one connection's speed — 25 to 40 MB/s from a home
  link 100 ms from Hugging Face, 72 from a gigabit datacenter link. Eight real
  connections measured 112 MB/s over a full install on that link (16 minutes)
  and 50 to 63 at home, and `pull` now prints the count it actually measured. The
  README's claim that Hugging Face caps the transfer near 55 MB/s was this bug
  seen from one link; it is withdrawn, as is the "R2 tested and rejected"
  verdict that rested on the same link (MEASUREMENTS.md, 2026-09-01).
- The Ollama CLI works again. 0.1.8's strict validator rejected the empty
  `name`/`system`/`template`/`options` the CLI's `/api/show` request always
  carries, so `ollama run` stopped before its first message. `/api/show` now
  accepts the deprecated `name` alias and empty overrides (non-empty ones stay
  a 400), advertises `capabilities`, chat/generate accept `keep_alive` and a
  null `options`, and generate accepts the empty `suffix`/`template` the
  CLI's one-shot mode sends (a non-empty suffix or template is still a 400).
  Ollama's documented "load" request (an empty prompt, or no messages), which
  the CLI sends when an interactive session opens, is acknowledged with
  `done_reason: "load"` instead of refused. Gated by `Tools/api_robustness.sh`
  with the CLI's exact request shapes.
- A weights directory reached through a symlink loads. Foundation refuses to
  list a symlinked directory, so `run` and `serve` failed with "couldn't be
  opened" while `doctor` and `pull --verify` worked; paths are now resolved
  once at the CLI boundary and in the shard index. Gated by `runtime-check`
  (weights-free) and a `verify.sh` run through a symlink.

## 0.2.0 — 2026-09-01

- Speculative decode with the model's draft head: `--mtp auto|on|off` on
  `run`, `serve`, and `doctor`. Auto enables it only at 120 or more experts per layer
  after its 1.6 GB charge, which raises the auto ceiling to 34.6 GB. Measured
  depth-1 accept rate 85.8%; ×0.96 at a 16 GB target, so it stays off there;
  the large-cache A/B is still pending.
- `Tools/mtp_convert.py` rebuilds `mtp.safetensors` (1.47 GB) from the
  official release with sha256 provenance; new `mtp-parity`, `mtp-accept`,
  `mtp-bench`, and `mtp-check` commands; `verify.sh` runs the MTP gates when
  the file is present.

## 0.1.10 — 2026-08-31

- Parity goldens ship in the repo, so a fresh clone can run the battery.

## 0.1.9 — 2026-08-31

- Installer and CI hardening; GitHub Actions runtimes updated.

## 0.1.8 — 2026-08-31

- Every weight file is checked against a sha256 manifest compiled into the
  binary; `pull --verify` covers all 24.
- Elastic drill and battery memory targets fixed; the `--memory-gb` promise
  re-verified and its measurements corrected.

## 0.1.7 — 2026-08-30

- Warm-decode estimates re-anchored on measurement; the planner no longer
  extrapolates past verified points.
- `Tools/e2e_release.sh`: acceptance run against the installed release.
- Live governor resize behavior observed and recorded.

## 0.1.6 — 2026-08-30

- Conversation prefix cache: follow-up turns prefill only what is new.
- Prefill pass size recalibrated.

## 0.1.5 and earlier — 2026-08-28 to 2026-08-29

- Serving robustness: every input that used to crash the server or corrupt
  its output is now a gated test.
- First public releases: the streaming engine, memory planner, `doctor`,
  `pull`, and the Ollama/OpenAI server. Details on the
  [releases page](https://github.com/carloslfu/slotstream/releases).