Cache Control

One authority

config/cache.toml is the only cache inventory and policy. Every independently accounted cache has exactly this common contract:

• description: what expensive work the bytes preserve;
• scope: disk, docker, or tart;
• max_size_bytes: the usage ceiling that triggers enforcement;
• warm_size_bytes: the retained target after crossing the ceiling;
• prune_strategy: lru, generational, ephemeral, none, docker, or tart.

Age and count fields are optional refinements. They never replace the common contract. Do not introduce storage-availability rails, warning thresholds, or a second config table. Cache decisions are based exclusively on owned usage.

This authority covers retained build and developer artifacts. Bounded runtime maps, connection pools, socket buffers and admission queues have a different lifetime: their shared capacity relationships live in config/data/runtime-limits.toml, are validated and generated by capsem-proto/build.rs, and are guarded by tests/citadel/test_runtime_limit_authority.py. Do not copy a runtime ceiling into cache code or put a runtime object cache in config/cache.toml. A runtime cache must remain bounded by the generated capacity, expose hit, miss, eviction and occupancy telemetry where those states exist, and preserve the logger DB boundary instead of caching ledger projections in a route or UI process.

The [cache] table owns every in-memory cache count and freshness window: DNS answers and positive/negative TTLs, TLS leaves, ledger query batches, SQLite prepared statements, service diagnostics and resume state, update checks, image-catalog refresh/retry and image-verification batching. Preserve their current behavior by importing capsem_proto::CACHE_*; a local literal, even when two crates currently agree, is a second authority. Add a new field, build-time validation, generated constant, benchmark runtime-limits field and Citadel consumer assertion together when a new bounded runtime cache is introduced. User overrides remain the typed-settings follow-up in issue #349.

Typed boundary

Programs use CacheRegistry, CacheRequest, CacheOperation, and CacheMutationResult from capsem_builder.cache. They select a stable cache ID and do not branch on storage mechanism or discover backend locations.

The registry owns backend dispatch:

• DiskBackend inventories repository cache stages through CachePaths;
• RuntimeBackend inventories and mutates Docker or Tart through their native adapters;
• Docker-on-Colima is still the docker cache. Colima is an implementation detail of that backend, never a separately billed cache;
• Docker image repositories are child cache IDs but use the same contract and request/result types as every other owner.

A producer that must place disk bytes asks CachePaths or the gate's cachelayout.stage_path() for its configured stage. It must not reconstruct a path from repository literals, environment variables, or a backend name. A consumer that only inspects or controls a cache uses CacheRegistry and never receives its path.

A shared cache stage must not contain or be contained by a prefix export. Exports are replaceable product views; shared stages are retained authorities. Overlapping them makes a successful export erase its own receipts and traps an exact-source run in a repair loop. VM generations live in the assets cache stage, while cache/target/assets remains only the assembled runtime view.

Pinned external executables use CachedToolPolicy and materialize(). The policy declares one HTTPS URL and SHA-256 per host platform; the primitive downloads once, verifies before atomic publication, installs mode 0555, and returns a typed hit/miss result. A subsystem must not invent its own downloader or tool directory.

Short-lived results from mutable services use CleanVerdict, reusable(), and record_clean(). The subject digest covers the complete typed policy and exact input bytes. Only success is recorded; malformed, future-dated, expired, or non-matching receipts are misses. The owning cache stage supplies the maximum age, count, warm size, and maximum size.

CachePaths is loaded through load_paths(). The loader resolves the policy-owned authority_environment; a gate prefix exports that variable once so every producer sees the outer shared cache even when its output path crosses a prefix symlink. Producers never read or interpret the environment variable themselves.

With no explicit authority, load_paths() resolves a linked Git worktree to the checkout that owns the common .git directory. All branches of one local repository therefore share one cache inventory. Do not create a cache beside each worktree or pass a worktree path as implicit storage. Administrative cleanup of a retired location uses the typed CLI's explicit --repository with --policy-repository; ordinary operators stay on just cache.

All models are strict, frozen Pydantic models with extra="forbid". Keep every module at or below 300 lines and put tests in the matching test module.

Operator interface

The public entry point is just cache. These are the complete supported inspection and lifecycle commands:

just cache stats                 # usage, warm/max contract, state, description
just cache stats --json          # typed machine-readable inventory
just cache stats --offline       # disk inventory without native runtime calls
just cache contract <cache-id>   # the five-field common contract
just cache verify                # containment and complete disk accounting
just cache prune [cache-id]      # deterministic preview; defaults to all
just cache prune <id> --apply --reason "why"
just cache enforce <cache-id> --reason "why"
just cache clean <cache-id>      # explicit cold-clean preview
just cache clean all --apply --reason "why"

prune follows ordinary age/count policy and only recovers to warm_size_bytes after usage crosses max_size_bytes. enforce is an applied preflight and fails if protected state prevents compliance. clean is the explicit cold operation; it still preserves active leases and protected generations.

Docker's per-repository generation retention refines the total capacity contract; it does not exempt inactive images from pressure. After ordinary reclamation, total-pressure eviction selects owned, inactive, noncurrent, unprotected images by last use until warm size is reached. Retire all owned aliases and count image bytes once; active/current/protected images retain their protection, and an unavoidable overage remains an error.

Cargo's ordinary retention selects whole compilation units under each cargo_target_roots entry: a unit's fingerprint, build output, libraries, dep-info, executables and verified signed copies share one metadata hash and are removed together, fingerprint first, least recently used first. Every gate prefix salts workspace units with its checkout path, so incremental-only retention let stale prefixes fill the stage until enforcement refused every run (issue #205). Paths Cargo does not name by unit, such as uplifted binaries, count toward capacity but are never selected.

Retention is ordered by owner, not only by clock (issue #276). The workspace wrapper links each unit's fingerprint directory to the checkout that compiled it (capsem-owner, a symlink, so reading it moves no atime and measuring never counts it). A unit whose checkout is gone is reclaimed on every prune, whatever its age. Under pressure other checkouts' units go first, then shared or unrecorded ones (third-party crates, sccache hits, units from before the link existed), and the working set -- the calling checkout (--policy-repository, the bounded command's root) and the cache authority -- last. Plain LRU made the next build of the current head cold whenever other checkouts had compiled since. The copies themselves stay: the salt cannot go until Cargo freshness is content-based, and -Zchecksum-freshness is still unstable on the pinned toolchain. Native Cargo output locks protect the whole stage and are acquired again through deletion; even an explicit cold clean preserves their lock inodes.

The Cargo maximum is held around compilation, not by any one step. Every exclusive gate command whose plan claims workspace_binaries holds CargoCacheBound (gate/cachecontrol.py), which enforces before the first step and after the last, failed runs included; run-bounded-command.py enforces before and after direct Cargo. Do not add an enforcement step to a fragment: a prerequisite of the host build left clippy, coverage and every other compile unbounded, and nothing enforced after any compile, until the target reached 237 GB against 180 GiB.

The objects store (object_store = true) is retained the same way, one generation at a time: a component receipt together with the objects no other receipt names, receipt first. An object two receipts share belongs to neither and is collected once no receipt names it; an unreferenced object (a canonicalized package or release view, whose bytes live on in the hardlinked view) is its own generation. componentcache.restore treats a missing or corrupt object as a miss, never an error, and a hit moves the receipt's clock; never touch an object's times, which are its hardlinked outputs' times. Every reader and publisher holds leases.shared_use(paths, "objects"), so a prune never removes bytes in use. Before this the stage was none, grew past its cap, and failed every prune on the machine.

A leased ephemeral stage (test-temp) is scratch, not retention: a generation lives exactly as long as a process holds its lease. The owner leases run-<pid> before creating it and, at exit, releases the lease and removes the run and its lease through operations.reclaim_generation, the same guarded, journaled path a prune takes. A run no process leases (its owner was killed) has no live owner, and prune reclaims it whatever its age or size; a lease file whose generation is gone is collected too, and never counts toward maximum_count. Removal takes each lease exclusively, so a reused pid that leased after the plan keeps its directory (ApplyResult.busy). Before this, a dead run aged like a retained generation below a 200 GiB maximum: one ~7 GB run per release precheck and 1,900 lock files filled the disk while prune offered 21 MB.

That liveness does not depend on the authority, so such a stage is swept whole (cache/namespaces.py, issue #272): every other sha256(authority)[:8] namespace's runs and leases are entries keyed <namespace>/<run> under the same lease guard, and an emptied namespace is removed with rmdir, never recursively. Only a real directory with a namespace name is entered; a symlink, a pre-namespace capsem-test-* directory, or a file that is not a run or a lease is never touched. Prune once saw only its own namespace, and about 350 retired ones held 2.4 GB.

The source-keyed Python stages (python-pycache, python-pytest) gain a generation for every Python-source state any checkout launches a gate or bounded command from, and nothing but a complete gate's package and install rails used to prune them: python-pycache reached 5.4 GB against 2 GiB. The launcher that creates a generation leases it and then enforces both stages (pythonenv.bound_source_keyed); a warm launch pays nothing.

Retained cache lifetime ends only through these typed operations. Do not add consumer-boundary releases, post-test eviction hooks, or other subsystem lifecycle paths that bypass the owner's warm/max/age/count policy.

Every applied mutation is exact, ownership-scoped, and journaled. Do not use broad docker system prune, direct Tart deletion, rm over cache roots, or backend-specific cache CLI commands. Foreign Docker/Tart resources are visible only as native totals and are never deleted by Capsem.

Repository-generation reclaim is anchored by an exact current tag. Docker's bulk image listing can briefly lag a completed BuildKit import, so the Docker adapter verifies that exact tag directly and supplies a protected typed resource to the planner. Never turn this into an unanchored retry or allow a missing exact inspection to prune older generations.

Prune and enforcement never remove an image a checkout's gate uses. The gate declares it: reclaim-image --apply records its --keep tag for the calling checkout, the host builder's require step reclaims around :latest, and the image backend records each architecture's dependency image (capsem_builder.cache.dockercurrent, stored in [control.docker] current_stage). A record is live while its checkout exists and it is younger than the Docker runtime's maximum_age_hours; a newer tag for the same checkout and slot supersedes it. Scanning a runtime requires paths= so every planner sees live records as current. Newest-by-creation is not current: a BuildKit cache hit keeps the old timestamp, one repository holds a tag per architecture, and two checkouts each have their own. On 2026-09-29 guessing it made enforcement evict the release proof's current kernel-dependency image and the host builder; each rebuild re-ran the slowest network phase. Pressure on current images is reported as a violation, never resolved by evicting them. Only an explicit clean may remove a current image.

Test efficiency

Start with the cheapest evidence that owns the change:

just fast-test
just focus-test <owner>

just test <source-commit> is the complete local proof a release requires. Its journal reuses an exact successful source identity, and its graph carries valid artifact frontiers instead of rebuilding them. Low-impact paths are routed by [test_admission]; forcing complete proof repeatedly is rate-limited by commit distance. Release commands consume its journal: they refuse a source without a passing run for the same commit or tree.

After a failed or interrupted full attempt, automatic retries are refused. Use focused checks; obtain explicit approval for each reasoned full retry. The existing run journals supply this guard, not another history store.

sccache --stop-server acknowledges shutdown before its daemon drains clients and exits. The last cache owner retains startup and exclusive cache-use locks through a bounded, read-only process-exit wait. Test this with a real open UNIX connection; an immediate PID assertion alone once passed locally and raced in CI. Do not sleep in the test or let the observer reacquire its caller's locks.

Working-tree gates derive their private prefix name from the exact source digest. Do not replace it with a random run ID: Cargo fingerprints contain absolute source paths, so random prefixes turn an unchanged repeat into a rebuild. The gate owns sccache as a scoped CompilerCache resource, exports SCCACHE_BASEDIRS (plural), and uses client-side mode. Gates and bounded compiler commands hold the cache stage's mutation locks shared. Startup alone is serialized by the configured startup lock; teardown and stale-server reaping require exclusive cache use. Another live compiler preserves the server, and only the last gate/bounded compiler user stops it. Kernel lock release lets the next reaper collect a server abandoned by a killed owner, without process-age guesses. Do not manage its daemon in shell.

The bounded wrapper recognizes Cargo, rustc and sccache through configured wrappers. An opaque Python or shell build driver uses --compiler-cache for shared cache lifetime without a machine lease; --machine-lease includes it when complete isolation is intended. Ordinary inspection and cache-control commands hold no compiler lease, so cleanup cannot block itself.

Cargo runs a workspace unit as sccache <workspace wrapper> /abs/rustc .... sccache 0.17 drops that compiler argument only when it is spelled rustc, so it parsed the absolute path as a second input ("multiple input files") and ran every workspace unit uncached (issue #277). The rustc workspace wrapper therefore hands a rustc unit to $RUSTC_WRAPPER itself when that is sccache; test_cargo_workspace_cache.py proves the second compile is a hit. Watch sccache --show-stats for hits on workspace crates, not just a low non-cacheable count: the outer call still counts as non-cacheable.

Gate commands and bounded builds compile with CARGO_INCREMENTAL=0 ([toolchain] cargo_incremental in config/gate.toml, exported by gatelaunch.contained_environment and cachetooling.compiler_environment). That is the measured workload-specific reason: a prefix is named by its source digest and workspace units are salted by its path, so an incremental session is never read back, an agent worktree lives one session, and on 2026-09-29 the sessions were 75-83 GB of a 171-208 GB target. sccache also refuses incremental output. Keep the dev profile at debug = "line-tables-only" for the same reason: full DWARF was 77% of every test executable (tests/citadel/test_hot_build_contract.py).

Keep the checkout-local build.rustc-workspace-wrapper configured in Cargo. Cargo includes its resolved path in workspace artifact keys, separating source trees while third-party dependencies stay shared. Without it, an older snapshot can be incorrectly considered fresh after a newer worktree compiled into the shared target, even with sccache disabled. Test both output correctness across trees and warm reuse on return; clearing the cache would hide this failure.

cargo clippy forces RUSTC_WORKSPACE_WRAPPER to its own clippy-driver, one path for every checkout, so plain clippy output in the shared target loses that key and a checkout can take another's lint result as its own (a false green or red). The gate and run-bounded-command.py therefore run clippy as cargo check through the checkout-local toolchain.clippy_workspace_wrapper (gate/clippyrun.py). Never add a literal cargo clippy gate step; build_system/tests/cache/test_clippy_checkout_key.py refuses one and proves per-checkout verdicts on the pinned toolchain.

Never make normal cache reuse opt-in. Do not clean caches to diagnose a product failure unless cold-state behavior is itself the subject under test.

Changing policy

1. Run just cache stats --json and preserve the before inventory.
2. Change the owning cache entry in config/cache.toml.
3. Update Pydantic validation and focused cache tests together.
4. Run just cache verify and the cache, gate, and Citadel suites.
5. For a size change, run one complete gate with permissive maxima, record the post-run inventory, and derive warm/max values from observed retained work plus measured generation needs. Validate the ratchet with an exact-repeat run so it proves reuse rather than merely fitting a cold build.

Do not infer a cache limit from filesystem capacity: retention decisions stay on owned usage. The maxima do answer to the disk as a set, through [budget] in config/cache.toml (capsem_builder.cache.budget). Every stage and runtime maximum together, plus headroom_bytes, must fit filesystem_fraction of the declared minimum_filesystem_bytes (checked when the policy loads) and of the real cache filesystem (checked by every exclusive gate command before it waits for the machine lock; an optional runtime that is not installed, such as Tart on Linux, counts zero; a GitHub-hosted runner is exempt). On 2026-09-29 the maxima summed to 839 GiB on a 484 GiB disk: every cache honoured its contract and the disk filled four times in two days. Raising one maximum means lowering another or raising the declared floor, deliberately.

Debugging

Read cache/target/gate-runs/DIGEST.md before reporting gate state. Then use:

uv run --project build_system --frozen capsem-gate runs last --failed
uv run --project build_system --frozen capsem-gate runs trend --step <label>
just cache stats --json
just cache contract <cache-id>

If runtime inventory fails, diagnose Docker/Colima or Tart through /dev-setup. Do not add fallback accounting or silently treat a required runtime as empty. OCI image creation time is optional. Keep its existing zero unknown-age sentinel and complete byte accounting; use a missing-key-safe projection for both bulk inventory and exact anchors rather than fabricating a creation date.