Rust Patterns

Data paths: zero-copy first, no speculative hashing or JSON

Read data-path performance before changing transport, admission, serialization, hashing or buffer ownership. Prefer moving or borrowing bytes and one bounded encoding into the final frame. Hashing and JSON require a concrete, documented contract; neither is a default safety step.

Async / non-blocking

Capsem uses tokio for all async I/O. The MITM proxy, vsock manager, and file monitor are all async.

Publish matching in-memory ownership inside the serialized blocking admission, before releasing its I/O permit. A reader can observe the durable record before the async caller resumes after spawn_blocking; publishing its ticket after .await exposed valid managed-session claims to recovery_pending. The controller regression pauses that continuation and claims through the real store.

Never block the tokio runtime

Long-running synchronous work (FUSE request processing, disk I/O, compression) must run on a dedicated thread via tokio::task::spawn_blocking or a dedicated std::thread. Blocking inside a tokio task starves other tasks.

The VirtioFS FUSE server runs on its own thread for this reason -- FUSE ops are synchronous by nature (read, write, lookup) and can't be made async without significant complexity.

Blocking-in-async anti-pattern (systemic -- audit, don't spot-fix)

Any code path that does blocking I/O inside an async function or while holding a tokio::sync::Mutex is a bug. This causes the tokio worker thread to stall, freezing the entire gateway, UI, or network stack until the blocking operation completes.

What counts as blocking I/O:

• std::process::Command (subprocess execution)
• std::fs::* (read, write, copy, remove_dir_all, create_dir_all)
• walkdir::WalkDir (directory traversal)
• blake3::Hasher on large data (hash computation)
• std::thread::sleep

The fix pattern -- in capsem-service, route handlers go through ServiceState::off_worker (crates/capsem-service/src/blocking.rs), and crates/capsem-service/src/tests/async_io_contract.rs refuses direct blocking calls. Elsewhere, the shape is:

let result = tokio::task::spawn_blocking(move || {
    let rt = tokio::runtime::Handle::current();
    rt.block_on(async {
        let mut guard = mutex.lock().await;
        sync_blocking_work(&mut guard)
    })
}).await.unwrap_or_else(|e| /* handle panic */);

Example site: the fork clone (crates/capsem-service/src/vm_files/fork.rs) runs clone_sandbox_state -- fsync, clonefile, and a full tree walk -- on the blocking pool. Any new tree copy, hash, or directory walk reachable from an async handler needs the same treatment.

Channel patterns

• tokio::sync::mpsc for producer-consumer (vsock data flow, telemetry events)
• tokio::sync::broadcast for fan-out (serial output to multiple subscribers)
• tokio::sync::oneshot for single-response request-reply (control messages)

Logger DB boundary

capsem-logger owns SQLite connection/thread/storage mechanics. Rust service, gateway, MCP, UI, and benchmark code must not call rusqlite::Connection::open or DbReader::open directly for telemetry/security ledgers, and must not add service-owned projection caches.

Callers may own query intent, but the DB object owns execution:

db.ready().await?;
db.query(sql, params).await?;
db.write(event).await?;

db.write(event).await is an accept boundary: the DB-owned producer buffer has taken responsibility for the event. Read-after-write tests must use the DB flush barrier or shutdown/reopen before asserting route-visible rows. Never add route sleeps, route projection caches, or caller-owned SQLite reads to force visibility.

Do not hide route SQL by adding route-specific helpers to DbWriter; the DB writer is not a product route registry. Put connection threads, mem/disk tables, batching, flushing, WAL tuning, and future FTS5/search in the DB layer. Empty tables are fine. Missing tables or columns are schema contract failures and must not be converted into empty results.

Coalescing buffer

Terminal output uses a CoalesceBuffer (8ms window, 64KB cap) to batch small vsock reads into larger writes. This prevents xterm.js from choking on thousands of tiny updates. The pattern: accumulate into a buffer, flush on timer or size threshold.

Graceful shutdown

Use tokio::select! with a cancellation token or shutdown signal. Every long-running task must respect shutdown. Dangling tasks after VM exit cause resource leaks.

Cross-compilation

Guest binaries target aarch64-unknown-linux-musl and x86_64-unknown-linux-musl. Key gotchas:

• Platform-specific types: libc::ioctl request param is c_ulong on macOS but c_int on Linux. Use as _ to let the compiler infer the correct type.
• Linker: .cargo/config.toml sets linker = "rust-lld" for both musl targets.
• No std dependencies: musl builds are fully static. Avoid crates that link to system libraries.
• Test on both: cargo check --target aarch64-unknown-linux-musl catches cross-compile errors without needing to boot a VM.

Error handling

• Use anyhow::Result for application code (capsem-app, scripts)
• Use thiserror for library errors in capsem-core (typed, matchable)
• Propagate errors up, don't swallow them. If a function returns Result, the caller must handle it.
• Log errors at the point where you have context, then propagate. Don't log AND propagate (causes duplicate log lines).

Host Unix boundary

Reusable host Unix mechanics belong in the small modules under capsem_foundation::unix; application crates consume their typed APIs instead of calling nix or libc directly. This keeps ownership, EINTR handling, errno classification, CLOEXEC, no-follow behavior, and race checks identical across the CLI, service, process, guard, credentials, and core.

• Accept BorrowedFd and return OwnedFd or File. Never return a raw descriptor whose close responsibility is implicit, and never pair ManuallyDrop<File> with a manual close.
• Expected races are typed outcomes: a missing process and lock contention are not generic errors. Keep expected disconnects quiet; log unexpected failures once where the caller knows the operation, object identity, and errno.
• Use unix::contained::ContainedDir for guest-influenced paths. Its descriptor-relative traversal, constrained open options, and regular-file checks must remain in the same syscall sequence; do not put a path-based check in front of a separate path-based action.
• Use unix::fs for owner-only directories, no-follow regular-file access, filesystem capacity, atomic private publication, permission bits (set_mode), durability (sync, sync_before_barrier, sync_filesystem) and copy-on-write cloning (clone_file_into, copy_sparse). Use unix::tree_clone to copy a directory tree a guest can write. Use unix::lock for advisory locks, including synchronous read-modify-write sections.
• When foundation lacks a primitive, add it to foundation with its tests; never reach for nix or libc at the call site "just this once".
• Inside foundation, nix is the syscall layer and unix::errno::io converts its errors. A raw libc:: call is allowed only where nix cannot express it (no wrapper, or a platform gap such as MSG_NOSIGNAL on macOS), and each one is listed with that reason in tests/citadel/unix_boundary_debt.toml.
• Put each primitive in its own module with tests in the sibling tests.rs. Cover errno preservation, ownership after the source is dropped, symlink and path-replacement races, special files, exact modes, and concurrent callers.

When transferring connected sockets with unix::router_channel, retain the original socket until a request-bound receipt confirms that the receiver owns and validated its descriptor. A completed sendmsg only queues SCM_RIGHTS; Darwin may collect sockets with no owner outside that queue. Bound the receipt wait, reject wrong identities, directions and extra descriptors, and preserve scope cancellation and positive child reaping. The native MCP regression is upstream_broker/mcp_spawn/tests.rs: it delays adoption, checks that the original stays alive, and proves it is released after the receipt. Keeping an extra socket for the child's entire lifetime masks disconnects and leaks descriptors. The same rule covers startup policy, proxy traffic/MCP/trace hints, ledger clients, gateway connections and the aggregator's owner connection. Auditing only the native MCP bootstrap missed these paths and left a macOS policy-update broken pipe. tests/citadel/test_descriptor_adoption_lifetime.py guards the early-drop sites, including private cable handoff: its table reference can be replaced by a reconnect before adoption. Gateway and upstream grant tests drop the received copy before acknowledging to prove the sender still owns its original, then assert EOF after adoption to rule out retention leaks.

Direct Unix calls remain valid only in modules that implement a kernel or platform ABI rather than reusable host policy, such as KVM/FUSE and guest audit plumbing. tests/citadel/test_unix_boundary.py and its exact domain-ABI inventory enforce that distinction; extending the inventory requires explicit review, not a convenience exemption. The same guard inventories every raw libc:: call and extern "C" binding inside foundation itself, rejects a new one that nix already wraps, and catches aliased imports (use ::libc as sys), extern "C" { fn ... } blocks and renamed or table-form manifest dependencies. Before it did, foundation's own internals were unscanned and raw calls with nix wrappers accumulated there unnoticed.

Guest RAM and VirtioFS performance

Before changing guest RAM, KVM storage or OCI boot, read Guest RAM and VirtioFS performance. Keep RAM and KVM slot offsets aligned to 2 MiB and request transparent huge pages through capsem_foundation::unix::guest_memory::GuestRam. A regular 4-KiB-backed mapping once made kernel startup take 1.1 seconds and a 352-MB EROFS scan wait 1.5 seconds in reads. The allocation and slot regression is guest_ram_and_kvm_slot_offsets_preserve_huge_page_alignment in hypervisor/kvm/memory/tests.rs; alignment alone is insufficient without the per-mapping huge-page hint.

Bidirectional I/O -- thread per direction

When bridging two blocking file descriptors bidirectionally (e.g., TCP socket to vsock in net_proxy.rs, or master PTY to vsock in capsem-pty-agent), doing both reads and writes in a single thread using poll(2) causes deadlocks. If both outgoing buffers fill simultaneously, a single thread blocks on writing and stops reading, creating mutual lockup. Always spawn a dedicated thread for at least one direction (std::thread::spawn for fd_b -> fd_a while the main thread handles fd_a -> fd_b).

Serde -- avoid serde_json::Value on LLM payloads

The MITM proxy and ai_traffic parsers handle massive HTTP payloads (megabytes of tool calls, histories, images). Parsing these into serde_json::Value does full DOM allocation, which is inefficient and risks memory exhaustion.

Rules:

• Define targeted structs with #[derive(Deserialize)]. Serde skips and discards fields not in the struct without allocating memory for them.
• For struct fields that hold large, unconstrained JSON (tool call arguments, function responses, full model outputs) and are only converted to strings: use Box<serde_json::value::RawValue> instead of serde_json::Value. RawValue keeps the JSON as an unparsed string slice -- zero DOM allocation. Access the raw JSON string via .get().
• Never add serde_json::Value fields to structs that parse LLM request/response bodies. If you only need a string representation, use RawValue. If you need to traverse nested fields, use a typed struct.
• Remove unused fields from deserialization structs -- they still force Serde to allocate.

Example -- before (bad):

struct FunctionCall {
    name: Option<String>,
    args: Option<serde_json::Value>,  // full DOM parse of potentially huge args
}
// later: let arguments = fc.args.as_ref().map(|v| v.to_string());

After (good):

struct FunctionCall {
    name: Option<String>,
    args: Option<Box<serde_json::value::RawValue>>,  // zero-copy string slice
}
// later: let arguments = fc.args.as_ref().map(|v| v.get().to_owned());

Memory and resource management

• File handle limits: VirtioFS caps at 4096 open file handles, returns EMFILE beyond that.
• Read size limits: VirtioFS clamps reads to 1MB, gather buffers to 2MB.
• Safe deserialization: read_struct returns Option<T> with bounds checks in all builds (not just debug).
• irqfd for interrupt delivery: Guest interrupt signaling uses irqfd to avoid cross-thread syscall overhead.

Concurrency patterns

• Shared runtime caches: Use capsem-cache primitives instead of a local map and eviction policy. Import capacities and freshness windows from the generated capsem_proto::CACHE_* constants; their only numeric source is [cache] in config/data/runtime-limits.toml.
• Arc for shared state: VM state, proxy config, and telemetry handles are Arc-wrapped for sharing across tasks.
• Per-connection tasks: The MITM proxy spawns a new tokio task per connection. Each task owns its TLS state and upstream connection. No shared mutable state between connections.
• Config-owned runtime ceilings: Network concurrency and in-memory cache policy are owned by config/data/runtime-limits.toml and generated into capsem-proto. Import the shared protocol constant in host and guest consumers; never duplicate a numeric capacity in individual crates. Ledger queue and batch capacities derive from the same supported-client ceiling, and bounded coalescing waits use the generated duration rather than a local sleep literal. Extend the runtime-limit Citadel guard for every new consumer. See /dev-mitm-proxy for the load proof required when a limit changes.

Host-serialization locks for per-host critical sections

When a service orchestrates N sibling child processes on a single host and some operations cannot safely run two-at-a-time on that host -- whether because of a framework constraint (Apple VZ save/restore) or because of shared-resource starvation (VZ teardown + WAL checkpoint + virtiofs drain all competing for main-thread and I/O bandwidth) -- park a tokio::sync::Mutex<()> on the service's shared state struct and acquire it at the top of the handler for the whole duration of the critical section. Mutex<()> isn't a weird construction: the unit value is the lock-token, the type signals "pure serialization, no protected payload". Semaphore::new(1) is equivalent -- pick one and stay consistent.

Current instances in crates/capsem-service/src/main.rs:

• save_restore_lock: serializes Apple VZ saveMachineStateToURL / restoreMachineStateFromURL across sibling VMs. Concurrent save/restore corrupts the VirtioFS ring state on the unlucky VM, surfaces as ext4-on-loop0 I/O errors after resume. Held through handle_suspend (IPC + child-exit wait) and handle_resume (spawn + wait_for_vm_ready). See web/docs/src/content/docs/gotchas/concurrent-suspend-resume.md.
• shutdown_lock: serializes VM teardown across handle_delete / handle_stop / handle_purge / handle_run. Without it, N concurrent deletes under load starve each other of the bandwidth each capsem-process needs to exit cleanly within the 1s fast-path budget; past the budget the service SIGKILLs mid-checkpoint and leaves a non-empty session.db-wal. Held through shutdown_vm_process for the whole SIGTERM + wait_for_process_exit window.

When to reach for this pattern:

• Symptom is "works solo, fails under concurrency on the same host."
• Root cause is a per-host resource, not per-VM: Apple VZ main thread, virtiofsd, DbWriter checkpoint, APFS fsync.
• Production runs exactly one service per host per user, so an in-process tokio mutex is enough -- no need for a file-lock or distributed primitive.

When NOT to reach for it:

• If the contention is per-VM (two handlers acting on the same VM), protect the VM entry in instances: Mutex<HashMap<...>> instead.
• If the "contention" is really a durability race (writer thread hasn't flushed), the right fix is usually the signal-handler explicit-cleanup pattern below, not another serialization lock.

Signal-driven explicit cleanup for background-thread owners

Any long-running Rust process that owns background threads (SQLite writer, the fs-monitor scan loop, MCP aggregator subprocess, vsock relay) and runs under a bounded SIGTERM-to-SIGKILL budget must NOT rely on Drop + tokio-runtime-drop ordering to finish cleanup. On SIGTERM, hand owned resources to the signal handler and drain them synchronously BEFORE letting the main run loop return.

Symptom when this is missing: under concurrent teardowns on one host, the service SIGKILLs a child mid-checkpoint or mid-flush. Visible as session.db-wal left non-empty, missing fs_events rows, dangling aggregator subprocesses. Works solo, fails under -n 4.

Concrete primitives in this tree:

• DbWriter::shutdown_blocking(&self) — takes the stored mpsc sender, joins the writer thread, runs the final PRAGMA wal_checkpoint(TRUNCATE). Arc-safe: other Arc<DbWriter> clones remain valid but their writes become no-ops. Idempotent. Drop delegates to it.
• FsMonitor::shutdown_and_join(&self) — sends on the shutdown channel so the scan loop runs one final scan and emission, then joins the thread. Bounded at two scans and two emissions. Must run BEFORE DbWriter shutdown, because fs_events fan into DbWriter.
• CAPSEM_TEST_SLOW_CHECKPOINT_MS — test-only env var in writer_loop that inserts a sleep before the final checkpoint. Use in tests that need to distinguish explicit cleanup from implicit runtime-drop ordering.

Canonical wiring in crates/capsem-process/src/main.rs:

struct Shutdown {
    db: Option<Arc<DbWriter>>,
    fs_monitor: Option<FsMonitor>,
}

impl Shutdown {
    fn drain_blocking(&mut self) {
        // fs_events fan into DbWriter -- flush fs_monitor first.
        if let Some(m) = self.fs_monitor.take() { m.shutdown_and_join(); }
        if let Some(db) = self.db.take() { db.shutdown_blocking(); }
    }
}

// Populate as owners are constructed:
shutdown.lock().await.db = Some(Arc::clone(&db));
shutdown.lock().await.fs_monitor = Some(monitor);

// Signal handler drains through spawn_blocking, then stops the run loop:
rt.spawn(async move {
    /* wait on SIGTERM/SIGINT */
    let mut owned = std::mem::take(&mut *shutdown.lock().await);
    let _ = tokio::task::spawn_blocking(move || owned.drain_blocking()).await;
    unsafe { core_foundation_sys::runloop::CFRunLoopStop(...); }
});

Key properties:

1. Deterministic order. The drain order is explicit (fs_monitor -> db), not "whatever reverse-declaration-order Drop happens to give us after tokio aborts tasks."
2. Synchronous join. The handler waits for each background thread to finish. No "hope the task finishes before the runtime drops."
3. Run loop stops last. CFRunLoopStop (macOS) fires only after drain returns. Main returns afterwards; the remaining tokio-runtime drop is now a no-op fast path because the heavy work already completed.
4. Arc-safe shutdown APIs. shutdown_blocking(&self) works through a shared Arc<DbWriter> — callers don't have to chase down every clone. Use std::sync::Mutex<Option<Sender>> internally; the hot-path write() clones the sender under the lock and releases it before .await.

When to reach for this pattern:

• The process has std::thread::spawn or tokio::task::spawn_blocking workers that run durability-critical work on shutdown (WAL checkpoint, queue flush, child-process wait).
• A parent sends SIGTERM then SIGKILLs after a short, fixed budget.
• Today's cleanup relies on Drop running inside tokio task abort — i.e., you can't draw a line between "cleanup finished" and "run loop exited."

Call out when NOT to use it:

• One-shot CLIs that exit on natural task completion (no run loop, no signal window).
• Workers whose only side effects are in-memory (no durability to lose).

When adding a new long-running process or a new background-thread owner, wire it through Shutdown from day one. Don't ship a new binary that "should be fine because Drop will run" — under load, Drop won't run in time.

Logging

• tracing crate with FmtSpan::CLOSE for timing spans
• RUST_LOG=capsem=debug for full boot timing breakdown
• RUST_LOG=capsem=info for top-level only
• Use structured fields: tracing::info!(domain = %domain, status = %code, "request completed")

One rule, one function

A rule that lives in its callers is a rule that will be wrong in some of them. Resolution, precedence, and lifetime rules belong in a function; callers call it.

This is the single most expensive pattern in this codebase's history. Two instances, both found the same day:

Both look locally correct in every copy. That is what makes them expensive: nothing is visibly wrong at any one site, and a fix applied to one never reaches the others.

Established wrappers -- use them, do not re-derive:

build_system/tests/gate/test_path_and_log_wrappers_are_mandatory.py enforces the first three.

When unifying, take the better implementation, not the first one. support_bundle's tail reader seeked to each file's end; the shared one read whole files. Guest console output is guest-controlled, so read-whole let a chatty VM decide how much memory capsem support allocated. The shared function now seeks.

Two copies that agree today are still a bug. checkpoint_complete_path existed in capsem-process and capsem-service, identical except that one hardcoded the fallback name and the other used a constant. Changing that constant would have left the process writing a resume marker the service never looked for, with nothing wrong at either site.

Lessons learned

1. Content-Encoding: Always handle response decompression generically. Gzip compressed SSE responses caused NULL telemetry because the parser got binary garbage. Never strip Accept-Encoding as a workaround.
2. Platform type widths: as _ is your friend for cross-platform libc calls. Explicit casts (as c_ulong) will fail on the other platform.
3. Debouncer timing: If a VM shuts down before debounced events flush, telemetry is lost. Add sleep 1 in test commands, or use explicit flush on shutdown.
4. VirtioFS whiteouts: Apple VZ's VirtioFS doesn't support mknod, so overlayfs can't use it directly as upper. The ext4 loopback workaround provides full POSIX.
5. setsid for controlling terminal: Without setsid, the PTY has no foreground process group and Ctrl-C (SIGINT) is not delivered. capsem-init uses setsid to fix this.
6. serde_json::Value on LLM hot path: Three ai_traffic struct fields (ResponseInfo.output, FunctionResponse.response, FunctionCall.args) used serde_json::Value for large payloads that were only stringified. This forced full DOM allocation on every streaming request. Fixed by removing unused fields and switching to Box<serde_json::value::RawValue>.
7. Prefer syscalls over subprocesses: std::process::Command costs 5-30ms per spawn (fork/exec). If a syscall does the same thing, use it. Example: cp -c -R for APFS clonefile was 20-30ms; a direct clone syscall is <1ms. capsem_foundation::unix::fs::clone_file_into uses fclonefileat on macOS and the FICLONE ioctl on Linux -- no subprocess. Always check if the OS provides a syscall before reaching for Command.
8. Blocking I/O in MCP tool handlers: Tool handlers once ran blocking I/O (clonefile subprocess, walkdir, blake3) directly on tokio worker threads while holding a tokio::sync::Mutex, so tool calls hung from the model's perspective. Fixed by wrapping in spawn_blocking everywhere; see the anti-pattern section above.
9. Clone guest-writable trees through descriptors, never paths: Everything below session_dir/guest/ is guest-controlled and may change mid-copy (an entry swapped for a symlink to a host path). Fork/create-from therefore go through capsem_core::session::clone_sandbox_state, which composes capsem_foundation::unix::tree_clone: descriptor-relative O_NOFOLLOW opens, exclusive destination creation, symlinks recreated but never followed, special files skipped, setuid/setgid/sticky bits dropped, and a non-regular rootfs.img refused. File contents clone from the open descriptor via unix::fs::clone_file_into (shared extents, or a sparse copy when the filesystem cannot share them). Never add a path-based std::fs::copy/walkdir copy of guest state. A CoW clone is metadata-only, so flush rootfs.img before cloning (flush_system_overlay) or the clone captures stale data.
10. Platform-gate all macOS-only APIs: Any code using macOS-only symbols (libc::clonefile, Apple framework bindings, etc.) must be wrapped in #[cfg(target_os = "macos")] -- both the struct/impl and the tests. The Linux app build (Tauri deb/AppImage) compiles the full workspace; ungated macOS symbols cause cannot find function errors on Linux CI. This burned v0.14.7: a clone helper used libc::clonefile without a cfg gate. Rule: when adding platform-specific code, gate the definition, the impl, and the tests.
11. Readiness gates must reflect actual state: handle_ipc_connection responded to Ping with Pong the moment the UDS socket existed -- before vsock connections, boot handshake, or command handler spawn. wait_for_vm_ready treated Pong as "ready", so exec commands were sent to a process that couldn't handle them yet, blocking silently in a channel until setup_vsock finished. Tests masked this with wait_exec_ready() client-side retry loops, creating a double-wait: 30 client retries x 30s server wait each. Fix: Arc<AtomicBool> (vm_ready) gated by setup_vsock after BootReady; IPC handler only sends Pong when the flag is set. One wait, one place -- the server waits; the client calls once. When adding any new IPC readiness check, never respond "ready" based on socket existence alone; check actual process state via a shared flag or state enum.
12. VirtioFS and FSEvents: Apple VZ VirtioFS guest writes bypass macOS FSEvents (the kernel's file notification subsystem), so notify::RecommendedWatcher silently drops guest-originated events and a host directory shared into a guest has to be polled. Do not reach for notify::poll::PollWatcher to do it: it walks with follow_links(true) hardcoded (notify-7.0.0/src/poll.rs) and its configure is a no-op, so one guest ln -s / workspace/evil turns every scan into a walk of the host's entire filesystem. Own the loop instead, as crates/capsem-core/src/fs_monitor.rs does: a walkdir walk with follow_links(false), a snapshot of (kind, len, mtime, ctime, ino) per entry, and the difference between two walks as the events. Owning it also lets the cadence track the measured scan cost rather than a constant, and lets a symlink be recorded as a symlink instead of being descended.
13. Process sandbox: env_clear() on child spawn: When spawning a child process (e.g., capsem-process from service), always call env_clear() then re-add only the minimal env vars needed (HOME, PATH, USER, TMPDIR, RUST_LOG). The service's shell environment may contain API keys, tokens, or secrets that the child process has no business seeing. The guest's --env args are a separate injection path and are already validated.
14. UDS socket permissions must be 0600: After UnixListener::bind(), immediately set_permissions(..., 0o600). The default umask leaves sockets world-accessible, meaning any local user can connect to a VM's IPC or terminal WebSocket with no auth. The gateway token file already does this; per-VM sockets must match.
15. Never process::exit() on guest-controlled I/O: A guest can close a vsock fd at any time. If the host handler calls process::exit(1) on read error, the guest has an unconditional DoS. Use break to exit the read loop and let the process shut down through normal channels.
16. Sensitive logs cross a worker boundary: Terminal and serial files contain raw user data. Use unix::fs single-link, no-follow file helpers and bound reads; a confined writer can still plant a redirect at its own log path. Below worker-writable directories, anchor access through ContainedDir descriptors. Prepare session::prepare_session_logs under the exclusive owner lock before confinement: it retains owner/logs/ and migrates legacy generations without overwriting conflicts. Hypervisor console and terminal output share one SharedLogWriter and rotation counter; never open an independent appender. Writers retain directory descriptors across rotation, which needs directory authority rather than exact-file grants. Keep session root and ledger outside that grant. Create files with mode 0600 and repair permissions before confinement or through the authorized broker; confined writers must not call set_permissions. Session readers use the shared console/transcript APIs, falling back only for a missing layout. Support exports redact recognized credential bytes even in mixed binary/text lines, preserving unrelated bytes without a UTF-8 bypass. The PTY, telemetry, support-export and actual owner-sandbox regressions prove retention and malicious redirects.
17. VirtioFS share boundary -- only guest/ subtree: The VirtioFS share must point at session_dir/guest/, not session_dir itself. Host-only state (session.db, owner/logs/, owner/checkpoint.vzsave, system/rootfs.img) stays outside the share. The system disk lives in the host-only system/ directory; never restore a compatibility link into guest storage. Put guest-visible content under guest/. Use capsem_core::guest_share_dir(session_dir) to get the share root.
18. VM route id is not the display name: SandboxInfo.id, route path ids, InstanceInfo.id, DB handle keys, session directory basenames, and CAPSEM_VM_ID must be the opaque VM/session id. Persistent registry name values such as co-work1 or code-vm1 are display aliases only. User surfaces may accept names, but they must translate to id before calling /vms/{id}/.... Do not use registry.get(&id) in route code unless the route id has first been resolved to the registry key; use the persistent identity helpers instead. When returning UI/API payloads, preserve both id and name. Regression coverage lives in persistent_session_routes_keep_uuid_id_separate_from_display_name, resume_sandbox_requires_uuid_route_id_not_display_name, and persistent_route_identity_source_guard.
19. Use capsem_core::poll::poll_until for all async polling: All "wait until ready" patterns must use the shared poll_until utility in capsem-core/src/poll.rs. It provides deadline-based timeout, exponential backoff, and structured tracing (attempt count, elapsed time, label). Never write ad-hoc for _ in 0..N { sleep(X) } or while now < deadline { sleep(fixed) } loops -- they lack logging, use fixed intervals instead of backoff, and hardcode timeouts. For sync code (guest agent), vsock_connect_retry in vsock_io.rs has the same pattern with eprintln logging. Every retry loop must have a total deadline.
20. DRY wait patterns -- one wait, one place: When a server endpoint already waits for a subprocess to become ready (e.g., wait_for_vm_ready in handle_exec), clients must not add their own retry loop on top. The test helper wait_exec_ready previously polled 30 times with 1s sleep, and each poll triggered a 30s server-side wait -- a 30x30s pathological cascade. After fixing the readiness gate (lesson 9), the client calls exec once with adequate HTTP timeout and the server handles the wait. Apply this DRY principle to any client/server readiness pattern: decide which layer owns the wait and make others pass through.
21. Companions must not outlive their parent -- kill_on_drop is not enough: tokio::process::Command::kill_on_drop(true) only fires when the parent's Child handle is dropped on graceful shutdown. Under SIGKILL/OOM/test-harness timeout/ pytest-xdist worker death, Drop never runs and companion processes (gateway, tray) get re-parented to PID 1 and survive forever. Every just test -n 4 run leaked a fresh batch of orphans; accumulated orphans caused VM-ready poll spins, UDS-port collisions, and the suspend/resume regression. Defense in depth is mandatory for any spawned companion process, enforced on the COMPANION side so the parent can't get it wrong:
    • Pass --parent-pid <spawner_pid> when spawning.
    • Companion calls capsem_guard::install(parent_pid, lock_path)? at startup:
      • Refuses to run if parent PID is missing, dead, or not our actual getppid() (parent_is_expected). Exit 0 — standalone launches become silent no-ops.
      • Acquires an flock(2) singleton at lock_path (O_CLOEXEC opened atomically; process-local registry covers the brief fork-to-exec window where the flock fd can be inherited). Second instance exits 0.
      • Spawns a 100ms-interval watcher thread that calls std::process::exit(0) the moment getppid() no longer equals the declared parent PID. getppid() is immune to zombie state and flips to 1 on re-parenting, which is the reliable signal across SIGKILL, SIGSEGV, and OOM.
      • The mandatory terminator runs before any best-effort tracing. Parent death commonly closes a companion's stdout/stderr pipes; logging to that dead sink can panic the watcher thread, so observability must never stand between detecting reparenting and exiting.
    • Lock paths: tray is SYSTEM-WIDE ($HOME/.capsem/run/tray.lock) because the macOS menu bar is a shared global resource; gateway is per-run_dir because each test's gateway bridges a distinct UDS. Regression tests in tests/capsem-service/test_companion_lifecycle.py cover: refuse-standalone (no parent / wrong parent), singleton (double spawn, 20-way hammer), and die-with-parent (SIGKILL the parent, companion exits within 5s). When adding any new companion process, wire it through capsem-guard — don't invent a new pattern.
22. Retry loops must classify errors, not time-bound a blanket wait: When waiting for a resource to come up, the retry closure must distinguish retryable errors from permanent ones, and the classification depends on the caller's context, not the error itself. Identical NotFound / ConnectionRefused errors mean "service is down, give up" on an initial probe but "socket not bound yet, keep waiting" one call later in the post-launch retry. Pattern:
    • Use capsem_core::poll::poll_until, not a hand-rolled backoff loop. The poll primitive already gives you deadline, exponential backoff, per-attempt logging (label + elapsed + attempts), and a typed TimedOut error. Every new retry site that reinvents these is a future bug -- the capsem doctor "Service manager started capsem but socket not ready" bug existed only because UdsClient::connect_with_timeout hand-rolled its own loop and fast-failed on ENOENT before the just-started service had bound its socket.
    • Inside the poll_until closure: return None on retryable errors (poll_until keeps polling), return Some(Err(...)) on permanent errors (poll_until exits immediately).
    • Thread a small enum, not a patient: bool, so every call site documents intent: ConnectMode::FailFast vs ConnectMode::AwaitStartup, ProbeMode::Expected vs ProbeMode::MustBeRunning. crates/capsem/src/client.rs::UdsClient::connect_with_timeout is the canonical example in the tree.
    • Don't .map_err(|_| anyhow!(...)) on the timeout branch. You erase the inner cause. Chain with Context so the root error lives in the error chain and {err:#} prints both the summary and the underlying io::Error kind.

Async reference

Read references/rust-async-patterns.md for comprehensive tokio patterns (tasks, channels, streams, error handling). From the community (6.4K installs).