AGENTS.md — VoidBox

VoidBox is a lightweight micro-VM runtime for sandboxed execution. It boots a guest agent from initramfs, optionally performs an OCI root switch, and exposes a host↔guest API over vsock. This file covers architecture, testing, validation, and debugging guidance for agents working on the project.

Design proposals: RFC + ADR process

Changes that are large, complex, cross-cutting, or that need peer review before commitment go through an RFC (proposal + discussion) in docs/rfc/; the resulting load-bearing decisions are recorded as ADRs (atomic, immutable, superseded-not-edited) in docs/adr/. Small, local, obvious changes skip the RFC and go straight to a PR. The full process — when to write an RFC, numbering, statuses, and templates — is documented once in docs/rfc/README.md.

Code intelligence

Always prefer LSP operations (goToDefinition, findReferences, hover, documentSymbol, workspaceSymbol, etc.) over Grep/Glob for code navigation. LSP provides compiler-aware results that understand types, scopes, and cross-file relationships. Fall back to Grep/Glob only for pattern-based searches LSP doesn't cover (comments, config files, non-Rust files).

Commit and PR messages

Write commit message bodies and pull-request descriptions as Markdown, with each paragraph on a single line — do not hard-wrap prose to a fixed column. The renderer (GitHub, git log viewers) wraps it for display; manual line breaks mid-paragraph render as ragged text and produce noisy diffs on edits. Keep the commit subject line short (~72 chars), and use blank lines to separate paragraphs and normal Markdown for lists. Reserve hard line breaks for content that is genuinely line-oriented (lists, tables, code blocks).

Do not add AI-agent attribution anywhere — no "Generated by"/"Co-Authored-By" tool credits, no session links, no "written with <agent>" notes — in commit messages, PR titles or descriptions, code, comments, or docs. The work stands on its own; tooling used to produce it is not part of the record.

Notable changes — user-facing fixes, features, API changes, security bumps — get an entry under [Unreleased] in CHANGELOG.md in the same PR (see CONTRIBUTING.md and the PR-template checklist; match the existing entry style: bold lead, symptom, root cause, fix, issue references).

PR descriptions must follow .github/PULL_REQUEST_TEMPLATE.md, including its Local validation block. GitHub applies the template only in the web UI — a PR created with gh pr create must reproduce it in the body explicitly.

Markdown documentation style

Conventions for Markdown prose in design docs — RFCs, ADRs, files under docs/, and READMEs — live in docs/agents/doc-style.md: one paragraph per line, introduce before use, mechanism before property, don't narrate, and prefer standard terms over coined jargon.

@docs/agents/doc-style.md

Rust coding conventions

Imports and constants

Always declare use imports and const / static items at module scope (top of the file, after the module-level doc comment), never inline inside function bodies. Inline declarations hide dependencies and make the file harder to scan.

// ✓ correct
use std::io::{Read, Seek, SeekFrom};
const OVERHEAD_BYTES: u64 = 208 * 1024 * 1024;

fn my_fn() { /* uses them */ }

// ✗ avoid
fn my_fn() {
    use std::io::{Read, Seek, SeekFrom};
    const OVERHEAD_BYTES: u64 = 208 * 1024 * 1024;
    // ...
}

Data model and policy design

Prefer explicit single-source models over overlapping flags that create ambiguous or invalid states. If a feature has a small closed set of modes, model it as an enum rather than separate booleans/options whose combinations must be interpreted later.

Keep execution policy separate from persisted or user-resolved config. Command-scoped behavior (for example, interactive terminal ownership) should be derived at the CLI/runtime boundary and passed inward as a narrow policy type, rather than mutating resolved config structs or threading broad command enums through lower layers.

Prefer descriptive local names and named constants over shorthand locals and inline literals. Variables like rc, n, msg, ws, k, and v are only acceptable in tiny, obvious scopes; otherwise use names that describe meaning directly. Repeated or behavior-defining literals should be lifted to documented module-scope constants.

VM pre-flight validation

Operations that can fail silently inside the guest (e.g. the kernel dropping initramfs files when memory is too low) must surface a clear diagnostic on the host, before the VM starts. The pattern:

1. Check before launch — validate the config in VmmBackend::start() before any irreversible side effect, using helpers on BackendConfig (e.g. initramfs_memory_warning()).
2. Log at WARN — use warn!() so the message appears without aborting the run. Return Err only when a successful boot is impossible regardless of retry.
3. Name the downstream symptom — mention the observable failure mode (e.g. "kernel may silently drop initramfs files (e.g. vsock.ko)") so an operator can connect the warning to a later guest error without a separate debugging session.

Doc-comment style

Module-level and function-level doc comments should explain why the code is structured the way it is, not narrate its evolution. Avoid phrases like "previously did X, now does Y," "this module replaces…," or "tests for X were removed in [ticket]" — they read fine in a PR description, but in code they rot when later changes land and they push the structural rationale onto the reader to reconstruct from history.

State the rationale in present tense, in terms of what the alternative would get wrong:

// ✓ structural — explains why the chosen approach is correct
//! Lexical path checks cannot defend against planted symlinks: the
//! string the check inspects and the inode the kernel writes to are
//! not the same object. This module gates every path through
//! `openat2` with `RESOLVE_NO_SYMLINKS` …

// ✗ historical — narrates evolution; ages poorly
//! `handle_write_file` previously gated paths through a lexical check.
//! This module replaces that with kernel-side resolution …

Keep concrete attack examples or worked use-cases when they're pedagogically useful — those are illustrations, not history.

Likewise, document the design, not the deliberation. State what the code is and the structural reason it holds; don't reproduce the internal reasoning that reached it — the alternatives weighed, why one won, or emphatic framing ("the whole point of X", a "X, not Y" heading, a rhetorical "this raises the obvious question"). That belongs in the PR or the RFC's "Alternatives considered" / "Risks & trade-offs" sections, not in code the reader is trying to understand.

Avoid ticket IDs (R-B*, R-arch-*) and PR/commit references inside doc comments and inline comments. They belong in commit messages and PR descriptions where they're audit trail; in code they age into noise as the ticketing context evolves. Likewise, never reference a private repository by name in public source (e.g. don't write // see void-security threat model …). If a reasoning trail lives in a private repo, the most a public file should expose is an opaque ID at most — and ideally not even that; prefer to inline the structural reasoning directly.

Platform parity

Contributions must work on both Linux (KVM) and macOS (VZ). Validate on both where applicable. Key platform differences:

Architecture overview

• Host runtime launches a VM backend (KvmBackend on Linux, VzBackend on macOS).
• Guest boots guest-agent from initramfs.
• If sandbox.image is set, guest performs OCI root switch with pivot_root.
• On Linux/KVM, OCI base rootfs is attached as cached virtio-blk disk.
• OCI skills are mounted as read-only tool roots.
• If agent.mode is service, the agent runs indefinitely; output is published while the process continues.
• If agent.messaging.enabled is true, a host-side sidecar coordinates inter-agent communication over HTTP; void-mcp bridges Claude Code to the sidecar via MCP tools.

Control channel I/O model

VoidBox uses one persistent, multiplexed vsock connection per sandbox for every host↔guest RPC (exec, write_file, mkdir_p, telemetry, pty, snapshot). The connection is established lazily on the first RPC, reused for everything after that, and reconstructed if the reader thread dies. Earlier versions used a per-RPC connect-handshake-close cycle; that is gone.

Multiplex design

After the Ping / Pong handshake negotiates PROTO_FLAG_SUPPORTS_MULTIPLEX on both sides, every subsequent frame carries [request_id: 4 B LE][body...] as its payload. The Message frame layout is unchanged — the request_id is an in-payload prefix. Helpers in src/backend/multiplex.rs (build_frame, decode_payload) centralize the layout so callers never hand-roll offsets.

Dispatch per RPC:

1. Caller allocates a fresh request_id via an AtomicU32 counter.
2. Caller registers a dispatch slot (oneshot or stream) keyed on that id.
3. Caller writes the framed request through a shared FrameSender behind a Mutex (serializes writes across concurrent RPCs). The write runs on a spawn_blocking task with a per-wait send timeout (SO_SNDTIMEO) on the stream and a hard per-frame deadline covering mutex queueing and trickle-fed writes. A send that fails before any byte reaches the stream fails only its own RPC; a send that truncates a frame mid-write (or exceeds the deadline) marks the channel dead — the wire can no longer be trusted for framing — and the next RPC reconnects.
4. Caller awaits the matching response on its slot channel.

One dedicated reader thread per channel owns the read half of the stream, demultiplexes each incoming frame by its request_id, and delivers the message to the registered slot. The read half carries a bounded SO_RCVTIMEO: between read attempts the reader checks a shared shutdown flag (set by explicit shutdown, by a send that truncated a frame or exceeded the per-frame deadline, or when the last channel handle drops), so the reader can exit without waiting for EOF; the timeout never surfaces to the framing layer, so an idle-but-healthy channel is unaffected. A peer that stays silent while its socket stays open is indistinguishable from an idle one at this layer — bounding that is the per-RPC timeouts' job. When the reader sees end-of-stream, a fatal protocol error, or the shutdown flag, it marks the channel dead, fails every still-pending slot, and exits. The next RPC reconstructs the channel.

VM teardown never depends on another thread consuming a message. MicroVm::stop() and snapshot() clear running and kick the vCPUs first, then call ControlChannel::shutdown() on the VM-internal channel — killing the live multiplex channel (failing every pending dispatch slot, which unblocks an event loop wedged inside an RPC that carries no timeout of its own), aborting any in-flight connect/handshake retry loop, and refusing re-establishment — and only then join the VM threads, each join bounded by a deadline that fails with an error naming the stuck thread. The backend-level channel that carries every run's RPCs (exec, telemetry, write_file) gets the same ControlChannel::shutdown() in KvmBackend::stop() / VzBackend::stop(), so service-mode execs and telemetry streams fail promptly instead of waiting to observe EOF. The userspace vsock worker is stopped explicitly at the end of teardown, even when a join deadline fired (non-blocking in that case, since the wedged thread may hold the device mutex); outside teardown its lifetime is the device's lifetime, so it services the host-facing listener before driver initialization and across guest resets, with host connects and buffered guest-bound data both capped so a driver that never comes up cannot grow host memory or the fd table without bound.

Synchronous I/O wrapped in spawn_blocking

All guest communication uses synchronous I/O (std::io::Read/Write) wrapped in tokio::task::spawn_blocking so that blocking syscalls never stall the tokio async runtime.

Why not fully async (AsyncRead/AsyncWrite)?

1. The VZ connector is fundamentally callback-based. VZVirtioSocketDevice.connectToPort:completionHandler: dispatches onto a GCD serial queue and fires an ObjC completion handler. There is no fd to poll until after the callback delivers one. A bridge channel between GCD and tokio is required regardless.
2. AsyncFd on raw vsock fds is fragile. The fd comes from dup(connection.fileDescriptor()) inside a GCD callback, outside tokio's control. AsyncFd requires O_NONBLOCK, which changes read/write semantics and demands WouldBlock handling. Blocking fds with SO_RCVTIMEO are simpler and correct.
3. Cancellation safety. With async reads, tokio::time::timeout cancels the future mid-read. If cancellation happens between a header read and a payload read, the stream is left in an inconsistent state. Avoiding this requires a cancel-safe framing layer — added complexity for no practical gain.
4. Concurrency is low per channel. A single VM has one control channel. Even with concurrent exec calls, the blocking thread pool (up to 512 threads, dynamic scaling) handles the workload comfortably. The async advantage doesn't apply.

RPC pattern

Every public async fn on ControlChannel follows the same shape:

1. Serialize the outgoing message on the caller's async task (cheap, non-blocking).
2. Establish the multiplex channel lazily. The connect + handshake runs in spawn_blocking, and the whole establishment await — including the channel mutex and the blocking-task join — is bounded by the connect deadline plus a fixed margin, because it runs before any per-RPC timeout wrapper is armed.
3. Allocate request_id → register dispatch slot → spawn_blocking the framed-request write (bounded by the per-frame send deadline) → await the slot on the async task. One-shot RPCs await under a per-RPC timeout; exec awaits under the exec timeout unless service mode requests none; telemetry streams await for the channel's lifetime by design.

Handshake (connect_with_handshake_sync) is a regular fn using std::thread::sleep for backoff. It runs once per channel lifetime and is never called from an async context directly.

Key files

Guest-side logging must go through /dev/kmsg, not eprintln! — see docs/war-histories.md for why.

OCI root switch internals

OCI root switch sequence (setup_oci_rootfs)

Linux/KVM (virtio-blk):

1. Host builds a cached ext4 disk image from the extracted OCI rootfs (build_oci_rootfs_disk in src/runtime.rs, #[cfg(target_os = "linux")]).
2. Disk is attached as a read-only virtio-blk device (/dev/vda).
3. Guest-agent mounts /dev/vda as ext4 with MS_RDONLY at /mnt/oci-lower.

macOS/VZ (virtiofs):

1. Host shares the extracted OCI rootfs directory into the guest via virtiofs (read_only: true), resolved in apply_oci_rootfs (src/runtime.rs).
2. Guest-agent finds the rootfs at /mnt/oci-rootfs (already mounted by mount_shared_dirs).
3. The virtiofs mount is used directly as the overlay lowerdir.

Common (both platforms):

4. A tmpfs is mounted for overlay upper + work directories.
5. Overlayfs is mounted at /mnt/newroot (lower=OCI rootfs RO, upper=tmpfs RW).
6. Essential mounts (/proc, /sys, /dev) are move-mounted into the new root.
7. Mount propagation is set to MS_REC | MS_PRIVATE on /.
8. pivot_root(".", "mnt/oldroot") switches the root.
9. Old root is detached with umount2(MNT_DETACH) and removed.
10. /tmp, /workspace, /home/sandbox are recreated; DNS config is restored.

If pivot_root returns EINVAL (initramfs can't be pivoted), a switch-root fallback uses MS_MOVE + chroot(".") and records OCI_OK_SWITCH_ROOT. Status is tracked via OCI_SETUP_STATUS (AtomicU8), with distinct codes for each failure point (e.g. OCI_FAIL_BLOCK_MOUNT, OCI_FAIL_OVERLAY_MOUNT, OCI_FAIL_PIVOT_ROOT_*).

Read-only strategy (defense-in-depth)

On Linux the three-layer block-device strategy applies. On macOS the virtiofs share is host-enforced RO and the guest adds MS_RDONLY. Both platforms use the overlayfs upper layer (tmpfs) to absorb writes, so the guest has a writable root without modifying the base rootfs.

Host directory mounts

VoidBox supports mounting host directories into the guest VM with explicit read-only or read-write access. RW mounts write directly to the host directory, so changes persist across VM restarts.

Data flow:

YAML spec (MountSpec)
  → runtime (MountConfig)
    → kernel cmdline: voidbox.mount0=mount0:/data:rw
      → guest-agent: mount 9p/virtiofs tag at /data

Spec-level: MountSpec in src/spec.rs defines the user-facing YAML fields: host (host directory path), guest (guest mount point), mode ("ro" default, "rw").

Backend-level: MountConfig in src/backend/mod.rs carries host_path, guest_path, and read_only (boolean). VmConfig.mounts (src/vmm/config.rs) holds the list of mount configs for the VM.

Linux/KVM transport: Each mount becomes a virtio-9p device (src/backend/kvm.rs). The kernel cmdline receives voidbox.mount<N>=<tag>:<guest_path>:<ro|rw> parameters (src/vmm/config.rs:244-248).

macOS/VZ transport: Each mount becomes a virtiofs share (src/backend/vz/backend.rs). The same kernel cmdline convention is used (src/backend/vz/config.rs).

Guest-agent: Parses voidbox.mount* params from /proc/cmdline and mounts each tag at the specified guest path with the declared mode.

Structured logging & observability

VoidBox has a structured logging abstraction that bridges application-level log calls to the tracing ecosystem. All workflow progress messages should go through this pipeline rather than raw eprintln! or bare tracing::info!.

Pipeline:

observer.logger().info(msg, attrs)
  → StructuredLogger::log()
    → tracing::info!()  (when output_to_tracing is true, which is the default)
      → tracing_subscriber (EnvFilter) → stderr

How to use it: In workflow/scheduler code, obtain the logger from the Observer and call .info(), .warn(), .error(), etc. with structured key-value attributes:

self.observer.logger().info(
    &format!("[workflow:{}] step {}/{}: \"{}\" running...", name, i, total, step),
    &[("step", step_name)],
);

Do not use eprintln! for progress messages — it bypasses the structured pipeline and won't carry trace context or attributes.

Log levels: LogConfig::default() sets the minimum level to INFO.

CLI subscriber: src/bin/voidbox/main.rs initializes tracing_subscriber with EnvFilter from the resolved log level (see src/bin/voidbox/cli_config.rs for merge order). Override at runtime with VOIDBOX_LOG_LEVEL or --log-level, or set RUST_LOG when no explicit CLI/config level is set:

VOIDBOX_LOG_LEVEL=debug cargo run --bin voidbox -- run --file spec.yaml
# RUST_LOG also applies when VOIDBOX_LOG_LEVEL / config omit log_level

Configuration and routing principles:

• Keep configuration opt-in by default unless a different default is needed to prevent a clearly broken UX or unsafe behavior.
• When a runtime limitation or fallback occurs, emit an explicit warning with actionable remediation rather than silently degrading behavior.
• Prefer making routing and storage locations user-customizable (for example, via config or CLI overrides) rather than hard-coding one path.
• Interactive PTY/TUI sessions should own the terminal. Do not mix runtime diagnostics, guest console output, or other non-PTY streams into the same terminal once the interactive session begins; route them to tracing/file sinks instead.

Convention: Workflow progress messages use the [workflow:<name>] prefix pattern, e.g. [workflow:my-flow] step 1/3: "build" running....

Key files:

Key source files

• guest-agent/src/main.rs — setup_oci_rootfs (~line 754), mount_oci_block_lowerdir (~line 1133)
• src/devices/virtio_blk.rs — RO feature flag (line 21), write rejection (line 394)
• src/runtime.rs — build_oci_rootfs_disk (~line 815, host-side ext4 image creation), resolve_oci_rootfs_plan (~line 776), apply_oci_rootfs (~line 745)
• src/backend/vz/backend.rs — VZ virtiofs mount setup

Snapshot / restore

VoidBox supports two snapshot types: Base (full dump) and Diff (dirty pages only). All snapshot features are explicit opt-in — no snapshot code runs unless the user sets a snapshot field.

Design principle

Every snapshot-related field defaults to None. No auto-detection, no env var fallback, no implicit behavior. The system behaves identically to before if no snapshot field is set.

Opt-in layers

Snapshots can be enabled at any layer of the stack:

Snapshot resolution

resolve_snapshot() in src/runtime.rs resolves a snapshot string:

1. Try as hash prefix → ~/.void-box/snapshots/<prefix>/ (if state.bin exists)
2. Try as literal directory path (if state.bin exists)
3. Neither found → print warning, return None (cold boot)

Per-box overrides (resolve_box_snapshot()) take priority over top-level spec.

Wire protocol

SnapshotReady = 17 in void-box-protocol/src/lib.rs. Guest-agent handles message type 17 as a no-op ack (sends SnapshotReady back). Host-side ControlChannel::wait_for_snapshot_ready() in src/backend/control_channel.rs connects and round-trips the message.

Key source files (snapshot)

Security considerations

• RNG entropy: Cloned VMs share /dev/urandom state. Mitigated by fresh CID
  • session secret per restore and hardware RDRAND.
• ASLR: Clones share page table layout. Mitigated by short-lived tasks, SLIRP NAT isolation, and command allowlists.
• Session secret: Each restored VM gets a unique secret via kernel cmdline; the snapshot's stored secret is for state continuity, not auth reuse.

Service mode

VoidBox agents run in one of two modes: task (default) or service. Task mode runs to completion and returns output. Service mode runs indefinitely — publishing structured output while the process continues — and is designed for long-running daemons like API gateways.

YAML configuration

Agent-level:

agent:
  prompt: "Run the gateway"
  mode: service
  output_file: /workspace/output.json

Workflow step-level:

steps:
  - name: gateway
    mode: service
    run:
      program: sh
      args: [-lc, "exec node /app/server.mjs"]

output_step: gateway

Validation rules

Service workflow steps set effective timeout to 0 (infinite).

Lifecycle differences

The daemon (spawn_service_run()) manages three oneshot channels:

• output_rx — fires when the output file is ready
• exit_rx — fires when the service process terminates
• stop_tx — signal to gracefully stop the service

ServiceExit maps to RunStatus: Exited { success: true } → Succeeded, Exited { success: false } → Failed, Canceled → Cancelled, Crashed(msg) → Failed.

Key files

Messaging and sidecar

VoidBox includes a sidecar — a host-side HTTP server that coordinates inter-agent communication. Agents exchange intents (typed messages with audience and priority) through the sidecar, enabling multi-agent collaboration within isolated VM sandboxes.

Enabling messaging

agent:
  prompt: "..."
  messaging:
    enabled: true
    provider_bridge: claude_channels  # optional

When messaging.enabled is true, the daemon starts a sidecar server and injects VOID_SIDECAR_URL=http://10.0.2.2:<port> into the guest environment.

Architecture

Guest VM                              Host
┌──────────────────────┐    HTTP    ┌─────────────┐
│  void-message CLI    │──────────→│  Sidecar     │
│  void-mcp server     │           │  /v1/inbox   │
│  (agent process)     │           │  /v1/intents │
└──────────────────────┘           │  /v1/context │
                                   └─────────────┘

The sidecar runs on the host network. Guest tools reach it via the SLIRP gateway address (10.0.2.2).

Intent model

Agents communicate through intents — structured messages with:

• kind: proposal, signal, or evaluation
• audience: broadcast (all agents) or leader (coordinator only)
• priority: high, normal, or low
• payload: JSON, max 4096 bytes

Constraints: max 3 intents per iteration, default TTL of 2 iterations. Idempotency keys prevent duplicate submissions.

Sidecar API endpoints

void-message CLI

A guest-side CLI for direct sidecar interaction:

void-message context              # read execution identity
void-message inbox [--since N]    # read peer messages
void-message send --kind signal --audience broadcast --summary "ready"
void-message health               # check sidecar

Key files

MCP integration

The void-mcp crate implements an MCP (Model Context Protocol) server that bridges Claude Code to the sidecar messaging system. It exposes four collaboration tools and handles the MCP JSON-RPC wire protocol.

Tools

Transport modes

• Stdio (default): Content-Length framed JSON-RPC 2.0 (same as LSP)
• Streamable HTTP (--sse): Listens on 127.0.0.1:<port>, handles POST /mcp — used inside minimal VMs where Claude Code can't spawn child processes

Provisioning flow

When a spec includes an MCP skill, VoidBox::provision_skills():

1. Starts void-mcp as a background HTTP server inside the guest (SSE mode, port 8222 + index)
2. Writes /workspace/.mcp.json:

   {
     "mcpServers": {
       "void-mcp": {
         "type": "http",
         "url": "http://127.0.0.1:8222/mcp"
       }
     }
   }

3. Passes --mcp-config /workspace/.mcp.json to Claude Code

The void-mcp binary is in the guest execution allowlist (DEFAULT_COMMAND_ALLOWLIST in src/backend/mod.rs).

Data flow

Claude Code → JSON-RPC → void-mcp (:8222) → HTTP → Sidecar (host :N) → Swarm state

Skill configuration

Skills can be declared in YAML specs or via the builder API:

agent:
  skills:
    - mcp:void-mcp

Or programmatically:

Skill::mcp("void-mcp")
    .args(vec!["--sse".into()])
    .env("VOID_SIDECAR_URL", url)

Key files

Auto image resolution

voidbox run --file spec.yaml and voidbox shell auto-resolve kernel and initramfs without requiring VOID_BOX_KERNEL or VOID_BOX_INITRAMFS env vars. Images are downloaded from GitHub Releases, verified with SHA-256 checksums, and cached under ~/.void-box/images/v{version}/.

Resolution chain

Both kernel and initramfs follow a fallback chain:

1. --kernel / --initramfs flag or VOID_BOX_KERNEL / VOID_BOX_INITRAMFS env var
2. Linux only (kernel): /boot/vmlinuz-$(uname -r) — use host kernel
3. Well-known install paths: /usr/lib/voidbox/ (Linux), /opt/homebrew/lib/voidbox/ (macOS)
4. Cache hit: ~/.void-box/images/v{version}/<artifact>
5. Download from GitHub Release → verify SHA-256 → cache
6. OCI fallback (ghcr.io/the-void-ia/voidbox-guest)

Flavor selection

The initramfs flavor is derived from spec.llm.provider:

The mapping is centralized in image::flavor_for_provider().

voidbox image subcommand

voidbox image pull <flavor>     # Download: base, claude, codex, agents, kernel, or all
voidbox image list              # Show cached images (version, flavor, arch, size, path)
voidbox image clean             # Remove old versions (keep current)
voidbox image clean --all       # Remove everything

Key files

Cache layout

~/.void-box/images/
  v0.1.2/
    void-box-claude-x86_64.cpio.gz
    void-box-claude-x86_64.cpio.gz.sha256
    vmlinuz-x86_64
    vmlinuz-x86_64.sha256

Testing

Static quality:

cargo fmt --all -- --check

Clippy (guest-agent builds as a no-op stub on non-Linux, so the same commands run on macOS):

cargo clippy --workspace --all-targets --all-features -- -D warnings

Core tests:

cargo test --workspace --all-features
cargo test --doc --workspace --all-features

Ignored/VM suites. The deterministic suites auto-provision the pinned kernel and test initramfs into target/ on first run, so VOID_BOX_KERNEL / VOID_BOX_INITRAMFS need not be set — an explicit override still wins (ADR-0011). The heavy agent suites e2e_agent_mcp and e2e_service_mode are the exception: they need a staged production image via those env vars and skip when it is absent. On a machine that genuinely cannot virtualize the suites skip loudly, but the reason prints only with --nocapture; on a machine you believe is capable, set VOID_BOX_REQUIRE_VM=1 so a skip becomes a failure and nothing hides. A real boot or RPC failure always fails.

The two agent suites skip a second way: they also need a real Anthropic credential, and without one they print a skip and pass. Set VOID_BOX_REQUIRE_AGENT_CREDS=1 when you have staged a key, and that skip becomes a failure too. The variable is separate from VOID_BOX_REQUIRE_VM=1 because the Linux CI lane asserts VM capability while holding no credential — it must keep skipping the agent suites without laundering a lost key into a pass.

To run the VM suites with bounded parallelism instead of one at a time, use nextest — each VM-booting binary joins the vm test group in .config/nextest.toml, capped at two concurrent boots. This runs every VM suite locally on a capable machine:

# VOIDBOX_TEST_GUEST_IMAGE makes guest_image_pull_and_extract run instead of
# skipping — the CI lane sets it, so without it a local run cannot reproduce
# the gate. The two agent suites are excluded because they need a staged
# production image and a credential: `e2e_.+` would sweep them in, and under
# VOID_BOX_REQUIRE_VM=1 their artifact check fails rather than skips.
VOID_BOX_REQUIRE_VM=1 \
VOIDBOX_TEST_GUEST_IMAGE=ghcr.io/the-void-ia/voidbox-guest:v0.2.0 \
cargo nextest run --run-ignored only --no-fail-fast \
  -E 'binary(/^(conformance|oci_integration|snapshot_integration|persistent_channel|telemetry|kvm_integration|e2e_.+)$/) & not binary(/^(e2e_agent_mcp|e2e_service_mode)$/)'

The Linux CI lane runs a fixed subset of these under VOID_BOX_REQUIRE_VM=1; .github/workflows/e2e.yml and .config/nextest.toml are the source of truth for exactly what CI runs. It covers every deterministic suite that runs on Linux — conformance, telemetry, kvm_integration, oci_integration, e2e_sidecar, e2e_telemetry, e2e_mount, e2e_skill_pipeline, e2e_pty, e2e_credential_proxy, persistent_channel, snapshot_integration. snapshot_vz_integration is macOS-only and has no CI lane at all (#158). e2e_service_mode and e2e_agent_mcp stay out of it: no pull-request runner holds an Anthropic credential, so there they would pass without starting an agent. Run them on your own machine and record the result in the pull request's Local validation block. .github/workflows/e2e-agent.yml runs the same pair under VOID_BOX_REQUIRE_AGENT_CREDS=1, but by manual dispatch only — every run spends Anthropic API credit, so nothing triggers it automatically.

skill_pipeline's two --ignored tests are off the gate for a different reason: they fetch a skill document from a third-party GitHub repository, so they would fail when that repository changes rather than when this one does. .github/workflows/e2e-external.yml runs them weekly and on demand. A red result there means the outside world moved.

The explicit-artifact form (env override; still one suite at a time):

export VOID_BOX_KERNEL=/path/to/vmlinuz

# Generic guest image suites:
export VOID_BOX_INITRAMFS=/tmp/void-box-rootfs.cpio.gz
cargo test --test conformance -- --ignored --test-threads=1
cargo test --test oci_integration -- --ignored --test-threads=1

# Snapshot integration suites (required for any snapshot/restore/vCPU changes):
export VOID_BOX_INITRAMFS=/tmp/void-box-test-rootfs.cpio.gz
cargo test --test snapshot_integration -- --ignored --nocapture --test-threads=1

# Claudio-based deterministic E2E suites:
scripts/build_test_image.sh
export VOID_BOX_INITRAMFS=/tmp/void-box-test-rootfs.cpio.gz
cargo test --test e2e_telemetry -- --ignored --test-threads=1
cargo test --test e2e_skill_pipeline -- --ignored --test-threads=1
cargo test --test e2e_mount -- --ignored --test-threads=1
cargo test --test e2e_sidecar -- --ignored --test-threads=1

# Agent suites — production image plus a real credential:
scripts/build_claude_rootfs.sh
export VOID_BOX_INITRAMFS=$PWD/target/void-box-claude.cpio.gz
export VOID_BOX_REQUIRE_AGENT_CREDS=1
ANTHROPIC_API_KEY=... cargo test --test e2e_service_mode -- --ignored --test-threads=1
ANTHROPIC_API_KEY=... cargo test --test e2e_agent_mcp -- --ignored --test-threads=1

Fuzzing the guest-facing parsers

Several host-side parsers read bytes a guest chooses: the control-channel frame decoder (void-box-protocol framing plus the multiplex request-id prefix), the userspace vsock connection state machine, the split-virtqueue reader that walks descriptor chains out of guest memory, the 9P server together with the transport beneath it, the virtio-net device's TX and RX descriptor walks, and the virtio-blk request handler. Their harnesses live in src/fuzz.rs, and two callers drive the same bodies — cargo fuzz and the replay gate.

The merge gate replays them. tests/fuzz_corpus.rs runs every file under fuzz/corpus/<target>/ and fuzz/artifacts/<target>/ through its harness on stable, as part of a plain cargo test. It is deterministic and costs milliseconds, so it needs no special invocation.

Discovery is out of band. cargo fuzz needs a nightly toolchain, and a search that finds a new bug on an unrelated change would block that change, so it runs weekly and on demand in .github/workflows/fuzz.yml — never on a pull request (ADR-0012). To run it locally:

rustup toolchain install nightly
cargo install cargo-fuzz
# libfuzzer-sys builds the libFuzzer runtime with cc, so a C++ compiler must be
# on PATH as `c++` (Fedora: gcc-c++; Debian/Ubuntu: g++).
# libFuzzer writes what it discovers to the first corpus directory and reads the
# rest without modifying them, so runs go to `fuzz/corpus-run/` (gitignored) and
# `fuzz/corpus/` stays the curated seed set.
mkdir -p fuzz/corpus-run/vsock_frame
cargo +nightly fuzz run vsock_frame fuzz/corpus-run/vsock_frame fuzz/corpus/vsock_frame \
  -- -max_total_time=60 -rss_limit_mb=2048
# targets: vsock_frame, vsock_packet, virtqueue, nine_p, nine_p_transport,
#          virtio_net, virtio_blk

Do not commit what a run accumulates in fuzz/corpus-run/. libFuzzer keeps an input because it reached new coverage, which is not the property the replay gate checks, so those inputs would be judged against a work floor written for curated seeds. Promote one deliberately when it covers a shape worth keeping: move it into fuzz/corpus/<target>/ and give it a name that says what it covers.

nine_p drives the 9P message parser directly; nine_p_transport drives the device through its MMIO registers and guest memory, covering the queue programming and descriptor walk beneath it. The split matters: the transport parses guest data of its own — descriptor lengths size host allocations and next indices steer the walk — and a harness aimed at the message parser never reaches it.

virtio_net and virtio_blk follow nine_p_transport: the input carries a guest memory image, then a bounded sequence of register writes chosen from a fixed list of queue bring-up registers, so one input lays out the rings, programs them, and kicks them. virtio_net adds two operations that hand the device an inbound frame, because the RX walk runs only when there is a frame to scatter. One pushes the frame onto the queue the net-poll thread fills and flushes it into the RX ring. The other buffers it on the device until the next RX kick. virtio_blk creates its backing disk under the scratch root the caller gives it. The disk is host-side, so a small fixed image serves, and the input goes to the request chains. fuzz/seeds/virtio_devices.py writes both seed corpora and documents their byte layout; regenerate from it after any change to a harness's read order.

When a run crashes, cargo fuzz writes the input under fuzz/artifacts/<target>/. Commit that file together with the parser fix — the replay test then keeps the bug fixed for everyone. Never commit a crashing input ahead of its fix: it reds the gate for every unrelated change.

Adding a target means adding a [[bin]] to fuzz/Cargo.toml, a harness body in src/fuzz.rs, an arm in tests/fuzz_corpus.rs, at least one seed under fuzz/corpus/<target>/, and a work floor in work_floor. every_fuzz_target_is_replayed fails if any of the middle three is missing, so a target cannot end up fuzzed but unguarded.

A harness returns the units of work it performed — frames decoded, packets routed, chains popped, requests dispatched, registers written — and the replay test holds every seed to that target's floor. Panic-freedom alone does not show a harness still reaches its parser: one that reaches nothing returns cleanly and replays green, so the count is what makes an inert target visible. A floor only bites where the harness drives its parser from a loop over the fuzzer's bytes, because that is what a consumption bug can starve; a harness that hands the raw input straight to its parser cannot go inert that way, and takes a floor of zero. Set a floor from what the seeds actually do, with enough margin that trimming one does not trip it, and check it against the negative-space seeds — an input written to prove the parser rejects something does zero units of successful work by construction. The floor applies to seeds only; a crash artifact may reach its bug before doing any countable work. Assert nothing about the count inside a harness body — under cargo fuzz an input that does no work is an ordinary outcome, and an assertion there would be reported as a crash.

A harness parameterizes itself from the raw input through a hand-rolled byte reader rather than the arbitrary crate. The committed corpus is tied to the exact byte-consumption order, and a crate upgrade that reorders its reads would silently repoint every seed at a different scenario.

Test initramfs and BusyBox

scripts/build_test_image.sh builds a minimal initramfs with guest-agent (as /init) and claudio (as /usr/local/bin/claude-code). The script auto-detects a statically linked BusyBox on the host and includes it if found. If BusyBox is not auto-detected, set the BUSYBOX env var explicitly:

BUSYBOX=/path/to/busybox-static scripts/build_test_image.sh

Ubuntu caveat (applies to build_guest_image.sh too): installing busybox-static places the static binary at /bin/busybox, but the dynamic busybox package's /usr/bin/busybox shadows it in PATH-based auto-detection — the build then silently produces an image with no /bin/sh. Pass BUSYBOX=/bin/busybox explicitly on Ubuntu hosts.

BusyBox provides /bin/sh and common utilities (echo, cat, mkdir, rm, mv, chmod, stat, dd, ls, wc, test, grep, sed, find, etc.) inside the guest. Without BusyBox, any test that runs sh -c "..." will fail with No such file or directory. The e2e_mount tests require BusyBox because they use shell commands to exercise the mounted filesystem.

9p kernel module loading order

The guest-agent loads kernel modules at boot time from /lib/modules/ inside the initramfs. For 9p shared mounts, the dependency chain must be loaded in order:

netfs.ko → 9pnet.ko → 9p.ko → 9pnet_virtio.ko

This order is enforced in guest-agent/src/main.rs (load_kernel_modules() ~line 540). The corresponding modules must also be included in the initramfs by scripts/build_test_image.sh. The overlay.ko module is also included for OCI rootfs overlay support. If any module in the chain is missing from the initramfs, 9p mounts will hang or fail silently.

Validation contract

Use this contract when shipping VM/OCI/OpenClaw changes. Commands are ordered and all required gates are explicit.

Preconditions

• Use repo-local temp dir to avoid /tmp pressure:

  export TMPDIR=$PWD/target/tmp
  mkdir -p "$TMPDIR"

• Set kernel once (Linux: host kernel; macOS: scripts/download_kernel.sh then VOID_BOX_KERNEL=target/vmlinux-arm64):

  export VOID_BOX_KERNEL=/boot/vmlinuz-$(uname -r)   # Linux
  export VOID_BOX_KERNEL=target/vmlinux-arm64        # macOS

• Guest image builds compile guest-agent for the musl target: rustup target add $(uname -m)-unknown-linux-musl once per host. .cargo/config.toml pins the linker to <arch>-linux-musl-gcc; on hosts without a musl cross-toolchain (e.g. Fedora), point cargo at the toolchain's bundled LLD instead — guest-agent is pure Rust and links statically with it (verified on x86_64 and aarch64):

  export CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER=rust-lld    # x86_64 host
  export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=rust-lld   # aarch64 host

• VM suites need usable KVM/vsock (not only device presence). A VOID_BOX_INITRAMFS that is set but points at a missing file is a configuration error and panics; unset auto-provisions. A genuine incapability skips, and the reason prints only with --nocapture, so a sub-second green suite still means nothing booted — set VOID_BOX_REQUIRE_VM=1 on a machine you believe is capable to turn that skip into a failure.

Standard validation sequence

Same on Linux and macOS — guest-agent builds as a no-op stub on non-Linux, so no per-crate exclusion is needed:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-features

macOS dev setup

After a fresh cargo build, codesign the binary so VZ accepts it: bash scripts/sign-macos.sh. Re-run after every rebuild — the script is idempotent and a no-op on non-Darwin hosts.

aarch64 cross-check (required when touching src/vmm/arch/aarch64/ or arch-neutral VMM code)

CI runs on native aarch64 (ubuntu-24.04-arm). To catch issues locally from an x86_64 host without waiting for CI:

# One-time setup (Fedora):
#   sudo dnf install -y gcc-aarch64-linux-gnu sysroot-aarch64-fc43-glibc
#   rustup target add aarch64-unknown-linux-gnu

CFLAGS_aarch64_unknown_linux_gnu="--sysroot=/usr/aarch64-redhat-linux/sys-root/fc43" \
  RUSTFLAGS="-D warnings" \
  cargo check --target aarch64-unknown-linux-gnu -p void-box --lib --tests

Common aarch64 pitfalls:

• kvm-bindings constants use kvm_device_type_ prefix (e.g. kvm_device_type_KVM_DEV_TYPE_ARM_VGIC_V3, not KVM_DEV_TYPE_ARM_VGIC_V3).
• kvm-ioctls get_preferred_target() takes &mut kvm_vcpu_init out-param.
• libc::SYS_epoll_wait and libc::SYS_poll don't exist on aarch64 — use SYS_epoll_pwait and SYS_ppoll.
• Unused variables/imports that are only used on x86_64 become errors with -D warnings on aarch64.

VM suites (required for VM/OCI/OpenClaw changes)

Linux (KVM):

# Generic VM suites
export VOID_BOX_INITRAMFS=/tmp/void-box-rootfs.cpio.gz
cargo test --test conformance -- --ignored --test-threads=1
cargo test --test oci_integration -- --ignored --test-threads=1

# Snapshot integration (required for snapshot/restore/vCPU/barrier changes)
scripts/build_test_image.sh
export VOID_BOX_INITRAMFS=/tmp/void-box-test-rootfs.cpio.gz
cargo test --test snapshot_integration -- --ignored --nocapture --test-threads=1

# Linux-only deterministic e2e suites (claudio + busybox)
cargo test --test e2e_telemetry -- --ignored --test-threads=1
cargo test --test e2e_skill_pipeline -- --ignored --test-threads=1
cargo test --test e2e_mount -- --ignored --test-threads=1

# Sidecar (deterministic — runs in CI; also builds on macOS/VZ):
cargo test --test e2e_sidecar -- --ignored --test-threads=1

# Agent suites. These need the production image and a real credential, and no
# CI pull-request runner has one, so run them here.
# VOID_BOX_REQUIRE_AGENT_CREDS=1 turns a missing credential into a failure
# instead of a green skip.
scripts/build_claude_rootfs.sh
export VOID_BOX_INITRAMFS=$PWD/target/void-box-claude.cpio.gz
export VOID_BOX_REQUIRE_AGENT_CREDS=1
ANTHROPIC_API_KEY=... cargo test --test e2e_service_mode -- --ignored --test-threads=1
ANTHROPIC_API_KEY=... cargo test --test e2e_agent_mcp -- --ignored --test-threads=1

macOS (VZ):

scripts/download_kernel.sh
export VOID_BOX_KERNEL=target/vmlinux-arm64
export VOID_BOX_INITRAMFS=/tmp/void-box-rootfs.cpio.gz
cargo test --test conformance -- --ignored --test-threads=1
cargo test --test oci_integration -- --ignored --test-threads=1
cargo test --test e2e_mount -- --ignored --test-threads=1
# The VZ guest→host gateway runs nowhere else — CI has no VZ VM lane (#158).
cargo test --test e2e_sidecar -- --ignored --test-threads=1

# VZ snapshot round-trip (requires test initramfs):
scripts/build_test_image.sh
export VOID_BOX_INITRAMFS=/tmp/void-box-test-rootfs.cpio.gz
cargo test --release --test snapshot_vz_integration -- --ignored --test-threads=1

e2e_telemetry and e2e_skill_pipeline are Linux-only (cfg(target_os = "linux")) and are not expected to run on macOS. Three others carry no such guard and do build on a Mac: e2e_sidecar, e2e_agent_mcp, and e2e_service_mode (cfg(any(linux, macos))). CI runs no VZ VM suites at all (#158), so a contributor's Mac is the only place e2e_sidecar ever exercises the VZ guest→host gateway — worth running there when touching sidecar or backend gateway code. The two agent suites additionally need the production image and a credential.

Interactive PTY / shell validation

When touching interactive PTY behavior, terminal handling, guest console routing, or shell-specific logging, do a manual smoke check on the relevant host platforms in addition to automated tests.

Linux (KVM):

export VOID_BOX_KERNEL=/boot/vmlinuz-$(uname -r)
export VOID_BOX_INITRAMFS=/tmp/void-box-rootfs.cpio.gz
target/release/voidbox shell --program sh --mount "$PWD:/workspace:rw"

macOS (VZ):

export VOID_BOX_KERNEL=target/vmlinux-arm64
export VOID_BOX_INITRAMFS=/tmp/void-box-rootfs.cpio.gz
target/release/voidbox shell --program sh --mount "$PWD:/workspace:rw"

For both platforms, verify:

• the PTY stays interactive and does not exit immediately
• terminal resize is reflected in the guest
• runtime logs and guest console output do not corrupt the interactive terminal
• guest console routing honors the configured host sink

OpenClaw production validation

OpenClaw gateway must run on the production image:

TMPDIR=$PWD/target/tmp scripts/build_claude_rootfs.sh
export VOID_BOX_INITRAMFS=$PWD/target/void-box-claude.cpio.gz

Then validate gateway workflow:

export TELEGRAM_BOT_TOKEN=...
export TELEGRAM_CHAT_ID=...
export ANTHROPIC_API_KEY=...
cargo run --bin voidbox -- run --file examples/openclaw/openclaw_telegram.yaml

Or validate the Ollama-backed gateway workflow:

export TELEGRAM_BOT_TOKEN=...
export TELEGRAM_CHAT_ID=...
export OLLAMA_BASE_URL=http://10.0.2.2:11434
export OLLAMA_API_KEY=ollama-local
export OLLAMA_MODEL=qwen2.5-coder:7b
ollama serve
ollama pull qwen2.5-coder:7b
cargo run --bin voidbox -- run --file examples/openclaw/openclaw_telegram_ollama.yaml

Do not use /tmp/void-box-test-rootfs.cpio.gz for OpenClaw gateway validation.

Exit gates

• fmt, clippy, and workspace tests must pass.
• VM suites must either pass or skip with explicit environment reason.
• OpenClaw validation must use production initramfs and reach startup/interaction.

Run examples

Use placeholders for secrets (...) and keep real tokens out of docs/commits.

Baseline smoke spec (auto-resolves images):

cargo run --bin voidbox -- run --file examples/specs/smoke_test.yaml

OCI node sanity (auto-resolves images):

cargo run --bin voidbox -- run --file examples/openclaw/node_version.yaml

To use a manually built image instead of auto-resolution, set the env vars:

export VOID_BOX_KERNEL=/boot/vmlinuz-$(uname -r)
export VOID_BOX_INITRAMFS=/tmp/void-box-rootfs.cpio.gz
cargo run --bin voidbox -- run --file examples/openclaw/node_version.yaml

OpenClaw Telegram gateway (production path):

export TELEGRAM_BOT_TOKEN=...
export TELEGRAM_CHAT_ID=...
export ANTHROPIC_API_KEY=...
cargo run --bin voidbox -- run --file examples/openclaw/openclaw_telegram.yaml

To use a locally built production image instead of auto-resolution:

TMPDIR=$PWD/target/tmp scripts/build_claude_rootfs.sh
export VOID_BOX_KERNEL=/boot/vmlinuz-$(uname -r)
export VOID_BOX_INITRAMFS=$PWD/target/void-box-claude.cpio.gz

OpenClaw Telegram gateway with host Ollama (production path):

export TELEGRAM_BOT_TOKEN=...
export TELEGRAM_CHAT_ID=...
export OLLAMA_BASE_URL=http://10.0.2.2:11434
export OLLAMA_API_KEY=ollama-local
export OLLAMA_MODEL=qwen2.5-coder:7b
ollama serve
ollama pull qwen2.5-coder:7b
cargo run --bin voidbox -- run --file examples/openclaw/openclaw_telegram_ollama.yaml

Do not use /tmp/void-box-test-rootfs.cpio.gz for OpenClaw gateway runs. That test image is only for deterministic claudio suites.

For the full catalog, see examples/README.md and examples/openclaw/README.md.

Deterministic e2e suites (test image with claudio + BusyBox):

scripts/build_test_image.sh
export VOID_BOX_KERNEL=/boot/vmlinuz-$(uname -r)
export VOID_BOX_INITRAMFS=/tmp/void-box-test-rootfs.cpio.gz
cargo test --test e2e_telemetry -- --ignored --test-threads=1
cargo test --test e2e_skill_pipeline -- --ignored --test-threads=1
cargo test --test e2e_mount -- --ignored --test-threads=1

Do not use target/void-box-claude.cpio.gz or target/void-box-codex.cpio.gz for these deterministic e2e suites. That production image is for real Claude/OpenClaw runtime paths. e2e_telemetry and e2e_skill_pipeline are Linux/KVM only. e2e_mount runs on both Linux (KVM, virtio-9p) and macOS (VZ, virtiofs).

Known issues

Vsock control channel timeout with large initramfs or missing ip

Symptom: control_channel: deadline reached (connect or handshake) — the host connects via AF_VSOCK but gets ECONNRESET on every attempt.

Root causes (two independent issues):

1. Missing ip binary → Command::new("ip").output() hangs PID 1. The guest-agent's setup_network() calls run_cmd("ip", ...) which internally does fork+execvp. When ip is not in PATH (e.g. no busybox symlink), execvp fails — but in the minimal initramfs PID 1 environment, the Rust Command::output() call can hang indefinitely instead of returning Err(ENOENT). The guest-agent never reaches create_vsock_listener(). Fix: Ensure ip (and which) are included as busybox symlinks in the initramfs (scripts/build_test_image.sh).
2. Large production initramfs (100+ MB) exceeds boot timeout. The control channel allows 4 s boot wait + 30 s connect deadline (34 s total). A 100 MB+ initramfs (with real claude-code, glibc, git, etc.) takes longer to decompress, load modules, and complete network setup — especially with only 256 MB of guest RAM. Fix: Use at least 3 GB of guest memory for production initramfs (memory_mb(3072)) so the kernel can decompress and boot within the timeout window. On hosts where boot is legitimately slow even with correct sizing — nested virtualization (e.g. a Lima validation VM) makes the production image take ~90 s+ — extend the connect deadline with VOID_BOX_CONNECT_DEADLINE_SECS (opt-in; can only lengthen the default, never shorten it).

Debugging tip: Boot a MicroVm directly with loglevel=7 in the kernel cmdline and read vm.read_serial_output() to see guest-agent progress messages. With loglevel=0, /dev/kmsg writes don't reach the serial console. Increase the serial channel buffer from 4096 to 65536 if kernel messages overflow it.

Snapshot restore: XCR0 and LAPIC timer

Root causes of guest crash after snapshot restore (2-day debugging effort):

1. Missing XCR0 restore → FPU #GP → kernel panic → reboot loop. After restore, XCR0 defaults to x87-only (bit 0). The guest kernel's XRSTORS instruction references SSE/AVX features in the XSAVE compact format, but those features aren't enabled in XCR0 → General Protection fault → "Bad FPU state detected at restore_fpregs_from_fpstate" → panic → emergency_restart writes 0xFE to port 0x64 (keyboard controller reset). Fix: Capture kvm_xcrs via KVM_GET_XCRS and restore via KVM_SET_XCRS before KVM_SET_XSAVE. Added xcrs field to VcpuState (backward-compatible via #[serde(default)]).
2. LAPIC timer masked in NO_HZ idle → scheduler never runs. When the guest is idle at snapshot time, the kernel disables the LAPIC timer (LVTT=0x10000, masked, vector=0). After restore the timer stays masked — no tick ever fires, scheduler stalls, vsock never processes packets. Fix: Detect masked+vector=0 state and bootstrap a periodic LAPIC timer (mode=periodic, vector=0xEC, TMICT=0x200000, TDCR=divide-by-1).
3. CID mismatch → guest silently drops vsock packets. Guest kernel caches CID during virtio-vsock probe at cold boot. If restore assigns a new random CID, the guest drops all incoming packets (dst_cid doesn't match cached value). Fix: snapshot_internal() stores self.cid in the snapshot; from_snapshot() reuses that CID.
4. Missing IA32_XSS restore → XRSTORS #GP → stack overflow → kernel panic. On kernels with CET (Control-flow Enforcement Technology, kernel ≥ 6.x on Fedora/Ubuntu), IA32_XSS (MSR 0x0DA0) has bits 11-12 set (CET_U/CET_S). The guest kernel's XRSTORS instruction uses the combined mask XCR0 ∪ IA32_XSS to restore process FPU state in compacted format. After snapshot restore, IA32_XSS defaulted to 0 because the MSR was not captured. XRSTORS then #GP'd on the CET features referenced by the fpstate's xcomp_bv but not enabled in XCR0 ∪ IA32_XSS. The #GP triggered the kernel's fixup_exception handler, which itself tried to context-switch (calling restore_fpregs_from_fpstate again), causing a recursive #GP that overflowed the TASK stack guard page → kernel panic → VcpuExit::Shutdown. Fix: Added IA32_XSS (0x0DA0) to SNAPSHOT_MSR_INDICES. Safe on older CPUs: the per-MSR capture loop silently skips unsupported MSRs, and restore only writes MSRs present in the snapshot.

Debugging tip: Guest reboots (port 0x64 write of 0xFE) indicate kernel panic, not a hang. Add panic=-1 loglevel=7 earlyprintk=serial to kernel cmdline and read serial output via vm.read_serial_output() to capture the actual panic message. Increase the serial channel buffer from 4096 to 65536 if kernel messages overflow it.

Key files: src/vmm/cpu.rs (capture/restore order), src/vmm/mod.rs (CID preservation), src/vmm/snapshot.rs (VcpuState with xcrs field), src/vmm/arch/x86_64/cpu.rs (SNAPSHOT_MSR_INDICES, capture_vcpu_state, restore_vcpu_state).

EPERM during OCI layer unpack

Container images (especially multi-layer ones like alpine/openclaw) can trigger Operation not permitted (os error 1) during tar extraction. The EPERM-resilient unpack code in voidbox-oci/src/unpack.rs handles files, symlinks, dirs, and hardlinks — but any bare ? on tar entry operations (e.g. entry.path()?) will bypass the EPERM handling and surface as an OciError::Io. When debugging OCI unpack failures, check for bare ? on entry.path(), entry.link_name(), or entry.unpack() calls that don't go through the EPERM catch-and-skip pattern.

Conformance expectations

• conformance: command execution, lifecycle, streaming, filesystem primitives.
• oci_integration: image pull/extract, rootfs mounting, readonly invariants.
• kvm_integration: direct MicroVm and Sandbox::local() boots over the real vsock + guest-agent path, without the backend abstraction in between.
• telemetry: TelemetryBatch wire format, host-side TelemetryAggregator, and a subscribed telemetry stream from a booted guest.
• snapshot_integration: cold snapshot capture, vCPU state persistence (including XCR0/xsave), restore pipeline, exec on restored VM, VM stop after restore. Must pass for any changes to snapshot, restore, vCPU, vsock, or stop() code paths. Uses userspace virtio-vsock backend.
• snapshot_vz_integration (macOS only): VZ snapshot round-trip using Apple's saveMachineStateToURL:/restoreMachineStateFromURL: APIs. Cold boot → exec → snapshot → stop → restore → exec. Must pass for VZ backend snapshot/restore changes.
• e2e_telemetry: telemetry flow from guest to host pipeline.
• e2e_skill_pipeline: multi-stage skill execution in VM mode.
• e2e_mount: host↔guest directory sharing via virtio-9p (Linux) / virtiofs (macOS) — RW/RO, write, read, mkdir, rename, delete, chmod, large files, pre-existing content, empty dirs.
• e2e_service_mode: service lifecycle — output publication while running, graceful stop, exit status mapping. Must pass for changes to service mode, daemon service lifecycle, or ServiceStageHandle. Needs the production image and a real credential, so it runs locally — or by manual dispatch of e2e-agent.yml — never on the pull-request lane.
• e2e_sidecar: sidecar intent flow — submit, inbox polling, context identity, idempotency. Must pass for changes to sidecar state, types, or server.
• e2e_agent_mcp: end-to-end agent (currently Claude Code) with void-mcp tools inside a real VM. void-mcp itself is agent-agnostic; the test uses Claude as the consumer because it's the only LlmProvider wired to consume MCP today. Requires ANTHROPIC_API_KEY. Must pass for changes to void-mcp tools or MCP provisioning. Same placement as e2e_service_mode: local, or a manual dispatch of e2e-agent.yml.

Once a suite starts booting, only a genuine hypervisor absence skips it — the typed Error::HypervisorUnavailable from the /dev/kvm probe, or VZ reporting virt-less hardware (ADR-0011). Every other boot or RPC error fails, and so does an absent network: oci_integration pulls alpine:3.20, so on a machine with no outbound network those tests fail rather than skip. Run them where the network is.

Deliberate skips sit before that gate, and a green local run does not prove they ran. The agent suites skip without a staged production image (env_artifacts_or_skip) or without an agent credential (skip_without_agent_creds); proxy_real_upstream skips without a credential too, and needs no VM at all; guest_image_pull_and_extract skips when VOIDBOX_TEST_GUEST_IMAGE is unset and its default localhost:5555 registry is absent. VOID_BOX_REQUIRE_VM=1 and VOID_BOX_REQUIRE_AGENT_CREDS=1 convert the credential and artifact skips into failures; the CI lane sets VOIDBOX_TEST_GUEST_IMAGE for the last. Run with --nocapture to see which fired.

Guest image build scripts

scripts/build_guest_image.sh:

• Base/initramfs for normal VM runs and OCI-rootfs workflows.
• Does not require bundling production Claude runtime.
• Preferred for general development and most integration tests.

@docs/agents/claude.md

@docs/agents/codex.md

Recommended default:

• Use build_guest_image.sh for broad test cycles.
• Use build_claude_rootfs.sh for production Claude gateway/runtime validation.
• Use build_codex_rootfs.sh for Codex CLI workflows.

Development image paths

Each build script writes to a distinct default output path so multiple flavors can coexist without overwriting each other. Set OUT_CPIO to override.

Which image for which test suite:

VM memory sizing

The VM's memory_mb must comfortably exceed the peak physical footprint during early-boot initramfs extraction. During that window the kernel holds both the compressed initramfs (loaded by the bootloader into guest RAM) and the decompressed tmpfs contents simultaneously, before releasing the compressed copy. If memory is exhausted mid-extraction the kernel silently stops creating files — the symptom surfaces as "file not found: /lib/modules/vsock.ko" inside the guest, with no obvious memory-pressure indicator.

Formula (INITRAMFS_OVERHEAD_BYTES = 208 MB covers kernel + runtime slack):

minimum_memory_mb = compressed_mb + uncompressed_mb + 208

Quick reference for the production image (build_claude_rootfs.sh, ~96 MB compressed / ~278 MB uncompressed):

The default spec memory_mb (src/spec.rs : default_memory()) must be set high enough to accommodate the largest supported initramfs. The current default is 1024 MB.

Pre-flight check: BackendConfig::initramfs_memory_warning() reads the gzip ISIZE field at VM start and emits a WARN log if memory_mb is below the computed minimum. When adding a new backend or changing the default memory, verify this check still fires correctly for undersized configs.