Hypervisor Testing

Architecture

The hypervisor module (crates/capsem-core/src/hypervisor/) has:

• Traits: Hypervisor, VmHandle, SerialConsole in mod.rs
• Apple VZ backend: apple_vz/ -- macOS only, uses Virtualization.framework
• KVM backend: kvm/ -- Linux only, uses rust-vmm crates

Tests must cover both backends where possible. macOS CI tests Apple VZ, Linux CI (ubuntu-24.04-arm with /dev/kvm) tests KVM.

Unit tests

VirtioFS FUSE operations have 30+ unit tests in kvm/virtio_fs/mod.rs:

• File I/O: open, read, write, create, release, flush, fsync, lseek
• Directory ops: opendir, readdir, mkdir, rmdir, unlink, rename, symlink, link
• Metadata: lookup, getattr, setattr, statfs, forget
• Adversarial: path traversal, truncated requests, invalid opcodes

Run them:

cargo test -p capsem-core virtio_fs    # VirtioFS tests only
cargo test -p capsem-core hypervisor   # All hypervisor tests

On macOS these run the KVM module's pure-logic tests (FUSE parsing, FDT generation) but skip anything that needs /dev/kvm. On Linux CI, all tests run including KVM integration.

Integration tests

Cross-crate VM lifecycle tests in crates/capsem-core/tests/:

cargo test -p capsem-core --test '*'   # All integration tests

These test the full boot path: config validation, device setup, serial output, vsock handshake. They require VM assets to be built.

CI setup

macOS (ci.yaml, test job)

• Tests capsem-core, capsem-agent, capsem-logger, capsem-proto
• Cross-compile check for aarch64 + x86_64 musl targets
• No VM boot (no VZ entitlement in CI)

Linux (ci.yaml, test-linux job)

• Runs on ubuntu-24.04-arm with KVM enabled
• Tests capsem-core, capsem-logger, capsem-proto (KVM backend compiles + tests)
• Verifies /dev/kvm is available (fails CI if KVM tests were silently skipped)

Apple VZ restore errors on a locked host

VZErrorDomain12 / permission denied does not establish a missing owner filesystem grant. Inspect the Apple virtualization worker's logs as well as the Capsem owner's log. On a locked console, the worker has returned SecKeyCreateDecryptedDataWithParameters / OSStatus-25308 (errSecInteractionNotAllowed) while restoring saved memory. Ask for a normal Mac unlock and repeat the native proof before changing confinement. Incidental owner denials during VZ error localization are consequences, not the cause.

A failed restore must preserve the checkpoint and report failure. Verify the original foreground process with key-driven interaction after resume: a 200 response or a reused guest PID can conceal a cold boot.

KVM warm-checkpoint device state

A warm checkpoint must preserve host device-model state as well as guest RAM, vCPUs, and virtqueue indices. In particular, the guest retains VirtioFS inode numbers and file-handle IDs across resume, so restoring a fresh host-side FUSE processor is invalid even when the VM reaches Ready and answers pings.

When changing KVM checkpoint or VirtioFS state:

• prove backend state is captured after queue drain and restored before queue activation;
• keep checkpoint lengths/counts bounded before allocation and reject changed device/share identity;
• permit a deleted cache-only inode to leave the snapshot only when no open file/directory handle references it, and preserve next_ino so a recreated path cannot inherit the stale guest node ID;
• on Linux/KVM, test a real guest process that holds both a file FD and directory FD under /root across suspend, then uses both after resume without sleeps or retries; this proves Capsem's userspace KVM VirtioFS handle table, not Apple VZ's framework-owned device state;
• rebuild capsem-process before black-box testing, because it owns the KVM hypervisor, and run the lifecycle test serially against one exact runtime asset cohort.

x86_64 KVM boot: known pitfalls

The x86_64 KVM backend boots bzImage kernels in 64-bit long mode. Key invariants:

• Entry point is KERNEL_LOAD_ADDR + 0x200 (startup_64), not KERNEL_LOAD_ADDR (startup_32). Setting the wrong entry point causes a silent hang -- the vCPU executes 32-bit code in 64-bit mode.
• setup_header must be preserved. The bzImage setup header (bytes 0x1F1..0x2B9) must be extracted from the raw kernel and copied into boot_params. The kernel reads fields (vid_mode, heap_end_ptr, etc.) from this header at boot.
• #[cfg(target_arch = "x86_64")] hides x86 bugs on macOS. All KVM x86_64 code is behind cfg gates, so it never compiles on macOS (aarch64). Bugs in the x86_64 code path are invisible during macOS development. Always check that the x86_64 CI job passes.
• VmConfig validates kernel architecture. VmConfigBuilder::build() reads kernel magic bytes and rejects wrong-arch kernels (bzImage on aarch64, ARM64 Image on x86_64) with ConfigError::ArchMismatch instead of silently hanging.

What to test when changing hypervisor code

Rust async reference

Read references/rust-async-patterns.md for tokio patterns (tasks, channels, streams, error handling). Relevant for vsock, MITM proxy, and VirtioFS async worker code.

Security invariants to test

• VirtioFS path traversal: FUSE lookup must reject .. components
• Resource limits: file handle cap (4096), read size clamp (1MB), gather buffer limit (2MB)
• Read-only rootfs: squashfs lower layer must not be writable through overlay
• Guest binary integrity: binaries deployed chmod 555, guest cannot modify them