Debugging

The rule

Never fix code before you understand why it broke. The temptation to "just make the test pass" or "just patch the symptom" leads to fragile fixes that hide deeper problems. Follow the three-step workflow below every time.

Choose the cheapest truthful feedback loop

Keep implementation moving. Use a minimal reproducer and focused TDD checks during coding, then return to the next implementation item. Broad qualification runs belong at the end of a major phase; a small fix does not justify another whole contract/workspace/VM/install run. The bounded wrapper retains timeout and cleanup without a routine machine lease.

Capsem now has two distinct execution surfaces:

• Just exposes the small public product interface.
• Python under build_system/builder/gate/ owns build/test/release plans, graph edges, locking, resources, evidence, and cleanup.

For a gate failure, debug the Python step and action that failed; do not infer the implementation from the Just recipe or recreate it with shell commands. Start with:

uv run --project build_system --frozen capsem-gate runs last --failed
uv run --project build_system --frozen capsem-gate <command> --dry-run
uv run --project build_system --frozen capsem-gate <command> --graph

Use the smallest focused pytest, cargo, pnpm, or script command for red/green work. Use just focus-test functional for focused integration feedback, and just fast-test for the fast gate itself. Use just test when the forward fix is ready for the complete local verification a release requires.

Direct diagnostics which may block, build, launch children, or wait on input must use the portable bounded-process wrapper rather than a bare shell command:

python3 build_system/scripts/ci/run-bounded-command.py --timeout-seconds 1800 -- <command> <args...>

The wrapper closes stdin, creates a dedicated process group, and terminates the whole group on timeout or interruption. This prevents a PTY-backed docker build -f -, compiler, test runner, or child helper from surviving its owning diagnostic. Pick a finite timeout appropriate to the focused operation; do not wrap just test or a release command, whose config-owned step timeouts, journal, teardown, and resumable graph are the authority.

Diagnostic continuation for a late gate failure

When a clean candidate fails after expensive predecessors have succeeded, use diagnostic continuation to reuse that retained state and reach the frontier quickly. The current CLI retains legacy flag names:

uv run --project build_system --frozen capsem-gate runs last --failed
uv run --project build_system --frozen capsem-gate candidate --prefix <retained-prefix> --from <failed-step>

The --from step runs; graph predecessors are reported as carried. Before using it, match the prefix and failed step to the origin run, confirm the required predecessor really completed there, and record the origin run ID in the diagnosis. Prefer a fresh focused test when the edit invalidates the expensive producer itself.

Carried steps with external runtime products are revalidated before resumed work. If one was reclaimed, use the exact owning --from label in the refusal; do not guess an earlier frontier or recreate the product by hand.

Call the result diagnostic-passed or diagnostic-failed, even if the current implementation still summarizes a zero exit as ok. It combines earlier outputs with current source and therefore answers only whether the new segment can proceed. It does not prove the complete current tree.

Never use diagnostic continuation with release-binaries or release-assets, and never let it authorize publication. After the fix, use the smallest owning focus-test group. Before release, run just test <commit> to success: the public release command requires its complete local journal for that commit or an identical Git tree by default. An explicitly authorized release force permits a focused test-only correction without repeating the complete local run; source guards and full hosted qualification remain. The unattended nightly scheduler is also exempt ([release].unattended_channels in config/gate.toml; see RELEASE.md section 8). The hosted lane then independently qualifies the artifact pairing it publishes.

Step 1: Reproduce with a test

Before touching any implementation code, write a test that captures the bug. This test must:

• Fail right now, demonstrating the broken behavior
• Be specific enough to distinguish the bug from correct behavior
• Live in the right test location (see dev-testing for where tests go)

If you can't reproduce it in a test, you don't understand it well enough to fix it. For VM-level issues, use capsem-doctor or write a targeted diagnostic command:

just exec "<command that triggers the bug>"

For telemetry issues, inspect a live session through the typed timeline, history, statistics, and security routes. Use check_session.py --db only on a coherent, quiesced evidence copy after its ledger worker has stopped; see /dev-session-debug.

Step 2: Diagnose the root cause

With a failing test in hand, investigate. Do not skip this step. Common diagnostic approaches:

MCP triage trio (run FIRST when an investigation is open-ended):

capsem_panics { since: "1h" }       # any Rust panic in any host log? -> rank highest
capsem_triage { id: "vm-1" }        # ranked recent ipc-warns + 4xx/5xx + slow_ops + session.db errors
capsem_timeline { id: "vm-1", trace_id: "<X>" }   # follow ONE logical operation across exec/tool/net/fs/model

These read post-W2 JSON logs (~/.capsem/run/{service,mcp,gateway,tray}.log + capsem-app's latest jsonl) and post-W6 session.db tables. The W4 target=fs op=fsync duration_ms=... markers feed capsem_triage's slow-op rank; the W3 schema_hash check appears in capsem_panics output as IPC handshake failed; refusing connection events. Always start with capsem_panics -- a single panic outranks a hundred warns.

Cross-version mix? The service.start log line emits protocol_version=N, schema_hash=<hex> per binary. If the support bundle (capsem support-bundle) shows two different schema_hash values across binaries, you're hitting the W3 cross-version-mix detection -- rebuild + restart the lagging binary.

Integration-test failures: read the preserved service log. When any integration test fails, the test fixture (tests/helpers/service.py::ServiceInstance, the e2e RealService, and the MCP _start_capsem_service) archives its tmp_dir to cache/target/tests/evidence/<timestamp>-<worker>-<nodeid>/<tmp-basename>/ before the usual rmtree. The failing test's stderr has the exact path: look for a line ARTIFACT: preserved /var/folders/... -> cache/target/tests/evidence/.... Inside that directory:

service.log                     host-side capsem-service debug log (RUST_LOG=debug)
logs/gateway.log                gateway stdout/stderr
logs/tray.log                   tray stdout/stderr (if spawned)
sessions/<vm-id>/process.log    per-VM capsem-process log (vsock bridge, IPC, spawn chain)
sessions/<vm-id>/serial.log     VM serial console (kernel boot, capsem-init, agent startup)
sessions/<vm-id>/session.db     SQLite telemetry DB (net_events, model_calls, ...)
persistent/<name>/...           persistent-VM state (checkpoint.vzsave, workspace)

cache/target/tests/evidence/ is gitignored. Multiple failures sharing a session-scoped service land in different subdirs but the latest run's name tags them by the most recent failing nodeid. First place to look for "VM didn't become exec-ready" style failures: sessions/<id>/serial.log (did the VM boot?) and sessions/<id>/process.log (did the agent come up + IPC handshake?). For "provision hung" or service-side contention: service.log, grep for the VM id.

Rust code: Read the code path the test exercises. Trace the data flow. Add tracing instrumentation if needed (RUST_LOG=capsem=debug). Check if the issue is in capsem-core, capsem-app, or capsem-agent.

Guest VM issues: Boot with targeted commands and inspect behavior:

just exec "capsem-doctor -k <category>"   # Run specific diagnostic category
just exec "<manual investigation command>"

Check boot logs for daemon startup failures, vsock connection issues, or timing problems.

Network/security issues: Check the network intercept path -- SNI parsing, HTTP/DNS/model normalization, cert minting, SecurityEvent construction, security rule evaluation, plugin execution, runtime materialization, and ledger materialization. Do not debug by adding credential handling to formatters, routes, DB readers, frontend transforms, or harnesses. Inspect the typed timeline and security routes to see what the live ledger recorded. For page or archive integrity, preserve a quiesced evidence copy and follow /dev-session-debug.

Frontend issues: Run just dev ui, open Chrome DevTools, check console errors, use take_screenshot to capture state. See dev-testing-frontend for the full visual verification workflow.

Build pipeline issues: Check cache/containers/logs/build.log -- all build infrastructure (runner, code signing, generation scripts) logs here. The runner (build_system/packaging/macos/run_signed.sh) and _generate-settings recipe both append to this file. Never write diagnostics to stdout from build scripts (it contaminates binary output like mcp-export).

For a Python gate or release failure, inspect the recorded run first. Step labels, argv, timing, captured output, resource events, and the first failed action live under cache/target/gate-runs/; capsem-gate runs last --failed is the supported reader. cache/containers/logs/build.log is supporting build evidence, not the gate's execution ledger.

Do not retry a release through its script, capsem-admin release, or a GitHub workflow. The public command first accepts the exact qualification journal, then runs prechecks, source publication, and dispatch as one graph. A direct script or workflow skips the edges that make release safe.

Telemetry pipeline issues: The canonical session ledgers (net_events, model_calls, tool_calls, tool_responses, fs_events, dns_events, security_rule_events) each have their own boundary. If a table is empty or has wrong data:

• Check if the guest daemon started (boot logs)
• Check if the vsock connection was accepted (host logs)
• Check timing -- did the VM shut down before the DB-owned buffer flushed? Use the DB flush barrier or shutdown/reopen proof; do not add a sleep or poll.

For route latency or stale stats, do not add service-owned logged-data projections. Logged-data hot state belongs inside the logger DB object as table-level mem/disk ownership with DB-layer tests and benchmarks. Service routes may describe the query they need, but production service code must not open rusqlite connections or DbReader directly.

Do not "fix" route latency by hardcoding route-specific query helpers in DbWriter, by adding service caches, or by swallowing missing tables/columns as empty data. The correct diagnosis target is the DB object: connection/thread ownership, mem/disk layout, batching, flush, and query execution. If the schema is missing, surface the broken ledger contract.

Write down what you find. The diagnosis should explain why the bug exists, not just where the symptom appears.

Concurrency flakes are product bugs, not test-tuning problems

just test runs the python suite under pytest -n 4 --dist=loadfile. Four real VMs boot in parallel; this is dogfooding. Capsem ships as a multi-VM sandbox for AI agents -- if the test suite cannot safely run 4 concurrent VMs, real users running an agent farm will hit the same bug. When a test flakes only under concurrency, the diagnosis target is Capsem's product code, not the test:

• "Suspend timed out" appearing only at -n 4 -> handle_suspend IPC race; investigate the with_quiescence path and the Suspend round-trip, not the test timeout
• "Session did not become ready" only with multiple parallel provisions -> Apple VZ resource contention, VirtioFS lock, or service handle_provision serialization gap
• Two tests collide on the same VM/session name -> validate_vm_name / persistent registry has a TOCTOU; UUID prefix in the test is not the bug
• "Connection refused" on a per-VM UDS only at -n 4 -> service spawned the process but didn't wait for the socket to be bound; race in the spawn path
• A test passes serial but hangs at n=4 -> a global lock somewhere (state mutex held across an await, blocking Tokio worker; or a sync std::Mutex on a hot path)

Anti-patterns to avoid:

• Adding time.sleep in the test "to let things settle"
• Bumping a per-test timeout from 30s to 120s "because it's flaky"
• Marking the test serial -- defeats the dogfooding signal
• Adding retries with backoff in the client

Right pattern: capture a service log of the failing run (set RUST_LOG=capsem=trace), find the operation that took unexpectedly long or returned an error, fix the underlying race in capsem-service / capsem-process / capsem-core. Then re-run at -n 4 to confirm.

Step 2.5: Fix the pattern, not the instance

For long HTTP operations, trace the complete deadline chain: service work, gateway response headers, generated operation, SDK facade and UI caller. Image pulls use the API-owned whole-operation budget and OpenAPI's x-capsem-request-timeout-ms default across all SDKs. An operation default extends a short transport default; an explicit per-call timeout still wins. Test ordinary calls, explicit deadlines, cancellation and absence of replay. Do not fix a gateway timeout by adding a longer timeout to one UI caller.

When diagnosis reveals a systemic pattern (the same mistake repeated across the codebase), the fix must cover every instance -- not just the one that was reported.

• Audit the entire codebase for the same pattern. If blocking I/O in async context caused one hang, grep for every other site that does the same thing. A bug is a symptom -- the pattern is the disease.
• Never simplify a fix to the minimum diff. A "quick fix" that patches one call site while 6 others have the identical problem is not a fix -- it's deferred breakage.
• Document the pattern in the relevant skill (e.g., dev-rust-patterns) so it's never reintroduced.
• Add tests that would catch the pattern if it recurs (e.g., a contract test between the frontend and backend response format).

Example: an MCP tool hang was caused by blocking I/O (clonefile, walkdir, blake3) on tokio worker threads. The same anti-pattern existed in 7 sibling tool handlers, a background timer, and asset hash verification. Fixing only the reported call would have left 9 other sites broken.

Step 3: Fix with a comprehensive solution

Now that you understand the root cause, write the fix. The fix should:

• Make your reproducing test pass
• Not break any existing tests (just test)
• Address the root cause, not just the symptom
• Include the test from Step 1 in the same commit

After the fix, run the full validation:

1. just test -- unit + cross-compile + frontend
2. just exec "capsem-doctor" -- VM smoke test
3. If the bug touched telemetry: inspect the real session through typed routes; run check_session.py --db only on a preserved, quiesced evidence copy

A diagnostic continuation may shorten investigation before this validation; it never replaces any item in the final proof.

Local/CI execution parity

When a bug appears only in CI, first identify the exact production entrypoint, runner dependency, architecture, environment variable, permission, device, or service-manager difference that local testing skipped. Reproduce every portable Linux difference in Docker and execute the same production entrypoint or shared predicate that CI executes. A hand-written approximation is not a regression test.

Keep an executable parity test after the fix and audit sibling workflows for the same one-sided assumption. If reproduction crosses an unavoidable platform boundary, document the boundary and preserve the nearest local contract plus the required owning release-job or physical-machine proof. Do not relabel an unreproduced CI failure as transient.

What NOT to do

• Do not "fix" a failing test by changing the test assertion. The test is telling you something. Listen to it. If the test is genuinely wrong, explain why in detail before changing it.
• Do not dismiss failures as "pre-existing" or "unrelated." Investigate every failure. If it truly is pre-existing, file it and fix it -- don't leave broken windows.
• Do not guess-and-check. Random changes hoping something sticks waste time and often introduce new bugs. Understand first, then act.
• Do not patch symptoms. If requests fail because gzip content-encoding isn't handled, don't strip the Accept-Encoding header -- implement proper decompression. Fix the system, not the surface.
• Do not apply narrow fixes to systemic problems. If the same anti-pattern exists in 7 places and you fix 1, you haven't fixed the bug -- you've hidden 6 more. Audit first, then fix all instances in a single pass.