Validate on a real Incus

FakeIncusDaemon proves what isx asks for, not what Incus does with it. Incus has semantics a fake easily gets wrong: a profile device cannot be removed from one instance, only masked; PATCH merges while PUT replaces; expanded_devices differs from devices; options vary between Incus versions. Anything that depends on those needs a real daemon.

1. Check what CI already proved

CI runs isx against a real Incus on every PR: the isx-integration-tests-native job in .github/workflows/test-integration.yml, described in .claude/rules/ci.md.

gh pr checks <n>                       # did the real-Incus job pass?
gh api repos/Sanne/incus-spawn/actions/jobs/<job-id> \
  --jq '.steps[] | select(.conclusion=="failure") | .name'

If that job failed, find out whether the failure is in the area the change touched, and compare it with recent runs on main (gh run list --workflow test-integration.yml) before blaming the change. Then grep .github/scripts/ and the workflow for the feature. What CI does not exercise is what needs checking by hand. CI also runs its own Incus version (Zabbly's latest), not the one a distro ships, so version-dependent behaviour is not covered.

2. Set up

./install.sh                    # JVM build of the working tree to ~/.local/bin (fast enough)
scripts/local-incus.sh          # Incus + isx init as CI does it; idempotent

This session does not gain the incus-admin group the script adds, so wrap every incus and isx command in sg incus-admin -c "...", as CI does. Then:

sg incus-admin -c "isx proxy start"                 # foreground: run it ALONE in the background
sg incus-admin -c "isx build tpl-minimal --yes"

Never chain anything after isx proxy start: a command queued behind it runs later, whenever the proxy stops, e.g. a build that starts while you are tearing things down.

Inside an isx instance

The script detects a nested instance (a user namespace that maps only part of the ID space) and works around what the kernel refuses there, saying so as it goes:

• The bridge: it creates incusbr0 without Incus's firewall rules, which nftables refuses in a user namespace, and on a free 10.166.N.0/24, since isx's default is the outer isx's subnet.
• Container IDs: it gives Incus the top 65536 IDs this namespace maps, since the usual range is not mapped and every container would fail with "Failed to handle idmapped storage".
• The 443 redirect: it adds the redirect with nftables where isx init could not (no legacy iptables nat table).

What it cannot fix:

• Templates do not build. They add a tun device with mode, which Incus refuses in a nested container. Test on a plain container instead: incus launch images:fedora/43 <n>, then push ~/.config/incus-spawn/ca.crt to /etc/pki/ca-trust/source/anchors/ and run update-ca-trust. Its traffic takes the same bridge DNS -> redirect -> proxy path as a branch. It has no registered address or proof, so credential enforcement relays its own authentication without injecting an account. When the change is in the build itself, a measurement-only patch that drops both incus.deviceAdd(buildName, "tun", ...) calls in BuildCommand lets templates build here; apply it to both sides identically (see Pitfalls) and never commit it.
• Branches do not get a disk size. Every branch sets its root disk's size, which Incus turns into a btrfs qgroup limit, and the nested kernel refuses quotas (btrfs quota enable gives EPERM), so every isx branch fails with btrfs qgroup limit ... invalid qgroupid. For a change that does not touch disk limits, a measurement-only patch dropping update.device("root", "size", settings.disk()) in InstanceLifecycle.configureBranch gets past it, under the same rules; anything about --disk needs a non-nested host.
• The outer proxy sits between this one and the real upstream. It intercepts the same domains and speaks only HTTP/1.1. To reach the real host, run the proxy binary directly with the benchmark override: ISX_BENCH_UPSTREAM=<host>=<ip>:443 isx-proxy --gateway-ip <gw>, under sg incus-admin so it can read the Incus socket. Local DNS answers with the outer gateway, so resolve <ip> elsewhere, e.g. curl -s -H 'accept: application/dns-json' 'https://1.1.1.1/dns-query?name=<host>&type=A'.

3. Exercise the change

A branch that starts tries to attach a shell; without a TTY it says "No terminal, so no shell opened" and exits, so isx branch is safe to script. Use --no-start and incus start when the change is about what happens on a later start rather than at branch time. Cover every path the change touches, typically:

• plain branch; --cpu/--memory/--disk; --proxy-only; --airgap
• --account <ns>=<account>, with and without the template pinning accounts
• a source with user.incus-spawn.instance-mode=kvm plus kvm/vhost-vsock devices (you can add them to an incus copy of a template), branched by default, --kvm and --no-kvm
• the TUI (isx inside tmux new-session -d -x 200 -y 50, drive with tmux send-keys, read with tmux capture-pane -p), since it has its own branch flow in ListCommand

What to inspect

• incus config show <n> --expanded and incus query /1.0/instances/<n> | jq .expanded_devices. Always the expanded view: a device that is gone from devices can still come from a profile. That is how airgap branches kept their NIC unnoticed (#813).
• Behaviour from inside, not just config: incus exec <n> -- ip -4 -br addr, ip route, and a curl to the internet as well as to an intercepted domain.
• Which account the proxy serves: pin the instance to a missing account (incus config set <n> user.incus-spawn.account.claude=ghost), signal the proxy (kill -USR1 <pid>, the pid is in /health), and expect Account 'ghost' is not configured from that instance only. It proves the proxy maps the instance's source address to its own pins.
• Raw Incus semantics, when unsure what a request will do: incus query -X PATCH /1.0/instances/<n> -d '{"config":{"k":null}}' against a throwaway instance.

4. Compare against the base commit

A bug you find may predate the change. Build the base in a worktree ($SCRATCH being any scratch directory outside the repo) and run the same scenario:

git worktree add "$SCRATCH/base" <base-sha>
(cd "$SCRATCH/base" && mvn -q package -DskipTests)
sg incus-admin -c "java -jar $SCRATCH/base/cli/target/quarkus-app/quarkus-run.jar branch ..."

5. Time it, when the change is about speed

bench/trace-branch.sh records the daemon's events around one isx branch. Build native for both sides (mvn package -Dnative -DskipTests -pl cli -am), each in its own worktree ($SCRATCH/base as above, $SCRATCH/head for the change), and run each tree's own bench/trace-branch.sh: it takes the newest runner in that tree's cli/target. Alternate a few runs of each; a single run is noise. Compare the writes before PUT .../state, the time of the start request, and any gap after "Stopping forkfile" (a push still in flight makes the start wait a full second).

Pitfalls

• mvn verify -DskipITs=false is not this: it runs isx init against the host and its template build IT has no definition to build. Use the CLI as above.
• pkill -f <pattern> also kills the shell whose command line contains the pattern. Kill by pid.
• kill <pid> of isx proxy start stops the isx-proxy it started too (#882), but kill -9 cannot: the orphan keeps the ports and the next proxy fails with "Address already in use", or a leftover older build keeps serving. Find it with sudo ss -ltnp | grep 18443 and kill that pid.
• Don't build the other side by git stash-ing the main tree: a git stash; <build>; git stash pop chain run in the background keeps your edits away until the job ends, and anything you edit meanwhile collides with the pop. Use a worktree per side, as in section 5.
• A patch needed only to measure (e.g. to get past something this host refuses) goes into both sides' worktrees identically, and is never committed.
• Don't ./install.sh (JVM) from a worktree: its launchers run the jar where it was built, so removing the worktree breaks the installed isx and the proxy. Install from the main checkout.
• Inside an isx instance, the outer isx's DNS and MITM proxy sit behind this one. A certificate issued by "incus-spawn MITM CA" might be the outer one: check which domains the inner proxy actually overrides (incus network get incusbr0 raw.dnsmasq) before reading a result.

Report

Say which scenarios ran and which did not, what was compared against the base, and file anything found that the change did not cause as its own issue rather than folding it into the review.

Clean up the instances you created (incus delete -f ...).