Profile a VoidBox Process

Profile CPU, off-CPU, and hardware counters using the eBPF-based perf-agent tool.

Prerequisites

Tell the user FIRST before doing anything else:

perf-agent requires Linux capabilities to run without sudo. Please run:

sudo setcap 'cap_sys_admin,cap_bpf,cap_perfmon,cap_sys_ptrace,cap_checkpoint_restore=ep' $HOME/.local/bin/perf-agent

(Only needed once after each download.)

Note: The binary must NOT be on a nosuid filesystem (e.g. /tmp), as file capabilities are silently ignored there.

Do NOT attempt to run sudo yourself. Ask the user and wait for confirmation.

Setup

Run these checks yourself (no sudo needed):

1. Detect architecture and download if needed:

ARCH=$(uname -m)
case "$ARCH" in
  x86_64)  SUFFIX="linux-amd64" ;;
  aarch64) SUFFIX="linux-arm64" ;;
esac

mkdir -p "$HOME/.local/bin"

LATEST=$(curl -s https://api.github.com/repos/dpsoft/perf-agent/releases/latest | grep '"tag_name"' | cut -d'"' -f4)

if [ -x "$HOME/.local/bin/perf-agent" ]; then
  CURRENT=$("$HOME/.local/bin/perf-agent" --version 2>/dev/null || echo "unknown")
  if echo "$CURRENT" | grep -q "${LATEST#v}"; then
    echo "perf-agent up-to-date ($LATEST)"
  else
    echo "Downloading perf-agent $LATEST..."
    curl -fSL -o "$HOME/.local/bin/perf-agent" "https://github.com/dpsoft/perf-agent/releases/download/${LATEST}/perf-agent-${SUFFIX}"
    chmod +x "$HOME/.local/bin/perf-agent"
  fi
else
  echo "Downloading perf-agent $LATEST..."
  curl -fSL -o "$HOME/.local/bin/perf-agent" "https://github.com/dpsoft/perf-agent/releases/download/${LATEST}/perf-agent-${SUFFIX}"
  chmod +x "$HOME/.local/bin/perf-agent"
fi

After download or upgrade, ask the user to run the setcap command.

2. Verify Go (required for pprof analysis):

go version

Find the PID

Auto-detect the running voidbox process:

pgrep -fa voidbox

If multiple processes, pick the voidbox shell one. If none found, tell the user to start a voidbox process first.

Collect

$HOME/.local/bin/perf-agent --pid <PID> --profile --offcpu --pmu \
  --duration 60s \
  --profile-output profile.pb.gz \
  --offcpu-output offcpu.pb.gz \
  --pmu-output pmu.txt

perf-agent flags

Analyze

go tool pprof -text -nodecount=50 -cum profile.pb.gz    # call tree view
go tool pprof -text -nodecount=50 -cum offcpu.pb.gz     # off-CPU call tree
go tool pprof -text -nodecount=20 profile.pb.gz          # flat hotspots
cat pmu.txt                                              # PMU metrics

Performance Investigation

Use this workflow to find bottlenecks in a running (non-frozen) VM.

1. Baseline profile — capture 60s under normal workload.

2. Read PMU summary from pmu.txt:

• IPC < 0.8 → memory-bound, look for cache thrashing
• IPC > 1.5 → compute-bound, look for hot loops
• Cache misses > 10/1K → data structures not cache-friendly
• P99.9 on-CPU > 100ms → some thread holding CPU too long
• Preempted > 20% → CPU saturation

3. CPU hotspots — functions consuming > 5% of flat CPU:

• Packet parsing (BigEndian::read_u16, checksum) → optimize hot path
• Filesystem ops (statx, getdents) → async or batch I/O
• Lock overhead (MutexGuard::drop) → reduce critical section
• Memory allocation (malloc, Vec::new) → pre-allocate or pool

4. Off-CPU bottlenecks — unexpected blocking:

• Mutex::lock > 3% → lock contention, reduce lock scope
• read/write on vCPU thread → blocking I/O on wrong thread
• nanosleep in net-poll → normal (5ms poll interval)
• epoll_wait in tokio → normal (idle reactor)

5. Thread-level snapshot with gstack:

gstack <PID> 2>&1 | awk '/^Thread/{t=$0} /void_box|slirp|virtio|vcpu_run|pty/{if(t){print t; t=""} print $0}'

6. Compare after optimization — re-profile and check IPC, cache misses, P99.9, and whether the target function dropped from top-20.

Frozen VM Triage

When a VM freezes (PTY stops producing output):

1. gstack first — identify which thread is stuck:

gstack <PID> 2>&1 | awk '/^Thread/{t=$0} /void_box|slirp|virtio|vcpu_run|pty|forward_dns/{if(t){print t; t=""} print $0}'

Key patterns:

• vcpu-N NOT in KVM_RUN → stuck in host-side handler (blocking I/O)
• vcpu-N in KVM_RUN + PTY at read_from_sync → guest process stopped output
• vcpu-N in KVM_RUN + PTY at writer_handle.join() → guest sent PtyClosed
• All threads normal → guest-side issue, check exit code

2. Profile for 30s to capture PMU metrics.

3. Exit code after process dies:

echo $?
# 153 = SIGXFSZ (file size limit), 137 = SIGKILL (OOM), 139 = SIGSEGV

PMU Reference

VoidBox Thread Reference

Known Issues

• vcpu-N stuck in forward_dns_query: DNS blocking vCPU (fixed in 50cdb9f)
• vcpu-N stuck in TcpStream::connect_timeout: TCP connect blocking vCPU (known, not yet fixed)
• Exit code 153 (SIGXFSZ): File size limit exceeded (fixed in cb2e2b1 for interactive mode)
• High CPU in Virtio9pDevice::handle_readdir: Host FS ops on vCPU thread
• Process dies during profiling: Profiles may not flush — rely on pmu.txt or stdout

Comparing Profiles