Pumba Development Guide

Build & Test Commands

• Build: make build (builds for current TARGETOS/TARGETARCH)
• Full pipeline: make all (format → lint → test → build)
• Unit tests: make test (requires CGO_ENABLED=1 for race detector)
• Test with coverage: make test-coverage
• Race detector: make test-race (linux/amd64 only)
• Lint: make lint (runs golangci-lint with .golangci.yaml)
• Format: make fmt
• Cross-compile: make release (darwin/linux/windows × amd64/arm64)
• Integration tests: make integration-tests (requires Docker, uses bats)
• All integration tests: make integration-tests-all (includes stress tests)
• Advanced Go integration tests: make integration-tests-advanced (Go-based tests in tests/integration/)
• Generate mocks: make mocks (uses mockery)
• Unit tests in sandbox: CGO_ENABLED=0 go test ./... (skips race detector, works without CGO toolchain)

Integration Testing

Integration tests use bats:

• Tests are in tests/*.bats with helpers in tests/test_helper.bash; install Bats >= 1.5.0 and the bats-support/bats-assert libraries at /usr/local/lib/
• Result-event and schema tests (tests/output.bats, tests/webhook.bats, tests/*status.bats, tests/schema.bats, ndjson smoke in *_global_flags.bats) need jq and use run --separate-stderr to assert on stdout alone
• Webhook HTTP acceptance (tests/webhook.bats) requires python3 (stdlib only); missing Python fails rather than skips. The integration-tests Docker stage includes it; the release image does not. Run fixture safety checks offline with bash tests/unit/webhook_fixtures_test.bash (no runtime needed)
• Run all tests locally (recommended): colima ssh -- sudo bats tests/*.bats tests/containerd_*.bats
  • Colima VM has native Docker + containerd sockets; sudo is required for containerd sidecar tests
• Docker tests only (via Docker image): docker run --rm -v /var/run/docker.sock:/var/run/docker.sock --entrypoint bats pumba:test tests/*.bats
• Containerd tests only: colima ssh -- sudo bats tests/containerd_*.bats
• Podman tests only (macOS, inside podman machine VM): podman machine ssh sudo bats tests/podman_*.bats
• Podman tests only (Linux, rootful): sudo bats tests/podman_*.bats
• CI builds a Docker image (pumba:test target integration-tests) and runs bats inside it
• Rebuild test image after code changes: docker build --target integration-tests -t pumba:test -f docker/Dockerfile .
• Copy updated binary to Colima: colima ssh -- sudo cp /Users/alexei/workspace/pumba/.bin/linux/pumba /usr/local/bin/pumba
• Bats teardown: Use sudo pkill -f "pumba.*<container-name>" to stop background pumba processes; kill %1 for job control fallback

Technical Stack

• Go version: 1.27 (see go.mod)
• CLI framework: github.com/urfave/cli (v1)
• Docker SDK: github.com/docker/docker v28.5.2
• Containerd SDK: github.com/containerd/containerd/v2 (containerd runtime support)
• Error handling: stdlib fmt.Errorf("...: %w", err) wrapping
• Logging: github.com/sirupsen/logrus
• Testing: github.com/stretchr/testify (assert, mock, require)
• Mocking: github.com/vektra/mockery/v3 v3.7.4
• Linting: golangci-lint with .golangci.yaml
• CI: GitHub Actions (build.yaml, release.yaml, codeql-analysis.yml, nettools-images.yaml)

Project Structure

cmd/
  main.go              — CLI entry point: main() → run(args, stdout, stderr), signal context, app construction
  events.go            — --output/--run-id handling, event fan-out to bounded per-sink queues (stdout NDJSON, webhook), run.end + exit code
  webhook.go           — --webhook sink: URL check, one POST per event, no retries/redirects, one stderr warning
  runtime.go           — createRuntimeClient, tlsConfig, runtime factory vars (newDockerClient/newContainerdClient/newPodmanClient)
  logging.go           — setupLogging: log-level switch + Slack hook wiring
  slackhook.go         — Best-effort async Slack webhook logrus hook + shutdown drain
  plan.go              — text-mode netem/iptables dry-run renderer (would run/apply/revert); NDJSON stays event-only
  status.go            — text-mode `pumba status` table (TARGET KIND DETAIL RUN) + verdict line; a StatusRecorder subscribed in text mode only
  flags.go             — globalFlags(rootCertPath) builder (+ resultFlags: --run-id, --output, --webhook, --max-duration)
  schema.go            — `pumba schema` command: pumba.cli/v1 document walked from the live app
  commands.go          — initializeCLICommands wiring chaos cmd builders into urfave/cli commands
pkg/
  chaos/
    command.go         — ChaosCommand interface, Runtime factory type, scheduling/interval runner
    runner.go          — chaos.RunOnContainers / RunOnContainersAll fanout helper (the canonical list-then-random-then-parallel/serial pattern)
    target.go          — per-target handle (*chaos.Target) that turns Applied/Reverted reports into target.applied/target.result events
    events.go          — pumba.event/v1 types, EventSink, status/error-code constants
    outcome.go         — Recorder (counts results before delivery), NewOutcome (run.end status + exit code), coded errors
    runid.go           — run ID generation and validation
    status.go          — status wire types (TargetState on target.result, StatusCoverage on run.end), Verdict, StatusReport + StatusRecorder hook
    cmd/               — Generic NewAction[P] CLI builder shared across all chaos packages
    cliflags/          — urfave/cli v1 adapter (Flags interface + V1, NewV1FromApp for app-level context) decoupling cmd builders from cli.Context; DescribeV1 is the single flag-type switch behind run.start params (`V1.CommandParams`) and `pumba schema` (`DescribeFlagsV1`/`DescribeCommandsV1`)
    lifecycle/         — Runtime-agnostic lifecycle chaos actions (kill, stop, pause, rm, exec, restart); revert.go holds the batch wait-then-revert for pause/stop --restart
    lifecycle/cmd/     — CLI command builders for lifecycle chaos actions
    netem/             — Network emulation (combine, delay, loss, corrupt, duplicate, rate, loss_ge, loss_state)
    netem/cmd/         — CLI command builders for netem
    iptables/          — iptables-based packet filtering
    iptables/cmd/      — CLI command builders for iptables
    stress/            — stress-ng based resource stress
    stress/cmd/        — CLI command builder for stress
    status/            — `pumba status [--check]`: lists Pumba sidecars (label FilterFunc, excludes kind=inspect), inspects each target via InspectNetns through RunOnContainers, coverage (orphans, missing names, listing gaps via ListOpts.OnSkip) and the clean/dirty/unknown verdict
    status/cmd/        — CLI command builder for status (`--run-id` here is a filter, not the invocation ID)
  container/           — Container model, interfaces (Client, Lister, Lifecycle, Executor, Netem, Inspector, etc.), filtering, NetemRequest/IPTablesRequest/StressRequest/StressResult/RemoveOpts/InspectRequest value objects, FaultState, sidecar label names (labels.go)
  runtime/
    docker/            — Docker runtime implementation of container.Client
    containerd/        — Containerd runtime implementation of container.Client
    podman/            — Podman runtime implementation (embeds Docker client, overrides stress cgroup resolution + rootless guards)
    netns/             — Runtime-agnostic glue shared by docker/containerd: iptables request → pkg/tc argv, InspectNetns run+parse
  tc/                  — tc and iptables commands Pumba runs in a target netns (netem scripts, the single iptables argv builder with the pumba:<run-id> comment) and parsers for `tc qdisc show` / `iptables -S`
  util/                — Shared utilities (IP/port parsing, ValidateInterfaceName for network interface name validation)
mocks/                 — Generated mock files (mockery)
tests/                 — Bats integration tests
docker/                — Dockerfiles (main, alpine-nettools, debian-nettools)
deploy/                — K8s/OpenShift deployment manifests
examples/              — Demo scripts

Architecture

Module boundaries are enforced by archfit; see docs/architecture.md for the approved module graph and how to change it.

• Container interfaces (pkg/container/client.go): Focused sub-interfaces (Lister, Lifecycle, Executor, Netem, IPTables, Stressor, Inspector) composed into a unified Client interface. Fat methods take request value objects from pkg/container/requests.go: Netem/IPTables take *NetemRequest/*IPTablesRequest, Stressor.StressContainer takes *StressRequest and returns *StressResult (SidecarID + Output/Errors channels), Lifecycle.RemoveContainer takes RemoveOpts (force/links/volumes/dryRun bools), Inspector.InspectNetns takes *InspectRequest and returns the parsed *FaultState (tagged iptables rules + 504d: qdiscs) of the target's netns. CommandPlan holds exact apply/revert argv and sidecar image or direct-exec mode. Netem.PlanNetemContainer and IPTables.PlanIPTablesContainer(add, del) are pure (no I/O); planning and execution use the SAME pkg/runtime/netns argv builders backed by pkg/tc. SidecarSpec carries the implementation-hint (Image, Pull) pair that runtime adapters may ignore. Netem/IPTables/Stress requests carry RunID (empty = untagged).
• Artifact tagging (pkg/container/labels.go): every sidecar Pumba creates (netem, iptables, stress, status's inspect) gets container.SidecarLabels(runID, targetID, kind): SkipLabel (com.gaiaadm.pumba.skip=true), RunIDLabel, TargetLabel, KindLabel (netem/iptables/stress/inspect). Use the constants; never hard-code a com.gaiaadm.pumba.* string in production code. iptables rules carry -m comment --comment pumba:<run-id> in both the add and the delete spec. qdiscs cannot be tagged; 504d: ownership is classified, never attributed to a run.
• pkg/tc owns netns argv and parsing: the netem scripts, the single iptables argv builder (tc.IPTablesCommands, adds the comment) and the tc qdisc show / iptables -S parsers live in pkg/tc. Docker and containerd reach them through pkg/runtime/netns; never rebuild iptables argv or parse tc/iptables output in a runtime or in pkg/chaos.
• Runtime factory injection (pkg/chaos/command.go): chaos.Runtime func() container.Client is a closure-based factory. Every CLI builder constructor takes runtime chaos.Runtime explicitly — no chaos.DockerClient global, no service locator. cmd/main.go::before constructs the closure once after global flag parsing and propagates it through initializeCLICommands.
• Canonical chaos fanout (pkg/chaos/runner.go): RunOnContainers(ctx, lister, gp, limit, random, parallel, fn) is the single helper every chaos action's Run() calls — it lists matching containers, optionally narrows to a random pick, checks the opt-in gp.MaxTargets cap before any injection, and runs fn either via errgroup (parallel) or sequentially (serial). RunOnContainersAll variant lists stopped + running for lifecycle ops that target stopped containers. New chaos actions MUST use this helper instead of hand-rolling the list-then-fanout loop.
• Interface name validation (pkg/util/util.go::ValidateInterfaceName): single source of truth for the eth0/en0/lo/vlan.10 regex check; both pkg/chaos/netem/parse.go and pkg/chaos/iptables/parse.go call it. Never re-introduce a local regexp.MustCompile for interface names.
• Generic CLI builder (pkg/chaos/cmd/builder.go): NewAction[P] keeps chaos action cmd files in the same shape — flag list + typed ParamParser[P] + CommandFactory[P]. Parsers receive cliflags.Flags, never *cli.Context directly.
• Shared cmd parsing (pkg/chaos/{netem,iptables}/parse.go::ParseRequestBase): per-action cmd parsers (delay.go, loss.go, …) call ParseRequestBase first to read parent-level flags via c.Parent() and build the populated base request (*NetemRequest for netem; iptables returns a small RequestBase carrying *IPTablesRequest + iface/protocol/limit). Per-action parsers then fill only their action-specific fields. New netem/iptables subcommands MUST follow this pattern — never re-parse the parent flag set inline.
• Result events (--output ndjson, schema pumba.event/v1, contract in docs/agents.md): run.start is emitted by NewAction after Build; target.applied / target.result only by RunOnContainers through the *chaos.Target it passes to each ContainerAction; run.end only by cmd/events.go when the app returns. chaos.Recorder counts target.result synchronously before delivery, and chaos.NewOutcome maps counts + returned error to run.end.status and the exit code (0/1 only). Delivering events (stdout, --webhook) happens only in cmd: each sink has its own bounded queue and goroutine, counts its own drops and writes that count into its run.end; chaos code never does I/O for events.
• CLI flags adapter (pkg/chaos/cliflags/): Flags interface wraps urfave/cli v1 via V1. Future v3 migration is a one-file swap (v3.go + wiring change in cmd/main.go).
• CLI schema (pumba schema, pumba.cli/v1, cmd/schema.go): walks the live c.App at run time through cliflags.DescribeFlagsV1/DescribeCommandsV1, so new commands and flags appear without schema edits. A new urfave flag type must be added to the DescribeV1 switch, else schema fails with cliflags.ErrUnsupportedFlagType. before() in cmd/main.go returns before createRuntimeClient when the command is schema (no socket needed). Error codes and statuses come from chaos.ErrorCodes()/RunStatuses()/TargetStatuses(): a new code or status constant must be added there and to the pinned list in pkg/chaos/events_test.go.
• Docker runtime (pkg/runtime/docker/): Docker SDK implementation of container.Client; split per-concern across client.go, http_client.go, inspect.go, lifecycle.go, exec.go, sidecar.go, netem.go, iptables.go, faultstate.go, stress.go, cgroup.go, pull.go (no monolith — every file < 350 LOC)
• Containerd runtime (pkg/runtime/containerd/): Containerd implementation of container.Client (socket: /run/containerd/containerd.sock, namespace: k8s.io); split per-concern across client.go, api.go, container.go, task.go, exec.go, cgroup.go, sidecar.go, netem.go, iptables.go, faultstate.go, stress.go, stress_sidecar.go (every production file ≤ 250 LOC)
• Podman runtime (pkg/runtime/podman/): Podman implementation of container.Client; reuses the Docker SDK against Podman's Docker-compat socket and overrides only what diverges: StressContainer (cgroup resolution + rootless guard) and rootless guards on NetemContainer/StopNetemContainer/IPTablesContainer/StopIPTablesContainer/PlanNetemContainer/PlanIPTablesContainer/InspectNetns. When container.Client grows a method that needs the target netns or a cgroup, add the override and update the list in client.go and doc.go. Socket auto-detected from $CONTAINER_HOST, $PODMAN_SOCK, podman machine inspect, /run/podman/podman.sock, and $XDG_RUNTIME_DIR/podman/podman.sock; override via --podman-socket. Cgroup parent/leaf derived host-side from /proc/<pid>/cgroup of the target container (see pkg/runtime/podman/cgroup.go) — pumba must run on the same kernel as the targets.
• Chaos commands: Each action implements ChaosCommand interface with Run(ctx, random) method
• Network emulation / iptables filtering: Both paths (NetemContainer, IPTablesContainer) execute commands inside an ephemeral sidecar container that joins the target's network namespace. The shared lifecycle — create, start, exec-per-args (with exit-code check), force-remove — lives in runSidecar/runSidecarExec (pkg/runtime/docker/sidecar.go). Neither netem.go nor iptables.go manage sidecar lifecycle directly.
• Stress testing: Two modes — (1) default child-cgroup mode places stress-ng sidecar in target's cgroup via Docker's --cgroup-parent; (2) inject-cgroup mode (--inject-cgroup) uses the cg-inject binary (shipped in ghcr.io/alexei-led/stress-ng) to write sidecar PID into target's cgroup.procs for shared resource accounting
• Target selection: Container names (exact), comma-separated lists, or re2: prefixed regex patterns
• Label filtering: --label key=value flags for container selection
• Interval mode: --interval flag for recurring chaos on a schedule

Code Conventions

• Error wrapping: fmt.Errorf("...: %w", err); do not add github.com/pkg/errors
• Interfaces: Define interfaces for testability (Client in pkg/container/client.go)
• Constructor injection over globals: Chaos cmd builders take runtime chaos.Runtime and produce a closure that reads it lazily. Never reintroduce a chaos.DockerClient-style global — visibility in the function signature is the rule.
• Request value objects for fat methods: Interface methods accept request structs from pkg/container/requests.go, not long positional arg lists. Examples: Netem.NetemContainer(*NetemRequest), IPTables.IPTablesContainer(*IPTablesRequest), Stressor.StressContainer(*StressRequest) (*StressResult, error), Lifecycle.RemoveContainer(*Container, RemoveOpts). New runtime methods that exceed 4 params must follow the same pattern. Pass pointer for ≥3-field requests; pass-by-value for tiny opts (like RemoveOpts's 4 bools).
• Chaos fanout via chaos.RunOnContainers: New chaos actions MUST route through chaos.RunOnContainers (or RunOnContainersAll for stopped-container targeting) — never hand-roll container.ListNContainers + manual sync.WaitGroup/errgroup fanout in a Run() body. The helper enforces list → limit/random narrowing → target-cap check → parallel-or-serial → collect-errors uniformly. Keep and pass the *GlobalParams you received — never copy it (it carries the run ID, event sink, tick counter and MaxTargets; go vet copylocks flags copies) — and wrap errors with %w so ErrMaxTargetsExceeded cannot be suppressed by --skip-error. Action-specific timeouts (e.g. context.WithTimeout(ctx, n.req.Duration)) belong inside the closure, not at the caller.
• ContainerAction contract: func(ctx, c, t *chaos.Target) error. Call t.Applied() right after inject (one-shot actions: after the call succeeded) and t.Reverted(err) with the revert outcome; the runner emits the terminal target.result when the action returns. Batch reverts (pause, stop --restart) call t.DeferResult() and invoke the returned func after that target's own revert (see lifecycle/revert.go). Dry-run must return without waiting for --duration. Netem/iptables dry-runs call only the pure plan method (not inject/revert), attach it with t.SetPlan, and emit planned target.result with no target.applied. A cleanup failure after a successful inject must be returned as an error (exit 1), not only logged.
• Interface name validation: call util.ValidateInterfaceName(name) for any new flag that takes a network interface name. Do not introduce a local regexp.MustCompile for the eth0/en0/vlan.10 pattern.
• Mocking: testify mocks, generated by mockery v3. Mock files live in mocks/, pkg/container/mock_*.go, and test-only pkg/runtime/containerd/mock_*_test.go. Never generate external SDK mocks into production .go files.
• Mock constructor: Always use container.NewMockClient(t) — never new(container.MockClient); auto-asserts expectations on test cleanup
• Mock request structs: EXPECT() calls for NetemContainer/IPTablesContainer/StressContainer/InspectNetns pass the corresponding *NetemRequest/*IPTablesRequest/*StressRequest/*InspectRequest literal (or mock.AnythingOfType("*container.NetemRequest") etc.); RemoveContainer takes a RemoveOpts{...} literal (value, not pointer). Never the old positional form.
• Mock context: Use mock.Anything only for context.Context args; use exact values for all business params
• Mock random container: Use mock.AnythingOfType("*container.Container") + .Once() when only one of N containers is targeted
• Logging: logrus with structured fields. Log level set via --log-level flag
• Constants: Magic numbers use mnd linter — use named constants
• Cleanup defer survives cancellation: Use context.WithoutCancel(ctx) with a timeout so defers (e.g. sidecar cleanup) run even when the caller cancels
• gocyclo violations (limit: 15): Extract loop bodies or complex branches into named helper methods
• funlen violations (limit: 105): Extract initialization blocks (flags, config setup) into named helper functions
• Default branch: master
• NEVER add AI co-author to git commits

Unit Test Coverage Strategy

• Skip from unit tests (covered by integration tests or untestable without real runtime): cmd/main.go and the rest of cmd/*.go except cmd/main_test.go (which exercises the createRuntimeClient seam in cmd/runtime.go), cmd/events_test.go (NDJSON stream, run.end per outcome, stalled-writer drops, usage errors), cmd/plan_test.go (text and NDJSON dry-run plans), cmd/schema_test.go (schema document, no runtime), cmd/webhook_test.go (httptest receiver: body/order, failures as drops, warning without URL, bounded flush) and cmd/slackhook_test.go (webhook payload, thresholds, failure handling, and shutdown drain), mocks/, NewClient/Close, sidecar.go (real container API for create/start/exec/remove), */cmd/ flag builders
• Run() method variants: Always add NoContainers + DryRun + WithRandom test cases
• Run unit tests in sandbox: CGO_ENABLED=0 go test ./... (no CGO needed outside CI)

Gotchas

• PID 1 signal handling: sleep/tail -f /dev/null as container PID 1 ignores SIGTERM — use top in bats tests that kill with SIGTERM
• iptables flag ordering: --source, --destination, --src-port, --dst-port are on the iptables parent command, NOT on the loss subcommand
• exec command parsing: --command "touch /tmp/foo" is wrong (treated as binary name with spaces); use --command "touch" --args "/tmp/foo"
• Containerd sidecar requires root: netem/iptables tests on containerd need sudo pumba — overlayfs mounts for sidecar creation require root in Colima VM
• Containerd namespaces: Docker-managed containers live in moby namespace; pure containerd in default; Kubernetes in k8s.io
• Podman requires rootful for netem/iptables/stress: rootless Podman is detected at NewClient time from Info.SecurityOptions and every netem/iptables/stress call fails fast with a message pointing at podman machine set --rootful (macOS) or the rootful systemd unit (Linux). Rootless support is out of scope — would need slirp4netns/pasta netns handling and user-namespace cgroup math.
• Podman cgroup leaf naming: Podman uses libpod-<id>.scope (or libpod-<id>.scope/container with init sub-cgroup) under systemd, vs Docker's docker-<id>.scope. Pointing --runtime docker at Podman's compat socket silently places stress-ng sidecars in the wrong cgroup; --runtime podman derives the correct path.
• Podman cgroup resolution reads host-side /proc/<pid>/cgroup: containers launched under Podman's default --cgroupns=private see only 0::/ or 0::/container from inside the container, so we read from pumba's own view of /proc (requires shared kernel with targets). On macOS this means running pumba inside the podman machine VM — the same pattern used for containerd testing in Colima. See pkg/runtime/podman/cgroup.go and the cgroupReader var for the test-injectable hook.
• ContainerExecStart empty options breaks on Podman: Docker's ContainerExecStart(ctx, id, ExecStartOptions{}) with no attach/detach flags is accepted by Docker (implicit sync via HTTP hijack) but rejected by Podman's compat API with "must provide at least one stream to attach to". The runExec helper in pkg/runtime/docker/exec.go goes through ContainerExecAttach + drain + inspect — works on both. When writing new exec-driven code, never call ContainerExecStart with empty options; use the helper.
• tc/iptables sidecar cleanup must survive ctx cancel: runSidecar (in pkg/runtime/docker/sidecar.go) calls removeSidecar (wraps ContainerRemove with context.WithoutCancel(ctx) + 15 s timeout). Without this, SIGTERM to pumba during the narrow window after tc exec and before sidecar reap leaks the sidecar AND leaves the netem qdisc on the target's netns.
• Sidecar StopSignal: "SIGKILL": tail -f /dev/null as PID 1 ignores SIGTERM. Podman's DELETE ?force=1 sends SIGTERM and waits the full StopTimeout (10 s) before escalating — that's 10 s per sidecar reap on every netem/iptables call. Config sets StopSignal: "SIGKILL" so force-remove is immediate.
• Podman inject-cgroup needs SYS_ADMIN + label=disable + nested leaf: cg-inject writes the sidecar's PID into the target's cgroup.procs. Three gotchas stack on cgroup v2 + systemd: (1) the target's scope may have a nested container/ leaf (Podman's libpod init sub-cgroup) — cgroup v2's "no internal processes" rule means we must target libpod-<id>.scope/container/cgroup.procs, not the outer scope. pkg/runtime/podman/stress.go::resolveCgroup reads /proc/<pid>/cgroup plus os.Stat to pick between them. (2) Writing across sibling scopes requires CAP_SYS_ADMIN in the initial user namespace. (3) SELinux container_t blocks cgroup writes even with SYS_ADMIN; SecurityOpt: ["label=disable"] on the sidecar bypasses this. All three are required on Fedora CoreOS / RHEL hosts.
• Podman 4.9.x transient /container sub-cgroup races inject-cgroup: Stock Podman 4.9.x on Ubuntu 24.04 creates <scope>/container during libpod init, migrates PID 1 to <scope> shortly after, and rmdirs /container mid-flight. The resolver's os.Stat check can pass and then /container is gone by the time cg-inject opens/writes cgroup.procs, yielding ENOENT on write (documented Podman behavior — see podman#20910). Podman 5.x (podman machine, Fedora CoreOS) keeps /container stable. The inject-cgroup bats test lives in tests/skip_ci/podman_stress_inject_cgroup.bats and is excluded from the GH CI glob tests/podman_*.bats for this reason. Proper fix: retry-on-ENOENT inside cg-inject (stress-ng sibling repo), then move the test back.
• Containerd sidecar and exec IDs are random across processes: pumba-sidecar-<random> / pumba-stress-<random> and their <id>-snapshot keys do not reuse a per-process counter. Crashed runs can still leave artifacts; Bats teardowns must remove leftovers by label (ctr c ls 'labels."com.gaiaadm.pumba.run-id"~=^<prefix>'), then t rm -f, c rm, snapshots rm.
• Kill a backgrounded sudo pumba by its own cmdline: $! is the sudo PID, and kill -9 on it leaves pumba running. Use sudo pkill -9 -f '^pumba .*--run-id <id>' (the ^ anchor keeps pkill off the sudo/shell wrappers whose cmdlines also contain the pattern).
• Ephemeral tc sidecar breaks poll-based bats assertions: the tc sidecar lives only for the duration of tc exec (sub-second) before removeSidecar reaps it. Bats tests that podman ps-polled for the sidecar (to verify the skip label) race the lifecycle. Rewrote tests/podman_sidecar.bats tests 68/69 to assert invariants instead: (a) create a fake container with the skip label manually and confirm pumba's re2: regex doesn't target it; (b) verify netem rules are removed from the TARGET netns after SIGTERM (via nsenter), not whether the sidecar itself was tracked.

Architecture guardrails (archfit)

• Before a cross-module import: archfit policy can-import <file> <target>. denied names the rule and what to do instead; unconstrained is not permission.
• Before you finish: archfit check --format agent. Follow next_action; change.status: pass is a correct finish, even with exit code 2.
• Module owner, layer, allowlists and rules for a path: archfit policy where <path>.
• At session start, or when unsure the hooks run: archfit doctor.
• Never edit .archfit.yaml, .archfit-baseline.json or .archfit-labels.yaml, and never add a waiver, to make a check pass.
• restore_evidence: do what the result names (install the analyzer, merge or rebase the target), then run again. ask_owner, or an unknown you cannot clear: stop, report the rule and what you need, and ask the architecture owner.