hil

---
name: hil
description: Use when running TinyUSB Hardware-in-the-Loop (HIL) tests on physical boards, when a HIL run fails, hangs, reports a board locked, or produces a report you need to interpret, or when copying firmware to a test rig (ci.lan, hifiphile/tusb, or a dev PC). For board/probe health scans ("pool check") use the hil-pool-check skill instead: it assesses the pool, this skill runs and interprets tests.
---

# Hardware-in-the-Loop (HIL) Testing

Run TinyUSB HIL tests on real boards. **Run `hostname` first** — it tells you which host you are on, which determines the default config and whether remote mode is possible. Rule of thumb: only `ci` and `tusb` are infra rigs; **any other hostname is a dev PC** and uses `local.json`.

| Host                              | Local config                         | Remote (SSH → ci.lan)?                                 |
|-----------------------------------|--------------------------------------|--------------------------------------------------------|
| `ci` (the rig)                    | `test/hil/tinyusb.json` (large pool) | no — boards are already local                          |
| `tusb` (hifiphile's external rig) | `test/hil/hfp.json`                  | no outbound SSH to dev PCs/ci; SSH-reachable FROM both |
| anything else (a dev PC)          | `test/hil/local.json`                | yes (large pool, `test/hil/tinyusb.json`)              |

Default to **local**. Use **remote** only when on a dev PC and the request or task scope names `remote`/`ci.lan`. Never attempt remote on `ci`.

`tusb` (ssh alias `hifiphile`) is an external rig (hosted by maintainer hifiphile), exercised by the
GitHub CI `hil-tinyusb (hfp.json)` matrix job — **never run HIL against it unless the user explicitly asks.**

## Board locks — the CI runner keeps running

The `ci` rig also hosts a GitHub Actions runner that flashes boards and runs HIL as part of CI. Hardware access is arbitrated **per board** with kernel flocks in `/tmp/tinyusb-hil-locks/` — do NOT stop the runner service.

- `hil_test.py` self-locks each board for its flash+test (holder reason `hil_test.py`). A locked board fails immediately (`<board>  Failed: board locked: {holder info}`) without flashing — in CI, re-run the failed job later; if your `hold` is refused with reason `hil_test.py`, a CI job is mid-test — wait a few minutes and retry rather than forcing.
- For usbtest bring-up or case diagnosis, load the `usbtest` skill before a manual case run; its `run_case.py` takes and releases the lock itself. This skill still owns board locks and rig selection.
- For hardware work outside `hil_test.py` (JLink/GDB, manual flashing, `usbtest.py`, serial poking), hold the lock first and release it when done — release is mandatory cleanup; the auto-release on holder death is a backstop, not the plan:

```bash
python3 test/hil/helper/hil_lock.py hold BOARD [BOARD...] --reason "why"
# ... hardware work ...
python3 test/hil/helper/hil_lock.py release BOARD [BOARD...]
```

- A hold refused by another holder: report holder and reason (`hil_lock.py status`), never kill the holder. A holder reason of `hil_test.py` is a CI job mid-test — wait a few minutes and retry once when the task allows, otherwise return the holder to the caller.

- A manual session on a dev-bench board with no entry in this host's HIL config locks it by an agreed board name, with no config: `hil_lock.py hold BOARD --reason "..."` only reserves that name, so verify the probe serial and board identity yourself. A named-board lock needs no config; config-driven tools (`hil_test.py`, `hil_pool_check.py` without an explicit config, `hil_lock.py hold --all`) still need this host's config.
- Never pre-hold boards you are about to run `hil_test.py` on — it self-locks and would treat your own hold as a conflict.
- Rig-wide operations (uhubctl power cycling, `usb_recover.sh root-cycle`, pci-rebind, controller resets — bus renumbering) affect every board: `hil_lock.py hold --all --config <this host's config> --reason "..."` first — `--all` defaults to `tinyusb.json`, so on `tusb` it would reserve 27 boards that do not exist there and none of the three that do. Even a single root-port bounce needs `--all`: nothing maps a sysfs busport to a board name, and `hil_lock.py hold` accepts any string, so a "just the siblings" hold reserves nothing while reporting success. If `--all` cannot be taken, wait: a partial hold is worse than none, because it reads as protection.
- `hil_lock.py status` lists holders. Locks auto-release when the holder process dies (kernel flock); `/tmp` clears on reboot.
- A board whose usbtest battery reported the device wedged (a HUNG case the in-run recovery could not clear, or a device it could no longer identify afterwards) gets a `board-wedged` cell and the rest of its examples are skipped for that run. No admission marker blocks the next run. Before a board's first flash, a run without `--skip-flash` looks for a D-state `testusb` on that board's usbfs node, which a cancelled or killed run leaves behind, and resets the board through its convoy-safe recovery flasher (`flasher_recover`, else `flasher`); a board it cannot free is reported `board-wedged` without being flashed. A stray on a board outside the run is not swept. Interactively, follow `usb-kernel-recover` from its triage before running it again; a delegated run follows Reporting.
- Forcing past a lock: `HIL_NO_BOARD_LOCK=1 python3 test/hil/hil_test.py ...` bypasses the guard without killing the holder. Only when the request or task scope explicitly names forcing that board — it risks colliding with whatever holds it; a refused hold alone never adds that scope.

## Pool check (board/probe health)

Board/probe health scanning (`test/hil/helper/hil_pool_check.py`) has its own skill: **hil-pool-check**.
Use it before a HIL campaign, after rig maintenance/reboot, or when boards fail to flash.

## PR-scoped selection

`tools/change_impact.py` maps a diff to affected boards/tests (used by CI on PRs; fail-open
to the full matrix). Manual use:

```bash
SEL=$(python3 tools/change_impact.py --base master test/hil/tinyusb.json)
FULL=$(printf '%s' "$SEL" | python3 -c "import json,sys; print(json.load(sys.stdin)['full'])")
ARGS=$(printf '%s' "$SEL" | python3 -c "import json,sys; print(json.load(sys.stdin)['args']['tinyusb.json'])")
if [ "$FULL" = "True" ] || [ -n "$ARGS" ]; then
  python3 test/hil/hil_test.py $ARGS test/hil/tinyusb.json   # $ARGS empty when full: run everything
else
  echo "diff affects nothing on this rig - skip HIL"
fi
```

Read `full`, never `args` alone: `args` is empty for BOTH `full: true` (run the whole matrix — a broad or
unclassified change) and "nothing selected" (skip). Skip only when `full` is false AND `args` is empty.

Unit suites (no hardware) live in `test/hil/test/test_*.py`; the `hil-test` pre-commit hook
runs every `test_hil*.py`, `change-impact-test` `test_change_impact.py`, `test_ci_metrics.py` and
`test_hil_util.BottomLayer`. `test_change_impact.py` covers only selection, `test_ci_metrics.py`
only the code-size plumbing; the bounded reads and the build and pool guards live in
`test_hil_bounded.py` and `test_hil_util.py`; `test_hil_report.py` covers the report document, `test_hil_rtt.py`
the RTT console and `test_hil_pool_check.py` the pool check's verdicts and output. Run them all when changing `test/hil`:
`for f in test/hil/test/test_*.py; do python3 "$f"; done` (about a minute, half of it
`test_hil_bounded.py`'s deliberate hang/timeout simulation).

## Stuck runs

What bounds a stuck run is `HIL_POOL_TIMEOUT` plus the job's `timeout-minutes`: a worker
the kernel will not let die (D state) holds the job until that ceiling. The `hil-pool-check`
skill finds which boards a wedge left unusable.

See the `usb-kernel-recover` skill for what a real wedge looks like and how to clear it, and the `usb-kernel-debug` skill to explain WHY the kernel rejected a device (dmesg analysis).

## Prerequisites

Examples must be built for the target board(s) — see [Build and Validate](../../../CLAUDE.md#build-and-validate). Build them with the `build` skill's `--shared --variants <config>`, naming the HIL config the run uses: it writes the `cmake-build/cmake-build-<variant>/` folders `hil_test.py` flashes from by default, each with its variant's `flags` and `defines`, and refuses a folder still configured with options neither the variant nor the command line supplies. A local `hil_test.py --build` runs that same build for the selected boards first. A **remote** run stages the same folders; see Remote execution below. (This applies to `hil_test.py`; `hil_pool_check.py` never builds: it flashes CI artifacts from its own cache, see **hil-pool-check**.)

A bare `--shared` takes nothing from the roster and builds only `cmake-build-<board>/`, so a self-named variant is built without its flags (`raspberry_pi_pico`'s `CFG_TUH_RPI_PIO_USB=1`) and any other variant stays unbuilt. An unbuilt variant's tests report `Skip (no binary)` without failing the run (`stm32f723disco-DMA` goes untested). On a run of one test per variant (a single-test `-bt`, or `-t` tests the board's `only` or capabilities leave at one) without `--skip-flash`, a later unbuilt variant instead fails `same-PID boundary ... not cleared (no board_test binary)`, naming `board_test`, not the missing build.

A board whose flasher probe has no VCOM (or whose BSP has no UART) uses RTT as its console — "No serial device found for /dev/serial/by-id/…" on every host test is the symptom. Config: `"logger": "rtt"` (jlink flashers only) plus a self-named variant carrying the define — `"variant": [{"name": "<board>", "defines": ["LOGGER=rtt"]}]` — and prebuilt example sets must carry the same `-DLOGGER=rtt`. Caveat: the cdc/msc-fixture host tests don't speak RTT yet, so such a board cannot carry `is_cdc`/`is_msc` fixtures (the config loader rejects it). Details: the `rtt` skill.

## Arguments

- **Board:** `-b BOARD_NAME`, repeatable for a subset (`-b a -b b`); omit to run all boards in the config. Give a whole set to ONE run rather than one run per board: it spreads the boards across host controllers in dispatch order, while a second `hil_test.py` running alongside multiplies the load on the same xHCI cards. (A marginal DUT port bouncing under concurrent batteries has killed a uPD720201: fix the port or pull the board.)
- **Pass-through:** `-v`, `-r N`, etc. forwarded unchanged.

If `local.json` is missing on a dev PC, ask the user to supply one before a `hil_test.py` or `hil_pool_check.py` run. An agent that cannot ask (`hil-operator`) does not run `hil_test.py`: it returns one `ran: false` row per requested board whose `detail` names the missing `test/hil/local.json`. Fall back to `tinyusb.json` only when the request or task scope says so; a manual session locks by board name as Board locks says.

## Local execution

Set `CONFIG` from `hostname` first (`test/hil/local.json` on a dev PC, `test/hil/tinyusb.json` on ci, `test/hil/hfp.json` on tusb):

```bash
CONFIG=test/hil/local.json      # on ci use: CONFIG=test/hil/tinyusb.json

# All boards in the config:
python3 test/hil/hil_test.py "$CONFIG"

# A single board (replace stm32f723disco):
python3 test/hil/hil_test.py -b stm32f723disco "$CONFIG"
```

## Remote execution (dev PC → ci.lan only)

`scripts/hil_remote.py` takes `hil_test.py`'s own arguments, minus the config. It stages the harness, the config and the firmware the run will read under `-B` (default `cmake-build`, the `build` skill's `--shared` layout), runs `hil_test.py` on `ci.lan` with `tinyusb.json`, and copies the report pair and `<config>.failed` back to the checkout root:

```bash
# All boards built under cmake-build/:
python3 .claude/skills/hil/scripts/hil_remote.py
# A subset — repeat -b, ONE invocation for the whole set:
python3 .claude/skills/hil/scripts/hil_remote.py -b raspberry_pi_pico2 -b stm32f723disco -t host/cdc_msc_hid -r 1
```

An agent launches it with `--run-id` and waits on it as Timing says.

One invocation per board is wrong here, not merely slow: each run `rm -rf`s `REMOTE_DIR`
and rewrites the report, so only the last board's rows survive. A second run sharing
`REMOTE_DIR` is refused, before the wipe, while the first holds `<REMOTE_DIR>.lock`. The rig
needs `flock`; setup fails without it.

Before touching the rig it refuses a board not in the config, then applies `--flasher` and
`--exclude-flasher` (`no board left after the flasher filter` when none survives), then refuses a
requested board the filter kept with none of its `<-B>/cmake-build-<variant>` dirs, naming the dirs
it looked for (a variant's build flags are in the config); a board the filter drops needs no build.
Without `-b` it refuses with `nothing to test` when no board the filter kept is built, and stages
what is built without a word about unbuilt variants. With `-b` it also warns for each variant of a
requested board with no build; Prerequisites says what `hil_test.py` then reports for it. `--build` is
refused: the rig receives binaries only.

A delegated run is bound to its build: the build writes a build receipt of the HEAD it built,
and the run passes it, refusing before it touches the rig when HEAD, the roster or a staged file
no longer match it, or a tracked file (the harness it stages, the roster) has changed since HEAD
(rebuild, which writes a new one). A retry of a subset of those boards reuses
it; a local `hil_test.py` run takes none.

```bash
python3 .claude/skills/build/scripts/check_build.py --board raspberry_pi_pico2 --board stm32f723disco --shared --variants test/hil/tinyusb.json --receipt .hil-remote/build-1790000000.json
python3 .claude/skills/hil/scripts/hil_remote.py --run-id hil-1790000000 --receipt .hil-remote/build-1790000000.json -b raspberry_pi_pico2 -b stm32f723disco
```

The build refuses a receipt for a tree that is not clean before or after it (commit a
`hw/bsp/family.json` it rewrote, then build again). `.hil-remote/` is ignored.

A CI firmware artifact is the one delegated run without a receipt: download it straight into a
fresh `-B` dir under `cmake-build/`, never a symlink to it (the staging rsync would copy the link).
Run it from a checkout whose harness and roster match the artifact's head, since those are staged
from the checkout. Report the artifact's name, run, head and staged sha256s beside the results.

```bash
mkdir -p cmake-build && mkdir cmake-build/ci-36166669952
gh run download 36166669952 -n 'binaries-arm-gcc--b raspberry_pi_pico_w' -D cmake-build/ci-36166669952
python3 .claude/skills/hil/scripts/hil_remote.py --run-id hil-1790000000 -B cmake-build/ci-36166669952 -b raspberry_pi_pico_w
```

Exit 200 means the remote tree stopped being this run's after staging (another run sharing
`REMOTE_DIR` replaced it): `hil_test.py` did not run and nothing was copied back, so any local
`hil_report` pair or `<config>.failed` is from an earlier run. Re-run; never report from it.

Env overrides: `REMOTE`, `REMOTE_DIR`, `CONFIG`, `ROOT_DIR`.

## Timing

Runs take 2-5 min per board, but a stuck fleet runs to `HIL_POOL_TIMEOUT` — 60 min
unless the env pins it. The run logs its guard in the startup line; never declare a run
stuck before THAT value has elapsed.
The Bash tool caps a foreground timeout at 10 min, so **run it in the background** --
never a foreground timeout, which would kill the run before its own guard can write a
report. NEVER cancel early.

A remote run is launched once, in the background, with a new `--run-id` (a used one is
refused), then waited on in the foreground; write the id literally in both calls, since a shell
variable does not survive from one tool call to the next:

```bash
# Bash run_in_background: true; a delegated run adds its --receipt (Remote execution)
python3 .claude/skills/hil/scripts/hil_remote.py --run-id hil-1790000000 --receipt .hil-remote/build-1790000000.json -b raspberry_pi_pico2 -b stm32f723disco
# foreground, Bash timeout 600000
python3 .claude/skills/hil/scripts/hil_remote.py wait hil-1790000000
```

`wait` blocks up to 9.5 min and prints one JSON line:

- `done` (exit 0): the run's `exit` and the `reports` it copied back. Read only those; an
  empty list means no report is from this run, whatever the checkout holds.
- `running` (exit 3): call `wait` again.
- `dead` (exit 4): the run ended without its done record (killed, or its session died); no
  local report is from it.

Never spend a tool call only to check on a run or to pass time: the next call is `wait`.
A local `hil_test.py` run has no done record; wait for its completion notification.

## Reporting

Two audiences, two shapes. Interactively, the answer to a HIL run IS the tool's summary table:
paste the complete per-board table (and footer counts) verbatim — never truncate rows or reduce
it to a prose digest. Commentary below it covers only what the table cannot show: a notice
from the list below, a retry, a wedged board.

A delegated run (the `hil-operator` role) returns the machine output instead: exactly
`{ pass, results, caveat, wedged }` and nothing else. From the directory the run wrote its
report to:

```bash
python3 test/hil/helper/hil_report.py <config> -b BOARD [-b BOARD...]
```

`pass`, `results` and `caveat` are copied from its output verbatim — never retyped,
reworded or re-ordered: rows are named per variant, a variant name need not start with the board
name, and lock contention is a cell rather than a phrase, so any of it re-derived by hand has come
out wrong before. `results` has exactly one entry per requested board. `pass` is the verdict of
this report snapshot: every row can pass on an aborted or no-boards run, so the run-level
`caveat` (aborted, no boards; empty on every other run) gates it. `--accumulate`
clears an earlier attempt's caveat by design, so the verdict of a retry sequence is the caller's:
keep every attempt's result, a clean subset re-run never erases an earlier run-level failure, and
a re-run's own caveat fails the sequence. Each row's `wedged`
is the report's own verdict (a `board-wedged` cell: a confirmed hang, or one the battery could
not rule out) and is copied with the row; the
top-level `wedged` — the boards the run left unresponsive, usually none — is the operator's
own observation and the only field it authors when a run happened; it names requested boards,
never a variant row name.
A run refused with `board(s) not in <config>` is re-run without the unknown names only while
a known name remains — an empty `-b` list runs every configured board — keeping the full
requested list on the `hil_report.py` call, which emits a `ran: false` row for each unknown
board. When no run started (a missing config, every name unknown, a refused hold with no
permitted retry, a scope gap, unbuilt firmware), it authors the rows instead: one per requested
board, `ran: false`, `pass: false`, `locked` as observed, the reason in `detail`, top-level
`pass: false`, `caveat` empty — and reads no stale report. The caller treats a
missing or malformed reply as inconclusive, never as a pass or a fail.

The caller decides what follows a run. The whole board set goes to ONE run (above); the failure
retry below runs only on a report with no `locked` or `wedged` board and an empty `caveat`, so
any other outcome returns to the caller as it is. On `locked` the caller bypasses with `HIL_NO_BOARD_LOCK=1` only
when the task scope names bypassing those boards' locks, never releasing or killing the holder;
or waits and re-runs the locked boards with `--accumulate`; or accepts, reporting the boards not
covered. A `wedged` board is never re-run. A delegated run recovers it only through usbtest's in-run
recovery (a probe reset, or a reflash where the flasher has no reset); a board still wedged
after that is reported, and any further recovery is a separately dispatched recovery action
(below).

A dispatched recovery action is its own operation, never part of a run: the prompt names one
board, its busport, the rung ceiling, the budget and the reservation. Follow `usb-kernel-recover`
from its triage up to that ceiling and never above it, holding `--all` for every rig-wide rung.
Rung 1 resets through the board's recovery flasher (`flasher_recover`, else `flasher`); one that
is not `convoy_safe`, JLinkExe included, runs only behind `usb_recover.sh shield` on the board's
busport, unshielded afterwards;
sysrq and the hypervisor rungs need the user and are returned as the blocker in a headless run.
Report the cleanup explicitly: the board's state, plus any shield record or hold still standing.

**First check what sits above the table.** Three notices can appear there; match on a
PREFIX, since each carries trailing detail:

- `**HIL run aborted: worker pool timed out after …s.**` and
  `**HIL run aborted: a worker raised …**` — the pool guard fired, or a worker crashed. The
  notice counts what happened: "N board(s) below finished and are this run's; K never
  reported and are NOT in the table: <names>. The re-run spec covers those", plus, when
  `check_build.py` refused a board under `--build`, ", and the M whose build check_build.py
  refused: <names>." The N finished boards' rows are this run's: report them. The K named
  boards are not this run's whatever the table shows — on a fresh run they have no row, on
  an `--accumulate` retry a previous attempt's row survives under the notice and
  `hil_report.py` still folds it into `results` as ran — so name them as not run. The M
  refused boards DO have a `build refused` row, reported ran=false/pass=false (not run,
  failed), but sit outside N; the `<config>.failed` re-run spec covers both K and M. Never
  `"pass": true`.
- `**HIL run selected no boards.**` — the filters intersected to nothing, or, under
  `--build`, `check_build.py` refused every board. On the filter case, a fresh run shows no
  table; an `--accumulate` run keeps the previous attempt's rows under the notice, and they
  are not this run's. On the all-refused-build case, a fresh run AND an `--accumulate` run
  alike instead write every board's `build refused` row as this run's — report them
  ran=false/pass=false; `--accumulate` also keeps an earlier LOCKED/WEDGED cell for that
  board, and the `<config>.failed` re-run spec is written or refreshed to name the boards.
  Either way the exit code is nonzero. Report what the message shows (the filter, or the
  refused boards), never `"pass": true`.

On a test failure, retry once with `-v` — only when the report has no `locked` or `wedged` board
and an empty `caveat`; otherwise return the snapshot without retrying, since the `.failed` spec
lists those boards too and an accumulated retry would clear a caveat the caller must keep. Retry
from the `<config>.failed` spec the run just wrote, which
already begins with `--accumulate` and restricts each board to its failed tests. A hand-scoped
`-b <board>` retry MUST pass `--accumulate` too: a fresh run unlinks the report, replacing the
whole-fleet table with a one-row table. A usbtest battery that produced per-case verdicts is not
auto-retried; its result already stands, and its FAIL, NOTRUN or HUNG cases go to the `usbtest`
skill for diagnosis. If a board or fixture stops enumerating, or a tool of
yours hangs in D state, that is a wedge: save `dmesg | tail -50` as an artifact beside the
report (never into the report's rows) and name the board in `wedged`, without recovering it.
If a retry is still not enough, an interactive
session may add temporary debug prints to `hil_test.py`; a non-editing operator returns the
failure for diagnosis instead.

More agent context in hathach/tinyusb

10 other files this repository gives its agents.