agentleFS
SensorsIotSensorsIot/Embedded-AI-Harness/.claude/skills/esp-idf-handling/SKILL.md

esp-idf-handling

Skill235 starsChanged 3 months agoMITView on GitHub

Complete ESP-IDF lifecycle: project setup, build, flash, monitor, and OTA. Automatically detects whether a testbench is available or the device is connected locally via USB. Covers sdkconfig, partition tables, esptool, RFC2217 remote flashing, GPIO download mode, OTA updates, crash recovery, and flapping. Triggers on "flash", "build", "upload", "idf.py", "monitor", "serial console", "slot", "testbench", "esptool", "OTA", "erase", "download mode", "crash loop", "flapping", "bricked", "menuconfig", "set-target", "sdkconfig", "partition".

  • Sends data out

The file

Show its rules and text

We split this skill into the rules it gives an agent

15 rules. About 0 are common, 9 are distinctive, and 6 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈9

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 6

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Put persistent config in sdkconfig.defaults (not sdkconfig which is generated):

    line 45
  2. Distinctive:

    Ask the bench — it reads the chip directly (FSD §6.7.3):

    line 124
  3. Distinctive:

    When the device is connected directly via USB (no testbench).

    line 165
  4. Distinctive:

    Stop debug before flashing native USB chips (serial + JTAG share USB).

    line 195
  5. Distinctive:

    When a testbench is available.

    line 205
  6. Distinctive:

    Every field, its default, and the response shape: FSD §6.7.1.

    line 280
  7. Very specific:

    Only viable when your client is on the same LAN as the testbench (sub-millisecond RTT).

    line 304
  8. Very specific:

    not: after /api/flash, which resets the device itself.

    line 329
  9. Distinctive:

    Use OTA when the device already runs firmware with an OTA HTTP endpoint and is on the WiFi network.

    line 340
  10. Distinctive:

    When DTR/RTS reset doesn't work (no auto-download circuit), use GPIO.

    line 388
  11. Distinctive:

    Not all boards have EN/BOOT wired to Pi GPIOs.

    line 429
  12. Very specific:

    When firmware crashes on boot (repeated rst:0xc (RTC_SW_CPU_RST) with backtraces), use the /api/flash endpoint to reflash working firmware.

    line 441
  13. Very specific:

    After portal reaches download_mode, upload and flash on the Pi:

    line 457
  14. Very specific:

    After 2 failed attempts, flash directly on the Pi with esptool --before=usb_reset.

    line 481
  15. Distinctive:

    If a case genuinely needs power removed from the DUT alone, use a per-port switchable hub (uhubctl's supported list) powering only the DUT …

    line 508

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

ESP-IDF Handling

Complete lifecycle for ESP-IDF projects — from project creation to flashing and monitoring. Automatically adapts to local USB or remote testbench.

Step 1: Detect Environment

Determine whether a testbench is available or the device is local.

curl -s $TESTBENCH_URL/api/info
  • Response received → testbench is available, use remote flashing (RFC2217/OTA)
  • Connection refused / timeout → try the discovery script:
    sudo python3 .claude/skills/esp-idf-handling/discover-testbench.py --hosts
  • Still no response → no testbench, use local USB flashing

Step 2: Project Setup

source /opt/esp-idf/export.sh
idf.py create-project <name>           # Create new project
idf.py set-target esp32s3              # Set target chip (esp32, esp32s3, esp32c3, etc.)
idf.py menuconfig                      # Interactive configuration (writes sdkconfig)

sdkconfig.defaults

Put persistent config in sdkconfig.defaults (not sdkconfig which is generated):

CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions-4mb.csv"

Step 3: Build

source /opt/esp-idf/export.sh
idf.py build                           # Build
idf.py fullclean                       # Clean build directory

Step 3b: When there is no local toolchain

export.sh missing is not a broken setup — a project set up by the setup-action skill builds on GitHub deliberately, and some machines never install ESP-IDF at all. Check before assuming:

ls /opt/esp-idf/export.sh ~/esp/esp-idf/export.sh 2>/dev/null || echo "build in CI"

Then the binaries have to come back before Step 4 can run:

# `gh` needs a token, and a tool call is a NON-interactive shell, where
# ~/.bashrc returns before it sources the secrets. Without this line gh says
# "not logged into any GitHub hosts" even though the terminal works fine.
# Never `gh auth login` — see the `github` skill.
. /run/secrets/env 2>/dev/null

gh run download -R <owner>/<repo> -D /tmp/fw            # latest successful run
gh run download -R <owner>/<repo> <run-id> -D /tmp/fw   # one specific run
gh release download v1.2.0 -R <owner>/<repo> -D /tmp/fw # a published release

Without a token, download the artefact zip from the run page in a browser and unpack it — the flash steps below only care that the files exist.

setup-action's template publishes exactly what the two flash paths need:

ArtefactFeeds
coldflash/ — flash_args plus every image it namesStep 4a/4b — a first flash over USB
firmware_v<version>.binStep 4c — the OTA image
sdkconfig.generatedthe effective config, since sdkconfig is not committed

Use the flash_args form in Step 4b when the artefact carries one, and the explicit-offset form only when it does not. Offsets are not constant: enabling OTA moves the app from 0x10000 to 0x20000 and adds ota_data_initial.bin at 0xf000. A remembered offset writes into the wrong partition and boots the previous firmware, which looks exactly like a flash that worked.

Confirm what you are about to flash rather than trusting the filename — the version is compiled into the image:

strings /tmp/fw/coldflash/firmware.bin | grep -m1 '^[0-9]\+\.[0-9]\+\.[0-9]\+'

Flash Size and Partition Tables

Measure the flash; never assume it. There is no universal default — ESP-IDF picks one per target, and it is often smaller than expected (an ESP32-C3 build with nothing set comes out at 2 MB). Whatever it picks is written into the image header, and the bootloader then prints that value back at you, so a boot log saying SPI Flash Size : 2MB is only repeating the config. It is not evidence about the chip, and reading it as such is an easy way to design a partition layout around a number nobody checked.

Getting it wrong is not loud. Configure more than the part has and the partition table extends past the end of flash: the build succeeds, the flash succeeds, and the failure arrives later as corruption at whatever offset first exceeds the physical device — usually OTA or NVS, rarely the app.

Ask the bench — it reads the chip directly (FSD §6.7.3):

curl -X POST $TESTBENCH_URL/api/chip/info \\
  -H 'Content-Type: application/json' -d '{"slot": "SLOT3"}'
# -> "flash_size": "4MB", plus chip, revision, MAC and USB mode

It runs esptool flash_id on the Pi with the proxy stopped, so it reboots the DUT — a deliberate call, not something to poll. Stop any debug session first or it returns 409.

Off the bench, the same thing over a local port:

esptool --port /dev/ttyACM0 flash_id     # "Detected flash size: ..."
/* Or from the firmware, the only way that survives a board swap */
uint32_t size = 0;
esp_flash_get_physical_size(NULL, &size);
ESP_LOGI(TAG, "flash %lu MB", (unsigned long)(size / (1024 * 1024)));

Then set CONFIG_ESPTOOLPY_FLASHSIZE_<n>MB=y in sdkconfig.defaults to the measured value and size the partition table to fit it. Setting CONFIG_ESPTOOLPY_HEADER_FLASHSIZE_UPDATE=y additionally lets esptool correct the header from the detected size at flash time — useful when one image serves boards that differ, but it does not resize the partition table, so it is a safety net rather than a substitute for measuring.

Check OTA fits before writing OTA code. Two app partitions plus NVS, PHY and the bootloader is the constraint that decides whether an OTA design is possible at all, and it is cheapest to discover before the firmware exists. test-firmware/ carries a worked pair — partitions-4mb.csv (1216K app) and partitions.csv for 8MB+ (1536K app).

Step 4a: Flash — Local USB

When the device is connected directly via USB (no testbench).

source /opt/esp-idf/export.sh
idf.py -p /dev/ttyUSB0 flash           # Flash to specific port
idf.py -p /dev/ttyUSB0 monitor         # Open serial monitor
idf.py -p /dev/ttyUSB0 flash monitor   # Flash and monitor

esptool flags by device type

Device--before--after
ESP32 (local USB, ttyUSB)default-resethard-reset
ESP32-C3/S3 (local USB, ttyACM)default-resetno-reset
Any chip via /api/flashfixed by the portalfixed by the portal

A DTR/RTS reset can take a native-USB board off the bus. POST /api/serial/reset on a C3/S3 re-enumerates USB, and the device normally comes back within a second or two. When it does not, the slot goes absent and stays there — seen once immediately after a reset, with USB remove in the activity log and nothing on lsusb afterwards. Prefer /api/serial/monitor for reading a running device, and keep resets for when you actually need a reboot.

/api/flash takes no reset flags. The portal hardcodes --before default_reset --after hard_reset, so the device reboots into the new firmware on its own — do not follow it with POST /api/serial/reset. The --after no-reset advice applies only to driving esptool yourself over RFC2217, where the reset has to be a separate call.

Stop debug before flashing native USB chips (serial + JTAG share USB).

Boot mode (manual)

  1. Hold BOOT button
  2. Press RESET button
  3. Release RESET, then BOOT

Step 4b: Flash — Testbench (RFC2217)

When a testbench is available. Use serial flashing when:

  • Device has no firmware (blank/bricked/first flash)
  • Firmware lacks OTA support
  • You need to erase NVS or flash a bootloader/partition table
  • Device has no WiFi connectivity

Discover devices and slot

Slots are mapped to physical USB hub ports via prefix matching. The portal auto-detects the slot count from the Pi's USB topology at startup (typically 3–4); testbench.json is optional and only needed for custom labels/ports. Slot labels are SLOT1, SLOT2, ..., SLOTn; TCP ports are 4000 + slot_index (e.g. SLOT1 = :4001). Always read slot info from /api/devices to learn the actual layout and verify the device is present.

curl -s $TESTBENCH_URL/api/devices | jq .

Response fields per slot: label, state, url (RFC2217, auto-assigned port), present, running, detected_chip, debugging.

A debug session on any slot can block this one. Native-USB parts share the VID:PID 303a:1001 and OpenOCD claims the interface by it, so a session on a different board fails this slot with A serial exception error occurred: Write timeout while it reads idle and debugging: false. Stop every session, not just this slot's — see testbench-debug.

state: "idle" is not enough — check debugging too. A slot with a live OpenOCD session reads idle while still holding the port, and /api/flash is refused. On native-USB parts serial and JTAG share the one interface, so this is the normal state of a board you have been debugging:

curl -X POST $TESTBENCH_URL/api/debug/stop \
  -H 'Content-Type: application/json' -d '{"slot": "SLOT3"}'

Flash via RFC2217

Bootloader offsets: Classic ESP32 → 0x1000, all newer chips (C3/S3/C6/H2) → 0x0000.

Preferred: POST /api/flash (Pi-side esptool)

The portal stops the proxy, runs esptool directly on the Pi against the local devnode, then restarts the proxy. Use this whenever your client is not on the same LAN as the testbench — RFC2217's SET_CONTROL roundtrip is too slow to keep the auto-reset window open from a high-latency path.

Multipart with ESP-IDF flash_args (recommended — uses what the build already produces):

cd build
curl -s -X POST $TESTBENCH_URL/api/flash \
  -F slot=SLOT1 -F chip=esp32 -F baud=921600 \
  -F flash_args=@flash_args \
  -F bootloader.bin=@bootloader/bootloader.bin \
  -F partition-table.bin=@partition_table/partition-table.bin \
  -F ota_data_initial.bin=@ota_data_initial.bin \
  -F firmware.bin=@firmware.bin \
  | jq .

Each .bin file's multipart part name must equal its basename as referenced by flash_args (e.g. bootloader.bin).

Multipart with explicit offsets — the form to use whenever there is no build tree to take flash_args from: a CI artefact (Step 3b), a released binary, or a one-off single-binary flash. The part name is bin@<offset>; a part named just <offset> is stored as a plain file and never flashed, so the request fails with no binaries to flash:

curl -s -X POST $TESTBENCH_URL/api/flash \
  -F slot=SLOT1 -F chip=esp32c3 \
  -F 'bin@0x0000=@bootloader.bin' \
  -F 'bin@0x8000=@partition-table.bin' \
  -F 'bin@0x10000=@firmware.bin'

Every field, its default, and the response shape: FSD §6.7.1.

Reading flash back: POST /api/flash/read

The counterpart to /api/flash. Pull a region of flash off the device without dropping to OpenOCD — a coredump partition, an NVS blob, the live partition table. JSON body, no upload; same proxy lifecycle as a flash (it stops the proxy and does a reset, so it disturbs the running firmware just like a flash).

curl -s -X POST $TESTBENCH_URL/api/flash/read \
  -H 'Content-Type: application/json' \
  -d '{"slot":"SLOT3","offset":"0x3F0000","length":"0x10000","chip":"esp32c3"}'
# -> {"ok":true,"offset":4128768,"length":65536,"sha256":"...","data_b64":"..."}

offset/length accept an int or a 0x… string. The bytes come back base64-encoded in data_b64 with a sha256 to verify. Stop any debug session first (/api/debug/stop) — a live OpenOCD holds the port and esptool cannot open it. For a coredump, decode the bytes then esp-coredump --chip <c> info_corefile --core <file> --core-format raw <elf>.

Fallback: direct esptool over RFC2217

Only viable when your client is on the same LAN as the testbench (sub-millisecond RTT). Slow paths break the auto-reset timing window because each pyserial SET_CONTROL is a network roundtrip.

SLOT_URL=$(curl -s $TESTBENCH_URL/api/devices | jq -r '.slots[0].url')
esptool --port "$SLOT_URL" --chip esp32c3 \
  --before default-reset --after no-reset \
  write-flash --flash-mode dio --flash-size 4MB \
  0x0000 bootloader.bin 0x8000 partition-table.bin 0x10000 firmware.bin
curl -X POST $TESTBENCH_URL/api/serial/reset \
  -H "Content-Type: application/json" -d '{"slot":"SLOT1"}'

Testbench API endpoints

The full endpoint reference is FSD Appendix D — D.1 discovery, D.2 serial, D.8 firmware repository, D.9 flashing. Read it there rather than from memory: this skill states which call to reach for and what breaks, not the request shapes.

Serial reset

POST /api/serial/reset (D.2) pulses DTR/RTS and returns the boot output. Needed after driving esptool yourself over RFC2217 — not after /api/flash, which resets the device itself.

Slot states

Defined with their transitions in the FSD. Only two accept a flash: idle (via /api/flash or RFC2217) and download_mode (direct serial on the Pi). Anything else means wait or recover first — see Flapping & Automatic Recovery below.

Step 4c: Flash — Testbench OTA

Use OTA when the device already runs firmware with an OTA HTTP endpoint and is on the WiFi network. Faster than serial and doesn't block the serial port.

# 1. Upload firmware to testbench
curl -X POST $TESTBENCH_URL/api/firmware/upload \
  -F "project=my-project" \
  -F "file=@build/firmware.bin"

# 2. Verify upload
curl -s $TESTBENCH_URL/api/firmware/list | jq .

# 3. Trigger OTA on the ESP32 via HTTP relay
OTA_BODY=$(echo -n '{"url":"$TESTBENCH_URL/firmware/my-project/firmware.bin"}' | base64)
curl -X POST $TESTBENCH_URL/api/wifi/http \
  -H 'Content-Type: application/json' \
  -d "{\"method\": \"POST\", \"url\": \"http://192.168.4.2/ota\", \"headers\": {\"Content-Type\": \"application/json\"}, \"body\": \"$OTA_BODY\", \"timeout\": 30}"

# 4. Monitor OTA progress via UDP logs
curl "$TESTBENCH_URL/api/udplog?limit=50"

Firmware repository management

Upload, list and delete are FSD Appendix D.8. The one detail this flow depends on: an uploaded image is served to the device at $TESTBENCH_URL/firmware/<project>/<filename>, and that URL is what step 3 above hands to the ESP32.

Step 5: Monitor

# Local
idf.py -p /dev/ttyUSB0 monitor

# Testbench — via serial monitor API (see testbench-logging skill)
# or via UDP logs
curl "$TESTBENCH_URL/api/udplog?limit=50"

Monitor shortcuts

  • Ctrl+] — Exit monitor
  • Ctrl+T Ctrl+H — Show help
  • Ctrl+T Ctrl+R — Reset target

GPIO Download Mode (Testbench)

When DTR/RTS reset doesn't work (no auto-download circuit), use GPIO.

Pin mapping

Pi GPIOESP32 PinFunction
GPIO17ENRST — pull LOW to reset
GPIO18GPIO0BOOT — hold LOW during reset to enter download mode

Allowed BCM pins: 5, 6, 12, 13, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27

Enter download mode

# 1. Hold BOOT LOW
curl -X POST $TESTBENCH_URL/api/gpio/set \
  -H 'Content-Type: application/json' -d '{"pin": 18, "value": 0}'
sleep 1
# 2. Pull EN LOW (reset)
curl -X POST $TESTBENCH_URL/api/gpio/set \
  -H 'Content-Type: application/json' -d '{"pin": 17, "value": 0}'
sleep 0.2
# 3. Release EN HIGH (ESP32 exits reset, samples BOOT=LOW → download mode)
curl -X POST $TESTBENCH_URL/api/gpio/set \
  -H 'Content-Type: application/json' -d '{"pin": 17, "value": 1}'
sleep 0.5
# 4. Release BOOT HIGH
curl -X POST $TESTBENCH_URL/api/gpio/set \
  -H 'Content-Type: application/json' -d '{"pin": 18, "value": 1}'

Flash after GPIO download mode

sleep 5  # Wait for USB re-enumeration
esptool.py --port "rfc2217://<bench>:<PORT>?ign_set_control" \
  --chip esp32s3 --before=no_reset write_flash @flash_args

GPIO probe — auto-detect board capabilities

Not all boards have EN/BOOT wired to Pi GPIOs. Run once per board:

GPIO probe outputUSB reset outputBoard type
boot:0x23 (DOWNLOAD)—GPIO-controlled — Pi GPIOs wired to EN/BOOT
No output / normal bootrst:0x15USB-controlled — use DTR/RTS
No outputNo outputNo control — check wiring or wrong slot

Dual-USB hub boards have onboard auto-download circuit — GPIO wiring not needed.

Crash-Loop Recovery (Testbench)

When firmware crashes on boot (repeated rst:0xc (RTC_SW_CPU_RST) with backtraces), use the /api/flash endpoint to reflash working firmware. The portal handles proxy lifecycle safely. If the device is completely unresponsive, use GPIO download mode (if available) or reflash directly on the Pi with the portal stopped.

Flapping & Automatic Recovery (Testbench)

Empty or corrupt flash can cause USB connection cycling (flapping state). The portal actively recovers by unbinding USB and re-entering download mode.

With GPIO

State flow: flapping → recovering → download_mode → (flash firmware) → idle

After portal reaches download_mode, upload and flash on the Pi:

scp build/bootloader/bootloader.bin build/partition_table/partition-table.bin \
    build/ota_data_initial.bin build/*.bin pi@<bench-hostname>:/tmp/
ssh pi@<bench-hostname> "python3 -m esptool --chip esp32s3 --port /dev/ttyACM1 \
  write_flash --flash_mode dio --flash_size 4MB \
  0x0 /tmp/bootloader.bin 0x8000 /tmp/partition-table.bin \
  0xf000 /tmp/ota_data_initial.bin 0x20000 /tmp/firmware.bin"

Then release GPIO:

curl -X POST $TESTBENCH_URL/api/serial/release \
  -H 'Content-Type: application/json' -d '{"slot": "SLOT1"}'

Without GPIO

State flow: flapping → recovering → idle (if stable) or flapping (retry, up to 2x)

After 2 failed attempts, flash directly on the Pi with esptool --before=usb_reset.

Manual recovery trigger

curl -X POST $TESTBENCH_URL/api/serial/recover \
  -H 'Content-Type: application/json' -d '{"slot": "SLOT1"}'

Power-cycling a DUT (don't automate it with uhubctl)

A test that needs a true power cycle — a POWERON reset reason, or clearing the ESP32's RTC SRAM to exercise a cold-boot guard — is tempting to automate by cutting USB VBUS with uhubctl. On a typical bench Pi this backfires two ways, both observed:

  • It can take the whole bench offline. The Pi's one switchable root port (uhubctl shows ppps on only that port) usually feeds a ganged hub that carries the DUT slots and the Pi's own USB-Ethernet uplink. Cutting it drops the Pi's network, and the port frequently does not restore on its own — the bench needs a physical power-cycle to come back. Never run uhubctl against the root port that carries the Pi's uplink.
  • A brief VBUS cut may not be a real power cycle anyway. The ESP32-C3's RTC domain holds its charge across a short outage, so reset_reason comes back USB, not POWERON, and RTC-SRAM guards still see stale state. Even a Pi power-cycle left the C3's RTC SRAM intact in one run.

If a case genuinely needs power removed from the DUT alone, use a per-port switchable hub (uhubctl's supported list) powering only the DUT slots, a smart-plug/relay on the DUT rail, or an operator replug — never the shared root port. Treat "true power cycle" as an operator-assisted tier, not an automated one, unless the bench has isolated DUT power.

Troubleshooting

Local USB

IssueSolution
Failed to connectEnter boot mode (BOOT+RESET sequence)
No serial port foundCheck USB cable, ls /dev/ttyUSB* or ls /dev/ttyACM*
Permission deniedsudo usermod -aG dialout $USER, re-login

Testbench

ProblemFix
Slot shows absentCheck USB cable, re-seat device
Slot went absent and stays thereCheck lsusb on the Pi. If no 303a: device appears, no API call can help — /api/serial/recover and GPIO download mode both drive a board that is still enumerated. A device off the bus needs hands: re-seat the cable or power-cycle the board
flapping stateRecovery should start automatically; if stuck, POST /api/serial/recover
recovering stateUSB unbound, recovery in progress — wait for download_mode or idle
download_mode stateFlash firmware on the Pi, then POST /api/serial/release
esptool can't connectUse POST /api/flash — never call esptool over RFC2217 directly
Device crash-loopingReflash via POST /api/flash with working firmware
Board occupies two slotsOnboard USB hub — identify JTAG vs UART via udevadm info

OTA

ProblemFix
Upload failsUse -F flags (multipart), not -d
ESP32 can't download firmwareDevice must reach testbench; check WiFi
OTA trigger times outCheck device's OTA endpoint URL; increase timeout

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.