AGENTS.md

How to build this firmware and flash it onto your own ESP32 board. See README.md for a shorter human overview.

What this is

The ESP32 Device SDK: the ESP-IDF firmware (muse-gadget) powering Muse Home Link. It pairs with a phone over BLE, joins Wi-Fi, and holds an encrypted Noise session to a VM, with an optional home-network tunnel. main/main.c is the entry point and main/app.c holds most of the logic. components/muse adds a avatar/voice/settings UI on boards with a display.

Prerequisites

• ESP-IDF v6.0.3, installed and activated (. $IDF_PATH/export.sh) so idf.py is on PATH. Other versions are unsupported. Errors about missing IDF headers such as gcm.h or gpio_ll.h usually mean the wrong IDF tag.
• Python 3 and a C/C++ compiler (cc, c++) for the host tests.
• A USB data cable. Flashing needs access to the serial port. A sandboxed agent usually has to run flash and monitor commands outside the sandbox.

Supported boards

Board overlays live in devices/, which has a README.md listing each board's hardware, features, and where to buy it, and an AGENTS.md recipe for adding one. Its Vendor sources table lists each vendor's code for its boards: read it before adding a feature to one.

The default profile expects the C5 DevKitC-1: an addressable status LED on GPIO27, the BOOT button on GPIO28 (active low), 8 MB flash and quad PSRAM. The DevKitC-1's LED takes red first (CONFIG_HOMEHUB_LED_RGB_ORDER, on by default). If yours shows green as red and amber as green, it takes green first like most WS2812s: turn the option off in idf.py menuconfig, or in build-devkit/sdkconfig, and rebuild.

Build

Every application build needs the user's SDK token (mgst_…, from gadgets.muse.ai > Account > SDK tokens). Ask for it, then set CONFIG_GADGET_SDK_TOKEN="mgst_…" in that build directory's sdkconfig (or with idf.py menuconfig) before building. Without it the build warns, and the gadget will stop pairing once Muse requires tokens. Never commit the token or print it in full.

The opt-in CONFIG_MUSE_POWER_TEST image is an isolated board-load characterization harness, not a pairing application, and needs no SDK token or network credentials. Power measurement starts at docs/power/README.md. Read docs/power/ppk2.md before any PPK2 use (authorization and fixture gates, continuous source ownership, USB unplug/replug), and docs/power/sensecap-watcher.md for the Watcher's build and procedure. Never enable source output with unknown wiring, a connected battery, or USB-charger backfeed into PPK2. Other boards reuse the shared muse_power_test_run() core through boards/muse_power_test_board.h hooks and board-owned matrices, driven by tools/power/power_sweep.py; do not fork or rename the core for the next board.

DevKitC-1 (default)

idf.py build                    # -> build/muse-gadget.bin, config in build/sdkconfig

Other boards, with the helper

tools/board.sh BOARD [build|flash|monitor|flash-monitor] [PORT] sets the target and overlays and gives each board its own build-<board>/ directory and build-<board>/sdkconfig:

tools/board.sh sensecap-indicator build
tools/board.sh ideaspark flash-monitor /dev/cu.usbserial-110

Without a port it uses the only matching serial port, and fails if it finds none or more than one. If idf.py is not on PATH, it sources export.sh from $IDF_PATH, ~/esp/esp-idf-v6.0.3, ~/esp/esp-idf-v6 or ~/esp/esp-idf. The Sticky's Muse and Link status profiles build in build-reterminal-sticky/ and build-reterminal-sticky-link/. Set the SDK token in the chosen build directory through idf.py -B <dir> menuconfig.

Home Assistant Voice Preview Edition

tools/board.sh home-assistant-voice build builds a status-and-voice gadget: the LED ring shows the status colours, holding the centre button records a voice note that Muse answers in the app, and the dial sets the speaker volume (shown on the ring, kept across restarts). It advertises as MuseGadget-ha-voice-XXXXXX.

• The console and flashing go through the S3's own USB-Serial-JTAG, which shows up as /dev/cu.usbmodem* like a DevKitC-1. With both plugged in, pass the port, and check it with python -m esptool read-mac before flashing.
• Flashing replaces the vendor firmware on the S3 only. The XMOS audio chip keeps its own firmware; never reflash it.
• While voice chat is ready, the centre button is push-to-talk. Turn the mute switch on to get its setup role back (pairing confirmation, the 5 s factory-reset hold).

Boards with the full UI, by hand

tools/muse/board.sh build|flash <s3|s3n|s3-18|s3-216|s3-206|143c|143|146|aipi|atk-box|box3|c6|c6-206|watcher|sticks3|crowpanel-43|plus2|cardputer-adv|stopwatch|xiao147|cores3|core2|jc3248w535|lcd7|lcd128|lcd349|spotpear|vn183|ai-passport|echoear|p4lcd35|rorolee|lcd19|smartring-plus> [SERIAL|PORT] builds one board in build-muse-<profile>/, logs to /tmp/muse_build_<board>.log, and clears managed_components/ before and after so it doesn't clash with other boards. When flashing, it finds the board's port with tools/muse/ports.py, by the USB device behind it rather than the port name, which changes when boards are re-cabled. With several boards of one kind attached, pass the device's USB serial number (on the chip's own USB serial port, its MAC) or the port; tools/muse/ports.py --list shows them. It finds ESP-IDF the way tools/board.sh does, trying ~/.espressif/esp-idf-v6.0.3 first. For an install anywhere else, set IDF_EXPORT to its export.sh, e.g. in your shell profile: export IDF_EXPORT=/path/to/esp-idf/export.sh. tools/muse/avatar.py builds through board.sh, so it needs the same.

For bench testing, MUSE_BENCH=1 tools/muse/board.sh build|flash ... adds devices/sdkconfig.muse-bench and uses build-muse-<profile>-bench/. That turns on screenshots: tools/muse/snap.py PORT KEYS OUT.png sends bench keys and saves the screen, and >face=thinking (or idle, listening, speaking, error, boot, off, happy) in KEYS picks the avatar mode first. On touch boards, >ui.settings=wifi (or home, muse, ble, sound, sleep, battery, power) opens that settings page for the shot. Screenshots are off in normal builds because each one takes a buffer the size of the screen. >face= works in any build. Or run idf.py directly:

idf.py -B build-muse-aipi -DIDF_TARGET=esp32s3 \
  -DSDKCONFIG=build-muse-aipi/sdkconfig \
  -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;devices/sdkconfig.muse;devices/sdkconfig.muse-aipi" build

To flash, add -p PORT flash with the same -B, target and defaults arguments, except on the SenseCAP Watcher: idf.py flash runs plain esptool, which fails on its USB bridge (see "Flash it"). Build the Watcher with idf.py, flash it with tools/muse/board.sh flash watcher, then idf.py … -p PORT monitor as usual; reading from the bridge works. Muse builds use partitions_muse.csv and need 16 MB of flash or more, except the StickS3, StickC Plus2, Cardputer ADV, AI Passport and Waveshare 1.43C, which have 8 MB and use partitions_muse_8mb.csv.

On the Watcher, rotation opens and pages through the latest reply. A short wheel click sleeps/wakes; a 300 ms hold starts push-to-talk. Pairing still confirms on the initial press. muse_review.c keeps up to 32 KB of the reply in PSRAM across the end of the turn. >review.next, >review.prev, and >review.close exercise navigation over the serial console.

All boards share managed_components/ and dependencies.lock in this directory. If the component manager fails after you switch between a board with the full UI and one without (lvgl is the usual offender), delete both and build again.

OSTB EchoEar-2ST (experimental)

tools/muse/board.sh build echoear uses the measured ESP32-S3 rev 0.2 unit with 16 MB DIO flash and 8 MB octal PSRAM at 80 MHz. Native USB Serial/JTAG (303a:1001) is shared by several boards, so it does not identify the model on its own. EchoEar setup records its pin map and limits. The bottom button controls physical power; GPIO0 is not a talk button. Talking and pairing confirmation use the touch screen. There's no software power-off, so Settings has no Power off option.

The final port was tested on hardware: pairing, touch, recording and playback, sleep and wake, and reconnect. See its setup guide for what wasn't checked.

A new board

Follow devices/AGENTS.md, which covers boards with a status light, a status screen or the full UI from the overlay to verification. In short: copy the closest overlay to devices/sdkconfig.<yourboard>, load it last, and add the board to devices/README.md. At minimum set CONFIG_IDF_TARGET, CONFIG_HOMEHUB_BUTTON_GPIO, the LED backend (CONFIG_HOMEHUB_LED_BACKEND_*, or ..._NONE), and the flash size and mode. Without PSRAM, also set CONFIG_SPIRAM=n, CONFIG_HOMEHUB_TUNNEL=n and the mbedtls internal-allocation options, as devices/sdkconfig.ideaspark does. Signed apps on a classic ESP32 need chip revision 3.0 or later (CONFIG_ESP32_REV_MIN_3=y).

Flash

Identify the board first

When you're asked to flash, work out which board is attached instead of asking which one it is or assuming the default. Say which board you found and what told you, then flash that profile. Getting it wrong writes another board's pin map, flash size and status backend.

1. Ask the running firmware. A board already running this firmware names itself, and this needs no guessing:

   python3 tools/muse/chat.py --status     # {"board": "M5Stack StickS3", ...}

   It also logs the name once at startup, as muse: board: <name>, which tools/muse/monitor.py PORT captures because it resets the board first. components/muse/muse_app.c logs it; each components/muse/boards/board_*.c sets its .name. Boards with the full UI only: the other overlays don't log a name.

2. Read the USB descriptor. This works with no write access to the port and whatever the board is running, including vendor firmware or nothing. ioreg -p IOUSB -l -w 0 on macOS, lsusb -v or udevadm info /dev/ttyACM0 on Linux:

   Descriptor	Board
   Espressif 303a:1001, "USB JTAG/serial debug unit"	the chip's own USB: C3, C5, C6, S3 and the S3 boards with the full UI. Its serial number is the MAC
   CH340 (1a86:7523)	ideaspark, CYD, SenseCAP Indicator, reTerminal E1001 and E1002
   CH340 (1a86:7522)	Elecrow CrowPanel Advance 4.3 V1.3
   CH9102	M5Stack StickC Plus2
   CH343 (1a86:55d3), "USB Single Serial"	Seeed reTerminal Sticky, Waveshare ESP32-S3-Touch-LCD-7, ESP32-S3-Touch-LCD-1.28 or ESP32-P4-WIFI6-Touch-LCD-3.5: chip-id and flash-id tell them apart (the P4 is an esp32p4; 8 MB of PSRAM on the Sticky and the 7-inch, 2 MB on the 1.28-inch; 32 MB of flash on the Sticky, which flash-id shows)
   CH342, two usbmodem ports	SenseCAP Watcher: the S3 console is the one ending in 3, the other is the Himax camera

3. Ask the chip. When you can write to the port (this resets the board):

   python -m esptool -p PORT chip-id      # read-mac for the MAC

   The target narrows it a long way: esp32c5 is the DevKitC-1, esp32c6 the Waveshare C6 or a C6 devkit, esp32c3 the AI Passport or the C3 with a 0.42" OLED (flash-id: 8 MB on the AI Passport, 4 MB on the OLED board), esp32 the ideaspark, the CYD or the StickC Plus2.

4. Fall back to a read-only capture. If the board is mid-run and you can't write to the port, the ## Monitor recipe below reads it without resetting, and a talk press logs the board's own button name as muse_input: waking (<hint>):

   Hint	Board
   top-right, or top-left with live calls (yellow on older firmware)	M5Stack StopWatch
   USR1	Seeed XIAO ESP32-S3 1.47 Touch Display
   power	M5Stack CoreS3
   front	M5Stack StickS3, or StickC Plus2 — tell them apart by the port: the StickS3 is native USB, the Plus2 is a CH9102 usbserial
   top	Waveshare ESP32-S3-Touch-AMOLED-1.75C
   key	Waveshare ESP32-S3-Touch-AMOLED-2.16
   bottom right	AIPI Lite
   wheel	Seeed SenseCAP Watcher
   boot	Waveshare ESP32-C6-Touch-AMOLED-1.8, the ESP32-S3-Touch-AMOLED-1.75, 1.8, 2.06, 1.43C or 1.43, the Elecrow CrowPanel Advance 4.3, the Guition JC3248W535, the Waveshare ESP32-S3-Touch-LCD-3.49, the VN ESP32-S3 1.83-inch NV3023, the Deotaland RoRoLee, the Waveshare ESP32-S3-Touch-LCD-1.9 or the SpotPear ESP32-AI-1.28 v1.0 — tell the C6 by its target (esp32c6), and the S3 boards by the muse: board: line a reset logs

Ask the user only when these come up empty or contradict each other, and say what you found and what's ambiguous rather than asking from scratch.

Flash it

idf.py -p PORT flash                           # default DevKitC-1 build
tools/board.sh devkit flash PORT               # same, in build-devkit/
cd build && python -m esptool --chip esp32c5 -b 460800 \
  --before default-reset --after hard-reset write-flash "@flash_args"

Typical ports: /dev/cu.usbmodem* or /dev/ttyACM* for native USB (C3, C5, S3 and C6 boards), and /dev/cu.usbserial-*, /dev/cu.wchusbserial* or /dev/ttyUSB* for CH340 bridges (ideaspark, CYD, SenseCAP Indicator, reTerminal E1001 and E1002, and CrowPanel Advance 4.3). On Linux, add yourself to the dialout (or uucp) group. If the chip won't enter the bootloader, hold BOOT, tap RESET, release BOOT, and flash again.

The M5Stack StickC Plus2's CH9102 bridge shows up as /dev/cu.usbserial-* or /dev/ttyACM*. It drops out above 230400 baud, so tools/muse/board.sh flashes it at 230400.

The SenseCAP Watcher's bottom USB-C port has a CH342 bridge with two ports, both /dev/cu.usbmodem* on macOS. The ESP32-S3 console is the second (ending in 3); the first is the Himax camera chip. Muse overwrites the Watcher's factory data, so back up its nvsfactory partition first (see devices/README.md).

The CH342 has no flow control and drops bytes when esptool sends a whole packet at once, at any baud and on any USB port. Plain esptool then fails with Failed to write to target RAM (result was 0107: Checksum error) while uploading its stub, or 0105: The format of the received message is invalid on the first flash block, by which point it has erased the bootloader. That leaves the Watcher unable to boot until a flash succeeds; the ROM loader still answers, so it isn't bricked. tools/muse/board.sh flash watcher goes through tools/muse/paced_esptool.py, which takes esptool's arguments and sends 64 bytes at a time at the line rate, at 115200 baud. Use it in place of python -m esptool for anything that writes to the Watcher. A lower baud or another USB port doesn't help. Pacing works with macOS's built-in driver, so WCH's driver isn't needed.

board.sh prints only esptool's last three lines, so a Watcher flash shows nothing for about three minutes. Don't interrupt it, which leaves the board unbootable again. To watch the progress, run the wrapper from the build directory:

cd build-muse-sensecap-watcher
python ../tools/muse/paced_esptool.py --chip esp32s3 -p PORT -b 115200 \
  --before default-reset --after hard-reset write-flash "@flash_args"

• flash writes the bootloader, the partition table (at 0x10000, not the IDF default 0x8000), otadata, phy_init and the app to ota_0. NVS is left alone, so pairing and Wi-Fi credentials survive a reflash.
• idf.py -p PORT erase-flash wipes everything, including pairing. Do this only when you mean to start from scratch.
• The app is signed with the committed dev_signing_key.pem, and Secure Boot is not enabled, so flashing never burns eFuses. Keep it that way: don't enable CONFIG_SECURE_BOOT, flash encryption, or CONFIG_HOMEHUB_PAIRING_EFUSE_AUTH on a board you want to keep reflashing.

Monitor

idf.py -p PORT monitor (Ctrl-] to quit) resets the board and needs an interactive terminal. An agent without a TTY can do a read-only capture that doesn't reset the board:

import os, termios, time
fd = os.open("/dev/cu.usbmodem1101", os.O_RDONLY | os.O_NOCTTY | os.O_NONBLOCK)
attrs = termios.tcgetattr(fd)
attrs[2] &= ~termios.HUPCL   # don't toggle DTR/RTS (which resets) on close
termios.tcsetattr(fd, termios.TCSANOW, attrs)
deadline = time.monotonic() + 10
while time.monotonic() < deadline:
    try:
        chunk = os.read(fd, 4096)
        if chunk:
            os.write(1, chunk)
    except BlockingIOError:
        time.sleep(0.05)

To capture from boot, use tools/muse/monitor.py PORT [secs], which resets the board first (it needs pyserial). A healthy boot logs link.main: Muse Gadget starting. For a board with the full UI, $(tools/muse/ports.py BOARD) gives the port, with BOARD as in tools/muse/board.sh.

Update my avatar

When someone asks to put their own avatar on their board (or to update or change it), run this from esp32/ with the sandbox off (it uses the serial port) and let it finish. It takes several minutes, mostly waiting for Muse:

python3 tools/muse/avatar.py                    # draw their avatar
python3 tools/muse/avatar.py --edit "CHANGE"    # change the one they have

It checks the board, asks their Muse for the renderer through the board, checks it on the host and builds the firmware with it in a copy of the tree, and only then saves it to components/muse/avatar/muse_pixel.c (gitignored) and flashes. It doesn't rebuild their own build-muse-*/: flashing that with tools/muse/board.sh before a board.sh build puts the previous avatar back. Progress and errors go to stderr. Relay the line that starts with Muse drew: and the GIF paths. On failure, pass the message on. The exit status says what kind of failure it was:

• 2: no board, or it can't do this (or another avatar.py is still running: let it finish). If it says the board doesn't answer, rerun with --board s3 or --board aipi (from the board's name, or ask) to flash firmware that can. On the C6 boards or Watcher, follow the manual steps in tools/muse/AVATAR_RECIPE.md.
• 3: the board isn't on Wi-Fi, or it isn't paired in the Muse app and has no device token, so their Muse isn't connected. Tell them, and point them to the Muse app or tools/muse/ble_setup.html.
• 1: Muse's file still fails after two fix rounds, or the build or flash failed. A file that fails a check or the build isn't saved: their avatar is as it was. If only the flash failed, the new avatar is saved (the one before is muse_pixel.c.prev) and its checked build stays in components/muse/avatar/firmware-BOARD/; the error says how to flash it.

Don't commit anything in components/muse/avatar/. To see what the board says, run python3 tools/muse/chat.py --status. To ask their Muse something, run python3 tools/muse/chat.py "question".

The default avatar is in avatar/: its renderer (muse_pixel.c) and its animation (jollybot.gif, and happy_anim.c/.h made from it by tools/gen_happy_anim.py).

Third-party code keeps its upstream license and header: minimp3.h (CC0), main/pixel_font.c (BSD-2-Clause, Adafruit), components/muse/boards/ostb_echoear_display_init.h (MIT, 78/xiaozhi-esp32), components/muse/boards/viewe_smartring_display_init.h (Apache-2.0, VIEWE), main/epd_ssd1677.c (MIT, Seeed-reTerminal-E10xx-Firmware), components/tinyh264/src (Apache-2.0, the Android Open Source Project) and components/libvpx_vp8/src (BSD-3-Clause and PATENTS, the WebM Project). Each third-party file is listed under License in README.md and the root README.md; add a new one to both. Don't restyle them or replace their headers with the Apache one; components/minimp3/README.md says how to update minimp3.

The Apache License doesn't cover the Jollybot avatar in avatar/. Its files carry only a Meta copyright line; don't add the Apache header to them.

First boot and pairing

The status LED (or the edge bars or avatar on display boards) shows the state:

Button (BOOT on the dev boards):

• short press: confirm a pending pairing, or reopen the setup window if setup isn't complete
• hold for 5 s: reset setup (unpair and forget Wi-Fi)

EchoEar-2ST confirms pending pairing with an explicit screen press; hold the on-screen microphone to record and release it to send once paired. It has no physical talk/setup button, so the GPIO button reset gesture does not apply. The VIEWE SmartRing-Plus works the same way; its BOOT button is inside the case.

The device advertises as MuseGadget-XXXXXX (MuseGadget-Disp-XXXXXX on the ideaspark, CYD, SenseCAP Indicator and reTerminal E1001 and E1002 overlays, MuseGadget-ha-voice-XXXXXX on the Voice PE, MuseGadget-respeaker-XXXXXX on the reSpeaker Lite, MuseGadget-Oled-XXXXXX on the ESP32-C3 OLED, which shows the status as words). It uses community pairing v5, so the phone app must support v5 and list community devices. Community pairing needs the explicit device press but has no manufacturer attestation, and it doesn't stop an active man-in-the-middle.

Until it's paired, a board with the full UI shows that name on its screen, dim under the state, so you can tell which gadget to pick in the Muse app. A screen too narrow for the whole name shows the XXXXXX tail on its own, and a square 128 px screen (AIPI Lite) leaves it out, the same as it leaves out the state. Once it's paired (or has a token set by hand), the name goes, and the mic icon and the touch boards' speaker button appear. They're hidden until then, since a press can't reach Muse and there are no replies to mute. The StopWatch is an exception: its dedicated microphone, speaker/mute and power rim icons stay visible before pairing as well, identifying its three physical controls.

To skip BLE Wi-Fi provisioning while you iterate, set CONFIG_HOMEHUB_WIFI_SSID and CONFIG_HOMEHUB_WIFI_PASSWORD in menuconfig. The device still needs to be paired once for its token.

Live voice calls (experimental)

CONFIG_MUSE_REALTIME_VOICE (off by default, "Live voice calls with Muse" under Muse in menuconfig) adds calls beside voice notes on boards with Muse's own voice session: the full-UI boards with PSRAM and the Voice PE (and reSpeaker Lite) builds. On StopWatch, hold yellow to talk or interrupt, release to keep the call open, and tap blue to hang up; voice notes are an explicit Sound setting. With the option on, the StopWatch's yellow key is its talk button and blue its speaker button (board_m5stack_stopwatch.c), swapped from the default build: outside a call blue still mutes, and during one, or once a press that will start one is queued, its press hangs up (muse_input.c). A hang-up that can't be queued moves the input epoch, which ends the call. Other boards retain tap-to-call and hold-for-note. Link-only builds, the DevKitC-1 default included, don't have the option and don't change.

• The call is a POST /voice/realtime over the voice session's Noise connection: components/muse/muse_voice_call_core.c has the protocol (a bounded JSON scanner, the 40 ms appends, the audio decode, the call's lifecycle and deadlines) with no IDF in it, and muse_voice_call.cpp runs it on the session task. tests/test_muse_voice_call.py covers both on the host with synthetic audio.
• The session connects with app_id (CONFIG_MUSE_REALTIME_VOICE_APP_ID, hatch-web by default): the VM takes the app identity from the connection, not from a request header, and refuses calls without one. With the option on, this applies to everything on that connection, voice notes and typed turns included.
• The voice session never sends link.register: Link's own control connection keeps the device's registration.
• Muse's greeting is off by default (CONFIG_MUSE_REALTIME_VOICE_GREETING).
• Every way out of a call sends POST /voice/end-call while the connection is up: closing the media alone doesn't end a call. Server policy may also end a call before the client's own limits. If the connection drops instead, the VM keeps the call through a disconnect grace (about 45 s, then up to 35 s more) before ending it, and in that window another connection of the same account can take the call over; end calls explicitly rather than by dropping. So a call whose end wasn't confirmed (the connection dropped first, or Muse didn't confirm the end-call) is kept as one pending cleanup: its end-call, unchanged, goes again when the session reconnects, before a new call starts (which waits for one try, at most 10 s), up to 3 tries and for at most 10 minutes; a refused (4xx) end-call isn't retried. It names the old call's id, so it can't end a new call, and nothing else of the old call is sent again. A new call's admission replaces the old one at Muse, which also ends the cleanup. The pending cleanup is in RAM only: if the gadget loses power or restarts first, the call stays until Muse's disconnect grace runs out. If Muse doesn't confirm the end-call ({"ok":true,...}), the session drops its connection once the media is down, which makes the VM end the call. The gadget hangs up after CONFIG_MUSE_REALTIME_VOICE_IDLE_S quiet seconds, and always at CONFIG_MUSE_REALTIME_VOICE_MAX_S. Held input, a caller mid-sentence (until that speech item's own stop) and a turn Muse is still working on aren't idle; silent keepalive audio is. Turns are tracked by submission_message_id and input item_id: a turn_settled retires its submission and the earlier ones for that item, never another item's or a later alias.
• On push-to-talk calls only replies to your latest hold play (muse_vc_reply_gate_t). The one exception: once that reply has finished, older work from before your last press reporting back (a delegated task's result, or the outcome after you answer a spoken approval) plays after it, with captions. The audio of a reply you cut off with a press never comes back, and older work arriving while a newer reply is still going is dropped. Approvals work by voice: Muse asks in the call and you hold to answer; the gadget has no approval card.
• On boards with the full LVGL UI (the e-paper UI keeps THINKING), a call shows WORKING (the thinking avatar's working variant) when nothing is playing, the caller isn't holding or talking, and Muse still has a caller turn open (hatch.voice.turn_started without its turn_settled; Muse keeps a delegating turn open until its result has been spoken). A settlement with async_handoffs_pending of 1 or more is a conversational boundary, such as a spoken approval prompt, where Muse waits on the caller: not WORKING (READY on push-to-talk, LISTENING hands-free) until a new caller turn starts, even though such a settlement may name none of the caller's turns. One reporting 0 retires every open turn, since Muse doesn't settle each caller turn by name, except the latest caller turn while its reply hasn't finished (an interrupted reply's settlement can arrive after the next caller turn started), and that only once. A repeat of an earlier settlement changes nothing. Its reply is a response whose response.submission_message_id names a turn for its item, or one that names none (early media the turn adopts, older VMs); older work's responses don't count. Only plain digits count: a string, sign, fraction or exponent, or no count, changes nothing. muse_vc_call_working() decides it; talking, Muse speaking and a pending reply still take precedence. On a hands-free call it waits for the turn's first reply to finish (before that the call shows LISTENING, as it always has), the mic stays open while it shows, and speech the server hears brings back LISTENING. The avatar gets muse_pose_t.working: the default one types on a little keyboard with a spinner beside its head in a warm scheme; an avatar that ignores the field shows THINKING.
• Nothing plays and the mic doesn't open until hatch.voice.session names this call's exact call_id and a session_id; a missing or wrong one fails the call. Before that, an error event fails it too; after, only session_dead does.
• Caller and reply text go to captions only, never the log.
• Hands-free calls shut the mic while Muse is audibly talking, plus 300 ms. The Voice PE keeps it open (full duplex) when its XMOS reports echo-cancelled outputs at start-up.
• The VM must accept calls from the gadget's credentials. A session_control stop on this call's realtime stream ends the call: the client drains queued playback and explicitly ends it. The admission body's second line, MUSE_VC_CAPABILITY_OFFER (hatch.voice.capability.offer with end_call_stop: 1), tells Muse the gadget ends calls that way, so it can hang up when the caller says goodbye; the gadget has no channel for ui.set.voice.end_call. A VM that doesn't support the offer ignores it and sends no stop on goodbye, so a press, the idle limit or the length limit is what reliably ends a call there.

Requests to Muse's VM from your code

CONFIG_HOMEHUB_DAEMON_REQUESTS (on by default, on every board, Link-only builds included) lets firmware open its own HTTP requests to the VM's daemon on Link's Noise session, with noise_ctrl_req_open(), noise_ctrl_req_send() and noise_ctrl_req_cancel() in main/noise_control.h. Muse's own voice notes and chat use the same calls. Each request gets a stream of its own; a refused, reset or cancelled request never touches Link's control stream. tests/test_link_daemon_requests.py runs them on the host with the Muse UI off.

Routes a gadget's credentials can reach, besides Link's own and the voice call's POST /voice/realtime and POST /voice/end-call:

• GET /chat/history: the chat's messages. Ask for one at a time (GET /chat/history?limit=1) so the response stays under the frame bound below. A single message larger than that bound can't be fetched this way.
• POST /fs/read and POST /fs/write: files on the user's VM, with JSON bodies. A path is relative to the user's home on the VM, under the VM's existing path rules; nothing confines it to a workspace folder.
  • read: {"path": "notes/today.md", "offset": 0, "len": 1024}. offset and len pick the part of the file to return. Keep len at 1024 or less: the content comes back escaped (or base64), which can grow it well past its size in the file, and the response has to fit the frame bound below.
  • write: {"path": "notes/today.md", "text": "...", "overwrite": true, "append": false, "create_parent": true}, or data_base64 in place of text (one of the two). overwrite, append and create_parent are required booleans: send all three every time. Write a large file in several requests, the later ones with "append": true.

There are no list or stat routes for gadgets.

Keep within the session's bounds:

• Up to 3 requests at once; another is refused (its callback gets -1).
• Send a body in pieces of at most 8 KB (4 KB on Muse boards without PSRAM and the CYD, 2 KB on the Cardputer ADV and AI Passport): a larger piece ends that request. Queued body bytes are capped at 16 KB with PSRAM and 4 KB without, and noise_ctrl_req_send() waits up to wait_ms for room.
• Each response frame has to fit the session's inbound buffer (12 KB, 17 KB on Muse boards without PSRAM and the CYD). A larger one ends the whole session, which reconnects, and the request with NOISE_CTRL_REQ_TOO_LARGE. Ask for responses that stay under it: read large files in parts.
• The callback runs on the session task: copy what you need and return. Do slow work, and any request that depends on its result, in a task of your own, as on_ws_command()'s asynchronous commands do.

Video from a phone

MUSE_VIDEO=1 tools/muse/board.sh build|flash <board> (overlay devices/sdkconfig.muse-video) adds video streaming (CONFIG_MUSE_VIDEO, CONFIG_MUSE_VIDEO_WIFI): Muse Gadgets Test (../android/musegadgetstest for Android, ../ios/musegadgetstest for iPhone) on a phone on the same Wi-Fi streams its camera, or a clip, as H.264 or VP8 (CONFIG_MUSE_VIDEO_VP8, on by default), and the board decodes it and draws it full screen. ../protocols/VIDEO.md has the design, the Wi-Fi protocol and the numbers.

• Boards: those whose panel driver has draw_frame and that have PSRAM: the Waveshare S3 1.75C (tested) and 1.75, the StopWatch and the ESP-Mosaico. Another QSPI AMOLED driven by boards/muse_lcd_bands.c gets it with .draw_frame = muse_lcd_bands_frame and its panel gap (muse_lcd_bands_set_gap); muse_lcd_bands_set_te if the TE line is wired.
• Pieces: components/tinyh264 and components/libvpx_vp8 (the decoders, behind muse_vdec.c), components/muse_vstream (the receive ring and frame format any transport shares), components/muse/muse_video.c (decode and show tasks, pacing, bands, the view), muse_yuv*.c/.S (colour conversion and scaling), muse_video_wifi.c (TCP on port 7766, mDNS _musevideo._tcp) and muse_video_wire.c (its messages). A new transport fills a muse_video_transport_t and calls muse_video_begin().
• Using it: Phone camera in Settings (touch boards) or the menu opens the view and asks the phone for a stream; any button closes it. The console's >video / >video=0 do the same. The board takes a stream only while the view is open: the port is unauthenticated (../protocols/VIDEO.md, Trust). CONFIG_MUSE_VIDEO_WIFI_START_ANYTIME lifts that for automated benches; never ship it. Stats every 2 s on muse_video: decode time, received-to-shown, kb/s, shown/dropped/skipped, each core's load.
• Radio: while a stream is on, Wi-Fi stays out of modem sleep and BLE stops advertising (muse_wifi_keep_awake, muse_link_ble_quiet): with BLE advertising, the radio's scheduler gives BLE half of every beacon period and the stream's packets wait out that half (50 ms round trips instead of 15, and the phone's bitrate falls to its floor).
• Memory: the ring (128 KB) and the decoders' memory are in PSRAM, and the decode tasks' stacks in main SRAM (MALLOC_CAP_DMA): the S31 also hands out LP RAM as heap once main SRAM runs low, several times slower, so keep anything per-frame out of plain internal malloc. The decode and show tasks take about 20 KB of internal RAM once the first stream starts. CONFIG_MUSE_VIDEO_DECODER_IRAM (off) puts the decoders' hot code in IRAM: 44 KB (and 33 KB for VP8; CONFIG_MUSE_VP8_IRAM picks less or more) that a Wi-Fi + BLE + UI build on the S3 can't spare (LVGL's task then fails to start).
• Test with tests/test_muse_video.py (the ring and the Wi-Fi messages, and the wiring under the options), tests/test_video_untrusted.py (hostile sizes, the port's guards), tests/test_tinyh264_pie.py and tests/test_libvpx_vp8.py; on a board, with Muse Gadgets Test (Android: adb logcat -s MuseCam; iPhone: Xcode's console).
• Untrusted input: everything from the network is. Keep the checks in ../protocols/VIDEO.md's Trust section when changing the transport, the decoders' glue (muse_vdec.c) or the drawing (fill() in muse_video.c).

Configuration gotchas

• sdkconfig.defaults.esp32c5 is loaded after sdkconfig.defaults for C5 builds only (ESP-IDF's target-specific defaults): it moves lwIP and Wi-Fi heap buffers to PSRAM, raises the TCP windows to 64 KiB and deepens the receive path to match (64 dynamic RX buffers, 12 queued out-of-order segments). Other chips keep 16 KiB windows in internal RAM; cmake/validate_config.cmake enforces both.
• Each build's generated sdkconfig lives in its build directory (build/sdkconfig, build-<board>/sdkconfig). Once it exists, it overrides later edits to sdkconfig.defaults or the overlays. After changing those, delete the generated sdkconfig (or the whole build directory) and rebuild. cmake/validate_config.cmake stops the build if a stale config has the old TCP buffer sizes or cJSON nesting limit, and warns if it still has the full root CA bundle.
• HTTPS trusts ESP-IDF's common root CAs only (sdkconfig.defaults), with cross-signed verification so chains that reach one of them through a cross-sign still verify. A server whose chain ends at another root (some display.draw_url image hosts, an OTA URL you host yourself) fails to connect. If you need one, select the full bundle in menuconfig (Component config > mbedTLS > Certificates > Certificate Bundle Configuration).
• Edit per-build settings with idf.py -B <dir> menuconfig (under "ESP32 Device SDK"). Don't commit a generated sdkconfig: it's gitignored for a reason.
• The e-ink UI's task stack (muse_eink) must be in internal RAM: Settings' handlers write NVS synchronously, and flash writes disable the cache. A PSRAM UI task stack asserts in esp_task_stack_is_sane_cache_disabled() when changing a setting. The framebuffer and font caches can remain in PSRAM.
• E-paper boards with the full UI (CONFIG_MUSE_UI_FREEINK, the Sticky so far) draw with FreeInk, not LVGL, which they don't build at all. To bring up another panel: give it a FreeInk board profile (or reuse one), select MUSE_UI_FREEINK and name the profile in MUSE_FREEINK_DEVICE (components/muse/Kconfig), and write a board file that supplies touch_read, buttons, audio and power and points display_pause at muse_eink_display_pause (eink/muse_eink.h); leave display_start and the other display hooks NULL. The UI's fonts are embedded TTFs (eink/fonts/, regenerated by tools/muse/gen_eink_fonts.sh). FreeInk is pinned to a commit in components/muse/idf_component.yml.
• OTA is off by default (CONFIG_HOMEHUB_OTA_ENABLED=n, version 999.0.0), except on boards with the full UI. Set a version with -DPROJECT_VER=1.0.0 or version.txt.
• BLE is shut down and its memory released once setup completes, except on boards with the full UI. CONFIG_HOMEHUB_BLE_PERSISTENT keeps NimBLE up beside Wi-Fi with a basic GATT service (main/ble_basic.c) advertised slowly; it costs roughly 40 KB of internal RAM for as long as the device runs (tools/coexist/sdkconfig.ble-mem wins most of it back on the C5). On the C5, CONFIG_HOMEHUB_BLE_COEX (main/ble_coex.c) keeps BLE advertising and connections from reserving half the radio's airtime once setup is complete, which doubles Wi-Fi throughput with BLE up. It relies on ESP-IDF's private coexistence status bits, so it builds on IDF v6.0.x only until the bits are rechecked on another version. tools/coexist/ holds the overlays used to size it, and the data-path bench (CONFIG_HOMEHUB_PIPELINE_BENCH) with its results.
• With BLE persistent, CONFIG_HOMEHUB_BLE_COMMANDS (default on) lets Muse drive the Link's BLE: ble.state, ble.power, ble.advertise, ble.scan, ble.connect, ble.disconnect, ble.discover, ble.read, ble.write and ble.subscribe (main/ble_commands.c over the central in main/ble_client.c). They run one at a time on their own task. ble.power off stops the stack without releasing the controller's memory, so it can come back; the default build, which releases BLE after setup, can't. tests/test_ble_commands.py runs them against a fake client.
• On the C5 with PSRAM, CONFIG_HOMEHUB_FAST_GCM (default on there) wraps ESP-IDF's one-shot AES-GCM with main/fast_gcm.c for TLS and Noise. It takes about 19 KB of internal RAM (8-bit GHASH tables and the PSRAM staging buffer; see its Kconfig help). Turn it off to compare against IDF's; tests/test_ghash32.py checks the tables and its failure handling.
• Don't move offsets in partitions.csv. prod_data and prod_bak are fixed manufacturing locations, and the table offset of 0x10000 leaves room for a larger Secure Boot bootloader. Check the check_sizes line in the build output: app slots are 2 MB (4 MB on Muse).

Adding a command

Muse calls a gadget's commands by name: the firmware lists them in link.register and answers each link.invoke. A command you add lives in two places, which must use the same name. (device.health and device.ota are handled in noise_control.cpp itself; everything else goes through on_ws_command().)

1. Advertise it in build_register_json() in main/noise_control.cpp: add_command(commands, "relay.set", "<description>", required, optional). required and optional map each parameter's name to {type, description} (string_param() makes a string one). Muse reads the descriptions, so say what the command does and what it returns. If it can take longer than the default 30 seconds, set its timeout_ms, as device.discover does.
2. Handle it in on_ws_command() in main/app.c. Return {"ok": true, "payload": {...}}, or command_error(code, message) for a failure. Only ok, payload (or a payload_json string) and the error's message reach the Muse. Always return a result: NULL reaches the Muse as a generic "command handler did not return a result" error. Validate the parameters yourself: params is NULL when the request has none, and the firmware doesn't check them against the advertised required and optional, so check each one's presence, type, length and allowed values.
3. Don't block. on_ws_command() runs on the Noise session's task, so anything slow (the network, a slow sensor, a camera) belongs in its own task. Copy request_id, session_generation and every parameter the task needs, strings included, into memory the task owns (device.discover uses cJSON_Duplicate()): the request is freed as soon as on_ws_command() returns. If an allocation or the task start fails, free what you allocated and return command_error(). Otherwise start the task and return {"_async": true}. The task then calls noise_ctrl_send_command_result() once, failures included, or the Muse waits out the timeout. It takes ownership of the result and frees it, so don't free or reuse it afterwards. camera.capture and device.discover work this way. A result from an earlier session is dropped.
4. Gate it on a Kconfig option in main/Kconfig.projbuild when it needs particular hardware, and wrap both places in the same #if, as sensors.read does with CONFIG_HOMEHUB_SENSECAP_SENSORS. Add new source files to main/CMakeLists.txt.
5. Keep link.register small. It's printed into at most 8 KB, and a device whose registration doesn't fit never registers.
6. Add a host test in tests/. test_link_sensecap_sensors.py checks that sensors.read is advertised and dispatched under the same option, and runs its parser against a harness.

Report environmental sensors (temperature, humidity, CO2, tVOC, light and so on) through sensors.read, on every board that has them, so Muse has one command to call wherever it runs. Don't add them to device.health, which is for the link itself (battery, power, network), or give them a board-specific command. Return each reading as {value, unit, age_s}, as sensecap_sensors.c does, and report a sensor with no recent reading as null. Read slow or I2C sensors in a background task and have sensors.read return the latest value.

Muse sees the command once the board reconnects with the new firmware. Keep the management commands that on_ws_command() also handles (device.list_vms, device.set_vm, device.reset_vm and device.unpair) out of link.register: tests/test_link_transport_contract.py checks they stay unadvertised.

Say Muse, never Hatch

Users never see the name Hatch.

• Anything a person reads says Muse, the Muse app, or the Muse's name:
  • screen text and error captions (CAN'T REACH MUSE, not CAN'T REACH HATCH)
  • settings labels
  • Kconfig prompts and help
  • log lines
  • tool and script output
  • docs
• Don't use hatch in a new file name or identifier. Use muse or muse_gadget. The Muse chat code is components/muse/muse_chat*.
• The ESP32 account clients use https://api.muse.ai, or the api_url_v2 base the app sends during pairing, with bare API paths. They ignore api_url: only older firmware reads it, and that firmware adds /hatch/ itself. Keep saving it so a device flashed back to older firmware still works.
• hatch stays only where the server or the Muse app depends on it. Don't rename these:
  • the VM host hatch.metaaivm.com
  • the hatch_refresh: auth prefix, the hatch-web app id and the HatchLink/ user agent
  • pairing labels and ids such as hatch-link-pairing-v%d, the hatch_link model and the hatch-link: device id
  • the bug report fields
  • the setup commands (hatch.token and the rest) and the "hatch" key in the status JSON, which the app and older tools use
• Some older identifiers still carry the name (muse_hatch_*, MUSE_HATCH_*, CONFIG_MUSE_HATCH). Leave them unless you're asked to rename them. Don't copy the name into new code.

Tests

These are host-side Python unittest harnesses that compile the firmware C/C++ against fakes, with no board or IDF environment required:

python3 -m unittest discover -s tests -p 'test_*.py'

Run one idf.py build first: test_link_discovery compiles cJSON from managed_components/, and that directory only exists after a build. Set CC or CXX to change compilers.

The e-ink navigation tests use the pinned FreeInk source: set FREEINK_SOURCE_DIR to its checkout, or use managed_components/freeink-sdk from a Sticky configure. CI checks out that exact pin for the host tests.

Two tests skip quietly when their inputs are missing; check the summary for skipped=:

• test_noise_core links against the host's PSA Crypto library, found with pkg-config mbedcrypto. Install libmbedtls-dev and pkg-config on Debian or Ubuntu (as CI does), or mbedtls and pkgconf with Homebrew.
• test_link_pairing_handshake builds Mbed TLS from source for its real-crypto case, so it needs IDF_PATH (set by export.sh) or MBEDTLS_SOURCE_DIR.

Before you hand back work

1. idf.py build (and the board build you touched) passes, with the size check under the slot limit.
2. The host tests pass.
3. If you flashed, the boot log reaches starting, and the log shows no panic or reboot loop.