RDK Embodied AI: LeRobot ACT & Pi0 VLA Deployment

Take a trained robot-control policy and run it on an RDK board's BPU: a LeRobot ACT imitation policy, or a Pi0 / openpi VLA model. This skill owns the software deployment half only — exporting a checkpoint to ONNX, compiling it to .hbm in the OpenExplorer (OE) Docker toolchain, and running the on-board control loop.

The single most important thing: a board can only run an OE-compiled .hbm. It cannot run a raw .pt/ONNX policy, and the on-board control script (bpu_control_robot.py) loads .hbm + .npy normalization files, never the PyTorch checkpoint. If the user copied a checkpoint to the board expecting it to drive the arm, stop and route them through the export→compile loop first.

Sources: official D-Robotics repos rdk_LeRobot_tools (stable / s100 / s600 branches), openpi_runtime (develop), huggingface.co/D-Robotics/openpi. Every non-trivial claim below is verified against those READMEs/scripts.

Scope boundary — read this before answering

This skill covers trained policy → BPU → on-board control loop ONLY.

• ✅ In scope: export ACT to ONNX, compile to .hbm, board-side hbm-runtime / C++ BPU runtime, bpu_control_robot.py, Pi0 client-server runtime.
• ❌ Out of scope (it's upstream LeRobot, not this repo): SO-101/SO-100 arm assembly, motor ID setup, zero-point calibration (lerobot-calibrate), teleoperation data collection (lerobot-record), ACT training (lerobot-train), serial ports/baud. Point the user to huggingface/lerobot and the SO-101 docs for that front half.

Which path / which branch (decision cheat-sheet)

Pick the path first, then the branch — they pin different LeRobot versions and toolchain settings.

Critical, recently-changed facts (the old guidance was the reverse — fix it):

• The active s100/s600 branches use UPSTREAM huggingface/lerobot v0.5.2 and explicitly say "Do not use the outdated D-Robotics/lerobot fork." The fork is only for the legacy stable branch (locked datasets, v2.1 datasets).
• ACT is now verified on both S100 (nash-e) and S600 (nash-p) via their dedicated branches. The stable branch README only claims S100; S100P shares the S100/Nash-e path. Other boards (X5/bayes*) are NOT verified — the export script has a bayes branch, but compiling a .bin ≠ a verified end-to-end arm deployment. State that boundary honestly.
• Both S100 and S600 ACT branches require the OE 3.7.0 S100/S600 toolchain for compilation.

Workflow A — LeRobot ACT deployment (three stages)

Full commands, bpu_export_config.yaml fields, and the C++ runtime build are in lerobot-workflow.md.

Stage 1 — Dev machine: export ONNX + compile configs

1. Set up the env. For the current path use upstream LeRobot, NOT the fork:

   git clone https://github.com/huggingface/lerobot.git && cd lerobot
   git clone https://github.com/D-Robotics/rdk_LeRobot_tools.git
   cd rdk_LeRobot_tools && git checkout s100   # or s600
   cd .. && pip install -e ".[feetech]"
   pip install onnx onnxsim termcolor tqdm safetensors

2. Edit the export YAML — dataset.root, act_path (ACT checkpoint dir with config.json + model.safetensors), and type = the march (nash-e for S100, nash-p for S600). On the s600 branch use the dedicated bpu_export_config_s600_calfix.yaml.
3. python export_bpu_actpolicy.py --config <yaml> → produces export_path/ with two ONNX submodels (BPU_ACTPolicy_VisionEncoder, BPU_ACTPolicy_TransformerLayers), per-model OE configs, calibration data, bpu_output/ normalization .npy, and build_all.sh. (ACT is split: VisionEncoder images → [1,512,15,20]; TransformerLayers states + features → Actions [1,100,6].)

Stage 2 — Dev machine: compile ONNX → .hbm

4. Inside the OE 3.7.0 Docker (x86 host, NOT the board) run cd export_path && bash build_all.sh. The OE toolchain (hb_compile) only accepts ONNX. Output: bpu_output/*.hbm plus the .npy files and new_actions.npy (pre-conversion PyTorch output, for accuracy checking).
5. scp the whole bpu_output/ folder to the board.

Stage 3 — Board: load .hbm and drive the arm

6. On the board, set up the runtime (branch-specific — see the pitfalls table for the Python-3.12 trap):
   • s600 branch: clone upstream lerobot + tools (git checkout s600), pip install -e ".[feetech]", pip install hbm-runtime.
   • s100 branch: Python 3.12 venv, then build the bundled bpu_runtime/ C++ pybind11 extension — the PyPI hbm-runtime wheel is Python-3.10-only and won't import. It builds bpu_act_runtime.*.so exposing BPUACTRuntime, a drop-in for hbm_runtime.HB_HBMRuntime.
7. Run the control loop:

   python bpu_control_robot.py --bpu-act-path ./bpu_output

   Key flags: --bpu-act-path (dir with .hbm + .npy), --fps (default 30), --inference-time (run seconds), --robot-port (default /dev/ttyACM0), --camera-index/--camera-name (default front). Default robot is so101 (change make_robot("so101") in code for another arm).
8. Verify correctness: compare board BPU output against new_actions.npy from the export step before trusting the arm.

Workflow B — Pi0 / openpi VLA (client-server, S600)

Minimal run sheet and data spec in lerobot-workflow.md section B.

Pi0 is a Vision-Language-Action runtime on RDK S600 (Ubuntu 24.04 / ROS2 Jazzy) driving a dual-arm setup. It is client-server, not a single board process:

• Client = the S600 inference node (s600_inference_node): time-synchronizes multi-camera ROS2 topics (TopicTimeSynchronizer, 100 ms window), preprocesses, calls the server, then smooths and executes actions.
• Server = Pi0 inference (OE-LLM): predicts the action sequence; piper_node drives the arm over CAN (can0).
• Input: 3× images [3,224,224] uint8 (head, left wrist; right wrist is a black placeholder) + state [14] float32 + a text prompt (e.g. "put the yellow mango on the blue plate").
• Output: action sequence [50,14] float32 (50 steps × [6 joints + gripper] per arm).
• Latency (reference): collect 0.1 ms / preprocess 3.2 ms / inference 192.5 ms (server) / postprocess 0.1 ms / execution ~700.5 ms. Action smoothing = 10-step interpolation in, first-order low-pass (alpha=0.15) in the middle, 10-step interpolation out; gripper bypasses filtering.
• Models: pre-quantized HBM + norm_stats.json from huggingface.co/D-Robotics/openpi; the norm_stats.json lives under <task>/torch/assets/trossen/ and must match the model.

Steps: download HBM + norm_stats.json → conda create -n s600_pi0 python=3.12 + pip install -r resource/requirements.txt → colcon build --packages-select openpi_runtime → launch the two D457 cameras, then piper_node, then s600_inference_node (all with the same ROS_DOMAIN_ID). Inference connects to the Pi0 server on port 8888.

Worked examples

Example 1 — "我训练好了一个 SO-101 的 ACT 策略,怎么部署到 RDK S100 上?" Walk Workflow A. Stage 1: clone upstream huggingface/lerobot (not the fork) + rdk_LeRobot_tools s100 branch, set type: nash-e and act_path in the export YAML, run export_bpu_actpolicy.py → two ONNX submodels. Stage 2: in OE 3.7.0 Docker run build_all.sh → bpu_output/*.hbm. Stage 3: on the board build the bpu_runtime/ C++ extension (Python 3.12, PyPI hbm-runtime won't import), then python bpu_control_robot.py --bpu-act-path ./bpu_output. Note that arm assembly/calibration/data collection are upstream LeRobot, not this repo.

Example 2 — "S600 上跑 ACT,march 填什么?跟 S100 一样吗?" No — S600 uses nash-p (S100 uses nash-e), via the rdk_LeRobot_tools s600 branch and bpu_export_config_s600_calfix.yaml. Both need OE 3.7.0. S600 board-side install uses pip install hbm-runtime directly (the Python-3.12 C++ extension build is the s100-branch path). Artifact is .hbm either way.

Example 3 — "Pi0 / openpi 是不是直接在板子上跑就行?" No — it's client-server. The S600 board runs the client (s600_inference_node: camera sync + preprocess + execute) and piper_node (arm control over CAN); the Pi0 model runs on the server (OE-LLM) and is reached on port 8888. Inputs are 3 images + a [14] state + a text prompt; output is [50,14] actions. Get the HBM model + matching norm_stats.json from huggingface.co/D-Robotics/openpi.

Example 4 — "我直接用 huggingface 最新的 lerobot 装,加载老数据集报错了" Two cases. If you're on the current s100/s600 branches, they target upstream v0.5.2 with new-format datasets — re-collect or convert, and don't use the D-Robotics fork. If you must load legacy v2.1 datasets, switch to the stable branch and use the D-Robotics/lerobot fork (locked datasets), or pin datasets==2.19.0 on upstream. Mixing a current branch with v2.1 data (or vice-versa) is the usual cause of the load error.

Common pitfalls

Anti-hallucination guardrails

When answering from this skill, follow these rules — never fabricate facts, commands, or file paths:

1. Report only observed data. Quote what scripts/commands actually return, not what you remember. If the probe says board_id: X5, answer for X5 — even if the user insists it's an S100.
2. No fabrication when tools are missing. If a script or reference doesn't exist, say "not found" — don't invent from memory. Route to the appropriate skill or doc instead.
3. Preserve null/false/empty on failure. If a probe returns null or false, report that — don't substitute a plausible value. Empty output is data, not an error to "fix".
4. No substitution off-platform. If off_platform: true, say "probe didn't run on an RDK board" — don't guess what it would have returned. Ask for on-board logs.
5. No hand-editing JSON. Scripts emit structured JSON; never hand-craft output. If the contract says {ok,off_platform,reason,fields}, that's what goes to the user.
6. Acknowledge sandbox limits. If you can't run a command, say so — don't pretend you did. Offer the command for the user to run.
7. Read-only boundary. Never modify the system — no dd, mkfs, rm -rf, apt install, reboot, or GPIO output without explicit user confirmation.

Reference map