RDK Multimedia Hardware Pipeline (codec / capture / scale / display)

This skill moves raw pixel streams between on-board hardware units: sensor in → ISP tuned → scaled → VPU/JPU encoded/decoded → pushed out to HDMI/MIPI. It does not cover model inference (→ rdk-device) or wrapping any of this into a ROS node (→ rdk-ros).

Sources: official D-Robotics docs. X-series (rdk_doc): docs/07_Advanced_development/03_multimedia_development/* and docs/03_Basic_Application/06_multi_media_sp_dev_api/RDK_X5/*. S-series (rdk_s_doc): docs/07_Advanced_development/03_multimedia_development/{01_S100,02_multimedia_application,03_S600_multimedia_application}/*. Every spec below was re-verified against these files; numbers nobody could confirm are flagged in the reference's "Unverified" section.

The one thing that matters most

X-series and S-series are two different multimedia stacks — do not map one onto the other. The X-series VPS/VOT vocabulary (HB_VPS_*, HB_VOT_*, sp_dev) does not exist on the S-series; the S-series uses Camsys/PYM/GDC + MediaCodec + IDE/IDU instead. Confirm the board family first, then pick the matching API column. A user who pastes HB_VOT_* code on an S600 is on the wrong stack entirely.

Stack cheat-sheet (pick the column for the board)

S-series adds STITCH (image stitching). The full X pipeline is VIN → VPS → VENC/VDEC+JPU → VOT; the full S pipeline is Camera+SerDes → VIN → ISP → YNR → PYM → (GDC) → CODEC → IDE/IDU.

Two development paths (ask the user which, before reading code)

1. sp_dev — the easy wrapper (X3/X5/Ultra, recommended for prototyping). On-board at /app/cdev_demo/, C prefix sp_* (plus a Python binding). Four modules: VIO / Encoder / Decoder / Display, chained with sp_module_bind. Ready-made demos: vio2encoder (camera→.h264), decoder2display (file→HDMI), rtsp2display (RTSP→HDMI), VPS scaling. Make the user run a demo to prove the path works before editing any code.
2. Low-level HB_* (need precise GOP/bitrate/ROI/multi-channel control). HB_MIPI_* / HB_VIN_* / HB_VPS_* / HB_VENC_* / HB_VDEC_* / HB_VOT_* + HB_SYS_* binding. On the S-series, use the /app/multimedia_samples/sample_{vin,isp,pym,gdc,codec,pipeline} programs as templates — sample_pipeline is the full VIN→ISP→YNR→PYM→(GDC)→CODEC chain.

Full function tables, codec specs, resolution limits, bitrate modes, and the complete X-vs-S map are in multimedia-pipeline.md. For a deterministic spec lookup, run scripts/codec_spec.py <board> <unit>.

Workflows

Workflow 1 — "Encoding is slow / CPU pinned / ffmpeg" (most common)

1. Confirm it is hardware, not software, encoding. ffmpeg defaults to a software encoder (libx264) and pins the CPU. Hardware H264/H265/JPEG must go through the VPU/JPU — i.e. sp_dev (sp_start_encode), HB_VENC_*, or the S-series MediaCodec / sample_codec.
2. Route to the right API by board family (cheat-sheet). Don't tune ffmpeg threads — switch the encode backend.
3. Prove the path with a demo: vio2encoder on X, sample_codec on S. If FPS jumps, it was the software-codec trap.

Workflow 2 — Encode reports a size / stride / alignment error

1. Apply the alignment rule for the codec (from the reference):
   • H.264/H.265: stride 32-byte aligned, width & height 8-byte aligned.
   • JPEG/MJPEG: stride 32-byte aligned, width 16-byte, height 8-byte aligned.
   • S-series VPU: width 32, height 8 aligned.
2. If the raw frame isn't aligned, crop with VIDEO_CROP_INFO_S (X-series) rather than forcing the encoder.
3. Re-check that the resolution is within the unit's min/max (e.g. H264 min 256×128 on X3) before assuming a bug.

Workflow 3 — "I need to scale / output multiple resolutions"

1. X-series → VPS. One IPU drives 7 output channels (chn0–chn6). chn0–chn4 downscale (to 1/8 of input, > 1/8); chn5 is the only upscale channel (≤ 1.5×); chn6 is the PYM online channel.
2. sp_dev shortcut (X): sp_open_camera / sp_open_vps set up to 5 resolution groups at once — 1 may upscale, 4 downscale. sp_open_camera upscale ≤ 1.5×; sp_open_vps's standalone VPS supports a larger upscale (doc lists up to 4×). Validate at a low resolution first.
3. S-series → PYM (shrink + ROI) and GDC for geometric correction — there is no "VPS". PYM min output is 32×32.

Workflow 4 — Display output (HDMI / MIPI panel)

1. X-series → VOT. Outputs RGB / BT1120(BT656) / MIPI, all max 1080P@60fps. sp_dev: sp_start_display + sp_display_set_image; overlay with sp_display_draw_rect / sp_display_draw_string.
2. S-series → IDE/IDU. S100 has 2 IDUs, 6 channels (ch 0/1/4/5 = YUV layers, 2/3 = RGB layers), max input 2880×2160 per channel; YUV layers Up-Scale up to 6×; output via MIPI DSI or MIPI CSI2 Device (they share one MIPI D-PHY). There is no HB_VOT_* here.

Workflow 5 — S600-specific: VPU multi-core

1. Only S600 exposes multiple VPU cores. In the MediaCodec sample_codec (the -m/-c/-w/-h/-p/-n/-u variant), -u selects the VPU core id (default 0, values 0/1/2). On S100/S100P the -u flag has no effect.
2. Don't confuse the two same-named samples: the -u core flag is on the MediaCodec API sample (./sample_codec --samplemode ...); the /app/multimedia_samples/sample_codec config-file program (-f codec_config.ini -e/-d <bitmask>) has no -u.
3. Encode ceilings are shared S100/S600: H264 High@L5.2, H265 Main / Main-tier @L5.1, MJPEG/JPEG ISO/IEC 10918-1 Baseline sequential.

Stop rule: if the same error repeats twice, stop retrying the same parameter — read the matching rdk_doc / rdk_s_doc chapter and the board log instead.

Worked examples

Example 1 — "我用 ffmpeg 编码 H264 把 CPU 占满了，RDK 上怎么硬件编码?" ffmpeg defaults to a software encoder, which is why the CPU is pinned. On X3/X5/Ultra use sp_dev (sp_start_encode with SP_ENCODER_H264) or HB_VENC_* to hit the VPU; the quickest proof is the vio2encoder demo in /app/cdev_demo/. On S100/S600 use the MediaCodec sample_codec. The VPU does up to 4K@60 (X3) / 4K@90 (S). Then point to Workflow 1.

Example 2 — "VPS 一个 IPU 能出几路?哪一路能放大?" On the X-series VPS, one IPU drives 7 channels (chn0–chn6): chn0–chn4 downscale (down to 1/8), chn5 is the only upscale channel, max 1.5×, chn6 is the PYM online channel. Via sp_dev you set up to 5 resolution groups in one sp_open_camera/sp_open_vps call (1 up, 4 down). Note this is X-series only — S-series uses PYM, not VPS.

Example 3 — "S600 编解码示例里的 -u 是干嘛的?S100 上为什么不生效?" -u selects the VPU core id (default 0, values 0/1/2) in the MediaCodec sample_codec (the --samplemode variant). Only S600 has multiple VPU cores, so -u only takes effect on S600 — on S100/S100P it does nothing. The config-file sample_codec (-f ... -e/-d) doesn't have -u at all. (Run scripts/codec_spec.py s600 vpu for the spec.)

Example 4 — "编码报 stride/尺寸错误,我的图是 1920×1082" H.264/H.265 require width & height 8-byte aligned and stride 32-byte aligned; 1082 isn't 8-aligned. Either capture/scale to an 8-aligned height (1080) or crop with VIDEO_CROP_INFO_S. Don't keep resubmitting the same frame to the encoder. Check Workflow 2.

Common pitfalls

Anti-hallucination guardrails

When answering from this skill, follow these rules — never fabricate facts, commands, or file paths:

1. Report only observed data. Quote what scripts/commands actually return, not what you remember. If the probe says board_id: X5, answer for X5 — even if the user insists it's an S100.
2. No fabrication when tools are missing. If a script or reference doesn't exist, say "not found" — don't invent from memory. Route to the appropriate skill or doc instead.
3. Preserve null/false/empty on failure. If a probe returns null or false, report that — don't substitute a plausible value. Empty output is data, not an error to "fix".
4. No substitution off-platform. If off_platform: true, say "probe didn't run on an RDK board" — don't guess what it would have returned. Ask for on-board logs.
5. No hand-editing JSON. Scripts emit structured JSON; never hand-craft output. If the contract says {ok,off_platform,reason,fields}, that's what goes to the user.
6. Acknowledge sandbox limits. If you can't run a command, say so — don't pretend you did. Offer the command for the user to run.
7. Read-only boundary. Never modify the system — no dd, mkfs, rm -rf, apt install, reboot, or GPIO output without explicit user confirmation.

Reference map