add-omni-model
Skill7.8k starsChanged 3 days agoApache-2.0View on GitHub
Integrate a new multi-stage autoregressive omni model (thinker/talker/code2wav style, multimodal input with text and speech output) into vLLM-Omni, from the Hugging Face reference through stage separation, pipeline and deploy config, stage input processors, async_chunk streaming, parity validation, tests, and recipe. Use when adding a model like Qwen3-Omni, Qwen2.5-Omni, or MiniCPM-o, splitting an omni checkpoint into stages, writing `pipeline.py` or `stage_input_processors/` hooks, wiring `_OMNI_MODELS` / `OMNI_PIPELINES`, or debugging parity of a newly integrated omni model. Not for diffusion models (`add-diffusion-model`) or TTS-only models (`add-tts-model`).
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Who's talking about vllm-project/vllm-omni
- verl-project/verl“[RFC] Multi-modal Generation RL 2026Q2 Roadmap”GitHub · Mar 26, 2026 · 16 reactions
- vllm-project/vllm“[Model][VLM] Add Qwen2.5-Omni model support (end-to-end full support)”GitHub · Apr 9, 2025 · 11 reactions
- verl-project/verl“[RFC] Support Qwen-Image Flow-GRPO Training based on vLLM-Omni”GitHub · Dec 22, 2025 · 10 reactions
- verl-project/verl“[RFC] Support Diffusion Generative Models and Async Reward during Rollout”GitHub · Dec 17, 2025 · 8 reactions
- vllm-project/vllm“[Migration] Migrate GGUF quantization support to plugin”GitHub · Apr 12, 2026 · 7 reactions
The biggest 5 of 51 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 51
Talked about on GitHub and Hacker News (51 times).
On GitHub · 50
- Mar 26, 2026[RFC] Multi-modal Generation RL 2026Q2 Roadmap — verl-project/verl16 reactions · 21 comments
- Apr 9, 2025[Model][VLM] Add Qwen2.5-Omni model support (end-to-end full support) — vllm-project/vllm11 reactions · 11 comments
- Dec 22, 2025[RFC] Support Qwen-Image Flow-GRPO Training based on vLLM-Omni — verl-project/verl10 reactions · 8 comments
- Dec 17, 2025[RFC] Support Diffusion Generative Models and Async Reward during Rollout — verl-project/verl8 reactions · 3 comments
- Apr 12, 2026[Migration] Migrate GGUF quantization support to plugin — vllm-project/vllm7 reactions · 16 comments
45 more on GitHub
- Aug 14, 2026[RFC] Tracking v1 upgrade for verl-omni v0.3.0: remaining recipe coverage — verl-project/verl-omni5 reactions · 19 comments
- Apr 29, 2026在vLLM-Omni中支持SenseNova-U1 — OpenSenseNova/SenseNova-U14 reactions · 6 comments
- Dec 7, 2025RL post training for omni-modality models — ByteDance-Seed/VeOmni4 reactions · 2 comments
- Oct 16, 2026Add MOSS-Image Base and SFT recipe — vllm-project/recipes2 reactions · 2 comments
- Aug 3, 2026[Draft] [Bugfix] Clone safetensors tensors before H2D copy on Ascend NPU to fix DMA hang — vllm-project/vllm2 reactions · 3 comments
- Jun 17, 2026[ROCm][Perf] Fix Qwen3-vLLM audio encoder TP when heads are not divisible by TP size — vllm-project/vllm2 reactions · 14 comments
- Jun 16, 2026[RFC]: Qwen3-Omni RL Support on FSDP with vLLM-Omni rollout — verl-project/verl-omni2 reactions · 1 comment
- May 11, 2026[Metrics] Scope unregister_vllm_metrics() to strictly "vllm:" metrics — vllm-project/vllm2 reactions · 10 comments
- Apr 14, 2026[RFC]: Support Mooncake Based ECConnector for EPD — vllm-project/vllm2 reactions · 3 comments
- Apr 7, 2026Collaboration: Integrate VoxCPM into vllm-omni — a710128/nanovllm-voxcpm2 reactions · 4 comments
- Feb 25, 2026vllm-omni is supporting the model inference — bytedance/mammothmoda2 reactions · 3 comments
- Jan 10, 2026[Models] Add `SharedFusedMoE` support to Qwen3MoE — vllm-project/vllm2 reactions · 2 comments
- Jan 9, 2026[Feature]: RL Integration: Support for the veRL + veOmni + vLLM-Omni stack. — JiusiServe/vllm-omni2 reactions · 3 comments
- Dec 20, 2025[RFC]: Optional Hidden States Transfer in `KVConnectorBase_V1` to Eliminate Decoder Prefix-Prefill in P/D Disaggregation — vllm-project/vllm2 reactions · 2 comments
- Nov 12, 2026Add opt-in single RTX 4090 hardware profile — vllm-project/recipes1 reaction · 1 comment
- Nov 10, 2026Sync recipes from vllm-project/vllm-omni@d8751cd14a85 — vllm-project/recipes1 reaction · 2 comments
- Oct 29, 2026Sync recipes from vllm-project/vllm@9a0f9b45df86, vllm-project/vllm-omni@46766eb27f3e — vllm-project/recipes1 reaction · 2 comments
- Oct 5, 2026feat: upgrade vllm from 0.30.0 to 0.31.0 for CUDA and ROCm backends — gpustack/runner1 reaction · 2 comments
- Oct 2, 2026[Model] Add Nemotron 3.5 ASR transcription support — vllm-project/vllm1 reaction · 9 comments
- Sep 23, 2026feat: upgrade vllm from 0.29.0 to 0.30.0 for CUDA and ROCm backends — gpustack/runner1 reaction · 2 comments
- Sep 21, 2026feat(voice): add VoxCPM as a local TTS provider — moltis-org/moltis1 reaction · 1 comment
- Sep 3, 2026Add FastH3 VSA recipe — vllm-project/recipes1 reaction · 1 comment
- Jul 23, 2026[Bug]: HunyuanImage-3.0-Instruct (AR+DiT) startup hangs when waitting engine core process start on Ascend A3 — vllm-project/vllm-ascend1 reaction · 4 comments
- Jul 21, 2026[vLLM backend][Feature Request] Support vLLM-Omni for native audio/video input — triton-inference-server/server1 reaction · 3 comments
- Jul 9, 2026[RFC] Refactor experimental step execution support by reusing vLLM-Omni interfaces — verl-project/verl-omni1 reaction · 7 comments
- Jun 16, 2026Add user guide for v2v and transfer for vllm-omni — NVIDIA/cosmos1 reaction · 5 comments
- Jun 15, 2026[Quantization] Extend ModelOpt mixed precision and NVFP4 runtime formats — vllm-project/vllm1 reaction · 6 comments
- Jun 12, 2026[Draft]Support Megatron LoRA adapter export for rollout — verl-project/verl1 reaction · 1 comment
- Jun 12, 2026[Draft]Support Qwen3-Omni Megatron LoRA rollout — verl-project/verl-omni1 reaction · 1 comment
- Jun 11, 2026310p 部署 Qwen3-TTS — vllm-project/vllm-ascend1 reaction · 11 comments
- Jun 9, 2026docs(diffusion): add Qwen-Image v0.20→v0.22 H200 retro — hsliuustc0106/vllm-omni-cookbook1 reaction · 6 comments
- Jun 6, 2026[Kernel] FlashInfer FP8 scaled_mm: restore N-D output shape — vllm-project/vllm1 reaction · 3 comments
- May 22, 2026[feat]: Cosmos3 port (Tier-A scaffold; SSIM + Tier-B deferred pending weight publish) — hao-ai-lab/FastVideo1 reaction · 8 comments
- May 20, 2026[RFC] Q3 Road Map (welcome contributions) — verl-project/verl-omni1 reaction · 30 comments
- May 15, 2026Add extra_labelnames to PrometheusStatLogger for extensibility — vllm-project/vllm1 reaction · 3 comments
- May 14, 2026Mxfp4 support — yiliu30/vllm-omni-fork1 reaction
- Apr 20, 2026Custom pipeline class fix — knlnguyen1901/vllm-omni1 reaction
- Mar 18, 2026Add SoulX-Duplug turn-serving support — nemoramo/vllm-omni1 reaction · 1 comment
- Jan 12, 2026vLLM/vLLM-Omni integration — NAVER-Cloud-HyperCLOVA-X/OmniServe1 reaction
- Jan 9, 2026[Feature][Community]: Model Support: Add support for Bagel and HunyuanImage3.0 models in disaggregated setups. — JiusiServe/vllm-omni1 reaction
- Oct 6, 2026[Feature] Support Irodori-TTS Japanese-focused TTS in SGLang-Omni — sgl-project/sglang-omni0 reactions
- Aug 5, 2026[Feature]: Diffusion kv block manager — JiusiServe/vllm-omni0 reactions
- Aug 2, 2026[Bug]: Shared prompt_token_ids causes stale M-RoPE positions — vllm-project/vllm0 reactions · 1 comment
- Apr 1, 2026[RFC]: HunyuanImage DIT Fa3-fp8 quantization — JiusiServe/vllm-omni0 reactions · 1 comment
- Feb 27, 2026Support realtime API for audio streaming — EricLBuehler/mistral.rs0 reactions · 1 comment
On Hacker News · 1
- Dec 2, 2025VLLM-Omni: A framework for efficient model inference with Omni-modality models2 points · 1 comment
In this repository29 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- 3d-animation-short-generatorSkill
- brand-promo-video-generatorSkill
- co-op-game-intro-generatorSkill
- h3-prompt-writingSkill
- handdrawn-live-video-generatorSkill
- minimalist-product-ad-generatorSkill
- music-video-subtitle-generatorSkill
- paper-collage-explainer-generatorSkill
- papercraft-stop-motion-explainerSkill
- vllm-omni-h3-character-replacementSkill
- add-diffusion-modelSkill
- add-recipeSkill
- add-tts-modelSkill
- diffusion-accuracy-testSkill
- diffusion-perf-optSkill
- diffusion-perf-testSkill
- dit-npu-fusion-analysisSkill
- find-simplificationsSkill
- omni-perf-ceilingSkill
- omni-perf-optSkill
- precheck-prSkill
- production-add-diffusion-modelSkill
- quantizationSkill
- review-prSkill
- vllm-omni-npu-model-runner-upgradeSkill
- vllm-omniSkill
- vllm-omni-testSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 2 are distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 36
Check these at the end of every phase.
- Distinctive: line 64
Run the official implementation end to end with fixed inputs (text-only, image+text, audio+text, and a speech-output request).
- Very specific: line 100
Keep __init__.py import-light.
- Very specific: line 101
If a stage needs a different class (MiniCPM-o's MiniCPMO45Code2Wav), give it its own arch and point the stage at it with model_arch.
- Very specific: line 106
Add every arch name to _OMNI_MODELS in vllm_omni/model_executor/models/registry.py as (folder, module, class).
- Very specific: line 119
Write vllm_omni/model_executor/stage_input_processors/<name>.py with, per edge:
- Very specific: line 138
Create vllm_omni/deploy/<name>.yaml with a header comment naming the verified hardware.
- Very specific: line 190
No new model-specific Python under examples/ (.claude/skills/precheck-pr/references/examples-policy.md).
- Very specific: line 195
Run the precheck-pr skill.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Add a Multi-Stage Omni Model
Overview
HF reference -> Stage split -> Pipeline + registry -> Stage hand-off -> Deploy YAML
(Phase 1) (Phase 2) (Phase 3) (Phase 4) (Phase 5)
-> Parity ladder -> Tests + CI -> Recipe + docs -> precheck-pr
(Phase 6) (Phase 7) (Phase 8) (Phase 9)Canonical in-tree examples:
| Model | Package | Stage names | Notable |
| Qwen3-Omni | vllm_omni/model_executor/models/qwen3_omni/ | thinker / talker / code2wav | Unified class, resolver for thinker-only variant, async_chunk default |
| Qwen2.5-Omni | vllm_omni/model_executor/models/qwen2_5_omni/ | thinker / talker / code2wav | async_chunk: false default |
| MiniCPM-o 4.5 | vllm_omni/model_executor/models/minicpmo_4_5/ | llm / tts / code2wav | Separate Code2Wav arch via model_arch, duplex plugin |
Developer guide: docs/contributing/model/adding_omni_model.md. Its stage
processor section still describes the older orchestrator-side
custom_process_input_func(stage_list, ...) signature. Current models use both
producer-side hooks and consumer-side custom_process_input_func (MiniCPM-o's
llm2tts); prefer the Qwen3-style producer hooks for new models. See the current
signatures in the pipeline reference.
The guide's e2e test path prefix vllm_omni/tests/... is wrong; tests live under
tests/.
Cross-cutting invariants
Check these at the end of every phase.
- I1. Topology in code, runtime in YAML.
model_stage,execution_type, input sources, processor functions, and final outputs live inpipeline.py. The deploy YAML only sets devices, memory, batch limits, graphs, connectors,async_chunk,session_mode, andmodel_runner. - I2. One stage per process. The unified class reads
vllm_config.model_config.model_stageand builds only that stage's modules and weights. Never instantiate all stages in every process. - I3. Isolate request-specific streaming/recurrent state. Key codec history
and recurrent state by request id; release them on completion or cancellation.
Shape-keyed CUDA graphs and static work buffers may persist across requests.
Synchronize their reuse, initialize every input region read by replay
(including padding), and consume or copy outputs before replay overwrites
them. See
vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.pyfor a shared graph cache. - I4. Delta streaming. Streaming paths emit only new audio per step;
every
forward()return path sets the output keys downstream expects. - I5. Both hand-off modes stay coherent. For every stage edge, the
*_full_payload(non-streaming) and*_async_chunk(streaming) processors must produce semantically equivalent next-stage inputs. - I6. Source is authoritative. Confirm field names in
vllm_omni/config/stage_config.pyand an in-tree model before copying from docs or this skill.
Phase 1: Hugging Face reference
- Run the official implementation end to end with fixed inputs (text-only, image+text, audio+text, and a speech-output request).
- Save golden artifacts: thinker text (greedy), talker codec ids for a fixed seed, and the code2wav waveform for those ids.
- Freeze the reference environment and numerical contract (dtype, attention backend, buffer initialization, and RNG behavior); see the parity reference.
- Record the architecture: sub-models, which hidden layers the talker
consumes, special tokens, codec codebooks and frame rate, hop length,
sample rate, and the HF
config.jsonsub-config names.
Deliverable: a reference script and golden files you can diff against at every later phase.
Phase 2: Stage split and model package
Map the dependency graph before splitting processes. Keep modules with per-step feedback together unless the runtime supports that feedback edge; sub-model names do not imply stage boundaries. For example, a temporal decoder that consumes its depth head's audio frame on the next step can share an AR stage with that head, with only code2wav downstream. See the LFM3 integration case.
Create vllm_omni/model_executor/models/<name>/:
- One module per stage (for example
<name>_thinker.py,<name>_talker.py,<name>_code2wav.py). - A unified class (
<Name>ForConditionalGeneration) that switches onmodel_stage, builds only that stage, and routesload_weights()by prefix (thinker.,talker.,code2wav.). Followqwen3_omni/qwen3_omni.py. forward()returnsOmniOutput(model_executor/models/output_templates.py):text_hidden_states,multimodal_outputs(the payload keys the next stage reads, such as layer hidden states or codec codes), and optionalnext_token_id.- Multimodal thinkers implement
SupportsMultiModal(plusSupportsMRoPE/SupportsPPwhen applicable) and register their processor withMULTIMODAL_REGISTRY.register_processor. - Keep
__init__.pyimport-light; the registry imports lazily. - If a stage needs a different class (MiniCPM-o's
MiniCPMO45Code2Wav), give it its own arch and point the stage at it withmodel_arch.
Phase 3: Pipeline and registry
- Add every arch name to
_OMNI_MODELSinvllm_omni/model_executor/models/registry.pyas(folder, module, class). - Define a frozen
PipelineConfigin<name>/pipeline.pywith oneStagePipelineConfigper stage anddefault_deploy_config_name. - Register it in
OMNI_PIPELINESinvllm_omni/config/pipeline_registry.py(alphabetical bymodel_type). Use@pipeline_cfg_resolverwhen the HF config selects a variant (Qwen3-Omni returns a thinker-only pipeline whenenable_audio_outputis false).
Field-by-field guidance: references/pipeline-and-handoff.md.
Phase 4: Stage hand-off
Write vllm_omni/model_executor/stage_input_processors/<name>.py with, per
edge:
<a>2<b>_full_payload: packs the complete upstream output whenasync_chunkis off.<a>2<b>_async_chunk: packs one incremental chunk whenasync_chunkis on.<a>2<b>_token_only(optional): orchestrator-side placeholder that only allocatesprompt_token_idsof the right length for KV reservation.
Wire them through custom_process_next_stage_input_func,
async_chunk_process_next_stage_input_func, and sync_process_input_func.
_select_processor_funcs in stage_config.py picks the variant from the
effective async_chunk. Reference: stage_input_processors/qwen3_omni.py.
Existing edges may instead use the receiving stage's
custom_process_input_func, such as MiniCPM-o stage 1's llm2tts; preserve
that consumer-side contract when extending an existing pipeline.
Phase 5: Deploy YAML
Create vllm_omni/deploy/<name>.yaml with a header comment naming the
verified hardware. Start conservative and correct:
- Choose placement from the actual stage topology and device memory budget. For a three-stage CUDA model, a starting point is a dedicated thinker GPU and a second GPU shared by talker and code2wav. Budget co-located stages together, including weights, KV, encoder peaks, and graph buffers.
enable_prefix_caching: falseon every stage unless the pipeline has been validated with it.- Connector block (
SharedMemoryConnectoron a single node) with codec chunk settings whenasync_chunk: true. - Talker
default_sampling_paramswith a fixedseedfor reproducible speech. platforms:sections only for platforms you actually ran.
Schema and layering: .claude/skills/vllm-omni/references/stage-config.md.
Performance profiles (MRv2, graph tuning, replicas) are a later step; use the
omni-perf-opt skill once parity passes.
Phase 6: Parity ladder
Climb in order; do not move up until the current rung passes. Details and comparison methods: references/parity-and-validation.md.
- Thinker text with
"modalities": ["text"]matches HF greedy output. - Talker codec ids match HF for a fixed seed and identical conditioning.
- Code2wav waveform matches HF for identical codec ids (within tolerance).
- Full pipeline,
async_chunkoff: audio matches rung 3. async_chunkon: compare against the reference streaming decoder with matching chunk settings; measure any reference full/stream difference separately. Check duration, truncation, repetition, and continuity.- Concurrency of at least 2: per-request outputs equal their single-request outputs (no crosstalk).
- Graphs on versus all relevant graphs off: matching output.
--enforce-eageralone may leave model-internal graphs enabled.
Phase 7: Tests and CI
- Component:
tests/model_executor/models/<name>/andtests/model_executor/stage_input_processors/test_<name>*.py(CPU-only, cover both full-payload and async-chunk processors). - E2E:
tests/e2e/offline_inference/test_<name>.py,tests/e2e/online_serving/test_<name>.py, optionaltests/e2e/accuracy/<name>/. - Optional perf rows:
tests/dfx/perf/tests/test_<name>.json(seetest_qwen3_omni_async_chunk.jsonfor the schema). - Markers, levels, Buildkite wiring, and run commands:
vllm-omni-testskill.
Phase 8: Recipe and docs
- Recipe under
recipes/<Vendor>/<Model>.mdplus itsrecipes/README.mdrow via theadd-recipeskill. - Row in
docs/models/supported_models.mdlisting only validated backends. - No new model-specific Python under
examples/(.claude/skills/precheck-pr/references/examples-policy.md).
Phase 9: Pre-PR
Run the precheck-pr skill. The PR must include a head-versus-reference
accuracy and performance comparison with pinned revisions
(.claude/skills/review-pr/references/checks/model-addition-checklist.md).
Integration checklist
Invariants
- [ ] I1 topology only in pipeline.py; deploy YAML has runtime keys only
- [ ] I2 each stage process builds and loads only its own modules
- [ ] I3 request streaming/recurrent state isolated and freed on finish/cancel
- [ ] I3 shared graph buffers synchronized, initialized, and outputs safe on reuse
- [ ] I4 streaming emits deltas; every forward() path sets payload keys
- [ ] I5 full_payload and async_chunk processors agree for each edge
Phases
- [ ] 1 HF reference runs; golden text, codec ids, waveform saved
- [ ] 2 Per-stage modules + unified class; forward returns OmniOutput
- [ ] 3 Arch names in _OMNI_MODELS; PipelineConfig in OMNI_PIPELINES
- [ ] 4 Stage input processors for every edge, both modes
- [ ] 5 Default deploy YAML with verified-hardware header
- [ ] 6 Parity ladder rungs 1-7 pass
- [ ] 7 Component + e2e tests, CI wiring via vllm-omni-test
- [ ] 8 Recipe, recipes/README.md row, supported_models.md row
- [ ] 9 precheck-pr passes; reference comparison in the PRReferences
- references/pipeline-and-handoff.md:
StagePipelineConfigfields, processor hook contracts, skeletons - references/parity-and-validation.md: parity ladder methods, common failure causes, evidence to attach
.claude/skills/vllm-omni/references/models/qwen3-omni.md,.claude/skills/vllm-omni/references/models/minicpm-o.md: worked examples.claude/skills/vllm-omni/references/debugging.md: symptom tableomni-perf-optskill: tuning after parity
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

