vllm-omni
Skill7.8k starsChanged 3 days agoApache-2.0View on GitHub
Deploy, serve, debug, benchmark, and extend multi-stage omni models in vLLM-Omni (Qwen3-Omni, Qwen2.5-Omni, MiniCPM-o, and similar thinker/talker/code2wav pipelines). Use when writing or tuning deploy YAML under vllm_omni/deploy, launching `vllm serve --omni`, diagnosing stage startup hangs, OOM, connector stalls, or broken streaming audio, measuring TTFT/AUDIO_TTFP/AUDIO_RTF with `vllm bench serve --omni`, or navigating the orchestrator and stage runtime code. Routes new omni model integration to `add-omni-model`, omni performance optimization to `omni-perf-opt`, and diffusion, TTS-only, quantization, testing, recipe, and PR tasks to their dedicated skills.
1 other project uses a version of this.
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Most recently taken up by
- air-gapped/skillstheir own version, first seen Oct 19, 2026
Who's talking about vllm-project/vllm-omni
- verl-project/verl“[RFC] Multi-modal Generation RL 2026Q2 Roadmap”GitHub · Mar 26, 2026 · 16 reactions
- vllm-project/vllm“[Model][VLM] Add Qwen2.5-Omni model support (end-to-end full support)”GitHub · Apr 9, 2025 · 11 reactions
- verl-project/verl“[RFC] Support Qwen-Image Flow-GRPO Training based on vLLM-Omni”GitHub · Dec 22, 2025 · 10 reactions
- verl-project/verl“[RFC] Support Diffusion Generative Models and Async Reward during Rollout”GitHub · Dec 17, 2025 · 8 reactions
- vllm-project/vllm“[Migration] Migrate GGUF quantization support to plugin”GitHub · Apr 12, 2026 · 7 reactions
The biggest 5 of 51 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 51
Talked about on GitHub and Hacker News (51 times).
On GitHub · 50
- Mar 26, 2026[RFC] Multi-modal Generation RL 2026Q2 Roadmap — verl-project/verl16 reactions · 21 comments
- Apr 9, 2025[Model][VLM] Add Qwen2.5-Omni model support (end-to-end full support) — vllm-project/vllm11 reactions · 11 comments
- Dec 22, 2025[RFC] Support Qwen-Image Flow-GRPO Training based on vLLM-Omni — verl-project/verl10 reactions · 8 comments
- Dec 17, 2025[RFC] Support Diffusion Generative Models and Async Reward during Rollout — verl-project/verl8 reactions · 3 comments
- Apr 12, 2026[Migration] Migrate GGUF quantization support to plugin — vllm-project/vllm7 reactions · 16 comments
45 more on GitHub
- Aug 14, 2026[RFC] Tracking v1 upgrade for verl-omni v0.3.0: remaining recipe coverage — verl-project/verl-omni5 reactions · 19 comments
- Apr 29, 2026在vLLM-Omni中支持SenseNova-U1 — OpenSenseNova/SenseNova-U14 reactions · 6 comments
- Dec 7, 2025RL post training for omni-modality models — ByteDance-Seed/VeOmni4 reactions · 2 comments
- Oct 16, 2026Add MOSS-Image Base and SFT recipe — vllm-project/recipes2 reactions · 2 comments
- Aug 3, 2026[Draft] [Bugfix] Clone safetensors tensors before H2D copy on Ascend NPU to fix DMA hang — vllm-project/vllm2 reactions · 3 comments
- Jun 17, 2026[ROCm][Perf] Fix Qwen3-vLLM audio encoder TP when heads are not divisible by TP size — vllm-project/vllm2 reactions · 14 comments
- Jun 16, 2026[RFC]: Qwen3-Omni RL Support on FSDP with vLLM-Omni rollout — verl-project/verl-omni2 reactions · 1 comment
- May 11, 2026[Metrics] Scope unregister_vllm_metrics() to strictly "vllm:" metrics — vllm-project/vllm2 reactions · 10 comments
- Apr 14, 2026[RFC]: Support Mooncake Based ECConnector for EPD — vllm-project/vllm2 reactions · 3 comments
- Apr 7, 2026Collaboration: Integrate VoxCPM into vllm-omni — a710128/nanovllm-voxcpm2 reactions · 4 comments
- Feb 25, 2026vllm-omni is supporting the model inference — bytedance/mammothmoda2 reactions · 3 comments
- Jan 10, 2026[Models] Add `SharedFusedMoE` support to Qwen3MoE — vllm-project/vllm2 reactions · 2 comments
- Jan 9, 2026[Feature]: RL Integration: Support for the veRL + veOmni + vLLM-Omni stack. — JiusiServe/vllm-omni2 reactions · 3 comments
- Dec 20, 2025[RFC]: Optional Hidden States Transfer in `KVConnectorBase_V1` to Eliminate Decoder Prefix-Prefill in P/D Disaggregation — vllm-project/vllm2 reactions · 2 comments
- Nov 12, 2026Add opt-in single RTX 4090 hardware profile — vllm-project/recipes1 reaction · 1 comment
- Nov 10, 2026Sync recipes from vllm-project/vllm-omni@d8751cd14a85 — vllm-project/recipes1 reaction · 2 comments
- Oct 29, 2026Sync recipes from vllm-project/vllm@9a0f9b45df86, vllm-project/vllm-omni@46766eb27f3e — vllm-project/recipes1 reaction · 2 comments
- Oct 5, 2026feat: upgrade vllm from 0.30.0 to 0.31.0 for CUDA and ROCm backends — gpustack/runner1 reaction · 2 comments
- Oct 2, 2026[Model] Add Nemotron 3.5 ASR transcription support — vllm-project/vllm1 reaction · 9 comments
- Sep 23, 2026feat: upgrade vllm from 0.29.0 to 0.30.0 for CUDA and ROCm backends — gpustack/runner1 reaction · 2 comments
- Sep 21, 2026feat(voice): add VoxCPM as a local TTS provider — moltis-org/moltis1 reaction · 1 comment
- Sep 3, 2026Add FastH3 VSA recipe — vllm-project/recipes1 reaction · 1 comment
- Jul 23, 2026[Bug]: HunyuanImage-3.0-Instruct (AR+DiT) startup hangs when waitting engine core process start on Ascend A3 — vllm-project/vllm-ascend1 reaction · 4 comments
- Jul 21, 2026[vLLM backend][Feature Request] Support vLLM-Omni for native audio/video input — triton-inference-server/server1 reaction · 3 comments
- Jul 9, 2026[RFC] Refactor experimental step execution support by reusing vLLM-Omni interfaces — verl-project/verl-omni1 reaction · 7 comments
- Jun 16, 2026Add user guide for v2v and transfer for vllm-omni — NVIDIA/cosmos1 reaction · 5 comments
- Jun 15, 2026[Quantization] Extend ModelOpt mixed precision and NVFP4 runtime formats — vllm-project/vllm1 reaction · 6 comments
- Jun 12, 2026[Draft]Support Megatron LoRA adapter export for rollout — verl-project/verl1 reaction · 1 comment
- Jun 12, 2026[Draft]Support Qwen3-Omni Megatron LoRA rollout — verl-project/verl-omni1 reaction · 1 comment
- Jun 11, 2026310p 部署 Qwen3-TTS — vllm-project/vllm-ascend1 reaction · 11 comments
- Jun 9, 2026docs(diffusion): add Qwen-Image v0.20→v0.22 H200 retro — hsliuustc0106/vllm-omni-cookbook1 reaction · 6 comments
- Jun 6, 2026[Kernel] FlashInfer FP8 scaled_mm: restore N-D output shape — vllm-project/vllm1 reaction · 3 comments
- May 22, 2026[feat]: Cosmos3 port (Tier-A scaffold; SSIM + Tier-B deferred pending weight publish) — hao-ai-lab/FastVideo1 reaction · 8 comments
- May 20, 2026[RFC] Q3 Road Map (welcome contributions) — verl-project/verl-omni1 reaction · 30 comments
- May 15, 2026Add extra_labelnames to PrometheusStatLogger for extensibility — vllm-project/vllm1 reaction · 3 comments
- May 14, 2026Mxfp4 support — yiliu30/vllm-omni-fork1 reaction
- Apr 20, 2026Custom pipeline class fix — knlnguyen1901/vllm-omni1 reaction
- Mar 18, 2026Add SoulX-Duplug turn-serving support — nemoramo/vllm-omni1 reaction · 1 comment
- Jan 12, 2026vLLM/vLLM-Omni integration — NAVER-Cloud-HyperCLOVA-X/OmniServe1 reaction
- Jan 9, 2026[Feature][Community]: Model Support: Add support for Bagel and HunyuanImage3.0 models in disaggregated setups. — JiusiServe/vllm-omni1 reaction
- Oct 6, 2026[Feature] Support Irodori-TTS Japanese-focused TTS in SGLang-Omni — sgl-project/sglang-omni0 reactions
- Aug 5, 2026[Feature]: Diffusion kv block manager — JiusiServe/vllm-omni0 reactions
- Aug 2, 2026[Bug]: Shared prompt_token_ids causes stale M-RoPE positions — vllm-project/vllm0 reactions · 1 comment
- Apr 1, 2026[RFC]: HunyuanImage DIT Fa3-fp8 quantization — JiusiServe/vllm-omni0 reactions · 1 comment
- Feb 27, 2026Support realtime API for audio streaming — EricLBuehler/mistral.rs0 reactions · 1 comment
On Hacker News · 1
- Dec 2, 2025VLLM-Omni: A framework for efficient model inference with Omni-modality models2 points · 1 comment
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- air-gapped/skills★ 6same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository29 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- 3d-animation-short-generatorSkill
- brand-promo-video-generatorSkill
- co-op-game-intro-generatorSkill
- h3-prompt-writingSkill
- handdrawn-live-video-generatorSkill
- minimalist-product-ad-generatorSkill
- music-video-subtitle-generatorSkill
- paper-collage-explainer-generatorSkill
- papercraft-stop-motion-explainerSkill
- vllm-omni-h3-character-replacementSkill
- add-diffusion-modelSkill
- add-omni-modelSkill
- add-recipeSkill
- add-tts-modelSkill
- diffusion-accuracy-testSkill
- diffusion-perf-optSkill
- diffusion-perf-testSkill
- dit-npu-fusion-analysisSkill
- find-simplificationsSkill
- omni-perf-ceilingSkill
- omni-perf-optSkill
- precheck-prSkill
- production-add-diffusion-modelSkill
- quantizationSkill
- review-prSkill
- vllm-omni-npu-model-runner-upgradeSkill
- vllm-omni-testSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 43
Read only the references the task needs.
- Very specific: line 67
Localize to a stage: Use stage ids in logs, --log-stats, and --print-stage.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
vLLM-Omni Multi-Stage Omni Models
Entry skill for multi-stage autoregressive omni models. Handle deployment, serving, debugging, and benchmarking here; hand off to the dedicated skill when the task belongs to one.
Mental model
A request flows through a stage graph. For Qwen3-Omni:
stage 0 thinker (LLM_AR) -> stage 1 talker (LLM_AR) -> stage 2 code2wav (LLM_GENERATION)
multimodal in, text out hidden states -> codec codes codes -> waveform- Pipeline (code,
models/<model>/pipeline.py) fixes the topology and the stage hand-off functions (model_executor/stage_input_processors/). - Deploy YAML (
vllm_omni/deploy/*.yaml) sets runtime: devices, memory, batch limits, CUDA graphs, connectors,async_chunk,session_mode,model_runner(v1 / MRv2). - Orchestrator (
vllm_omni/engine/orchestrator.py) routes each stage output to the next stage or the client; connectors move the data. async_chunkstreams partial outputs between stages; it determines first audio packet latency.
Task routing
| Task | Read |
| Understand code paths, find where something lives | references/architecture.md |
| Write or tune a deploy YAML, GPU layout, replicas | references/stage-config.md |
Launch a server, call APIs, offline Omni | references/serving.md |
| Hang, OOM, crash, stall, bad or broken audio | references/debugging.md |
| Benchmark, compare runs, accuracy evals, profiling | references/benchmarking.md |
| Qwen3-Omni or Qwen2.5-Omni specifics | references/models/qwen3-omni.md |
| MiniCPM-o specifics | references/models/minicpm-o.md |
Read only the references the task needs. For model-specific work, read the model file together with the task reference.
Hand off to dedicated skills
| Task | Skill |
| New thinker/talker/code2wav omni model | add-omni-model |
| Omni latency, throughput, streaming smoothness, or GPU-count optimization | omni-perf-opt |
| New diffusion model / productionizing one | add-diffusion-model, production-add-diffusion-model |
| Diffusion performance | diffusion-perf-opt |
| TTS-only model | add-tts-model |
| Quantization | quantization |
| Writing or running tests, CI wiring | vllm-omni-test |
Recipes under recipes/ | add-recipe |
| Self-check before opening a PR | precheck-pr |
| Reviewing a PR | review-pr |
| NPU model runner upgrade | vllm-omni-npu-model-runner-upgrade |
Working rules
- Resolve the effective config first. Identify the pipeline, the deploy YAML
actually loaded (default or
--deploy-config), itsbase_configchain, the platform section, and CLI overrides before reasoning about behavior. - Localize to a stage. Use stage ids in logs,
--log-stats, and--print-stage. Isolate the thinker with"modalities": ["text"]. - Change one variable at a time and keep the rest of the command identical.
- Check CUDA graphs both ways. Compare suspicious output against
--enforce-eager(or the stage'senforce_eager) before blaming model code. - Treat source as authoritative. Docs and design notes can lag the code;
cite paths and confirm field names in
vllm_omni/config/stage_config.pyand the model package. - Back claims with evidence. For any performance or correctness claim, record vLLM and vLLM-Omni commits, hardware, deploy YAML, and exact commands, and pair performance results with an accuracy check when numerics may change.
Reporting template
## Summary
[One or two sentences: outcome and root cause or result]
## Setup
- vllm-omni: <commit>, vllm: <commit>, hardware: <GPUs>
- Deploy: <yaml + overrides>
- Command(s): <serve / bench / test>
## Findings
- [Stage-attributed observation with numbers or log evidence]
## Next steps
- [Concrete action]Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

