omni-perf-opt
Skill7.8k starsChanged 3 days agoApache-2.0View on GitHub
Diagnose and optimize serving performance of multi-stage autoregressive omni models in vLLM-Omni (Qwen3-Omni, Qwen2.5-Omni, MiniCPM-o, and similar thinker/talker/code2wav pipelines). Use when asked to lower TTFT or first-audio latency (AUDIO_TTFP), keep AUDIO_RTF below 1 under load, raise throughput or concurrency, fit stages on fewer GPUs, fix choppy streaming audio, attribute latency to a stage or connector, tune deploy YAML performance knobs (batching, CUDA graphs, async_chunk codec chunking, MRv2, replicas), or plan and validate omni performance PRs. Diffusion workloads use `diffusion-perf-opt`.
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Who's talking about vllm-project/vllm-omni
- verl-project/verl“[RFC] Multi-modal Generation RL 2026Q2 Roadmap”GitHub · Mar 26, 2026 · 16 reactions
- vllm-project/vllm“[Model][VLM] Add Qwen2.5-Omni model support (end-to-end full support)”GitHub · Apr 9, 2025 · 11 reactions
- verl-project/verl“[RFC] Support Qwen-Image Flow-GRPO Training based on vLLM-Omni”GitHub · Dec 22, 2025 · 10 reactions
- verl-project/verl“[RFC] Support Diffusion Generative Models and Async Reward during Rollout”GitHub · Dec 17, 2025 · 8 reactions
- vllm-project/vllm“[Migration] Migrate GGUF quantization support to plugin”GitHub · Apr 12, 2026 · 7 reactions
The biggest 5 of 51 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 51
Talked about on GitHub and Hacker News (51 times).
On GitHub · 50
- Mar 26, 2026[RFC] Multi-modal Generation RL 2026Q2 Roadmap — verl-project/verl16 reactions · 21 comments
- Apr 9, 2025[Model][VLM] Add Qwen2.5-Omni model support (end-to-end full support) — vllm-project/vllm11 reactions · 11 comments
- Dec 22, 2025[RFC] Support Qwen-Image Flow-GRPO Training based on vLLM-Omni — verl-project/verl10 reactions · 8 comments
- Dec 17, 2025[RFC] Support Diffusion Generative Models and Async Reward during Rollout — verl-project/verl8 reactions · 3 comments
- Apr 12, 2026[Migration] Migrate GGUF quantization support to plugin — vllm-project/vllm7 reactions · 16 comments
45 more on GitHub
- Aug 14, 2026[RFC] Tracking v1 upgrade for verl-omni v0.3.0: remaining recipe coverage — verl-project/verl-omni5 reactions · 19 comments
- Apr 29, 2026在vLLM-Omni中支持SenseNova-U1 — OpenSenseNova/SenseNova-U14 reactions · 6 comments
- Dec 7, 2025RL post training for omni-modality models — ByteDance-Seed/VeOmni4 reactions · 2 comments
- Oct 16, 2026Add MOSS-Image Base and SFT recipe — vllm-project/recipes2 reactions · 2 comments
- Aug 3, 2026[Draft] [Bugfix] Clone safetensors tensors before H2D copy on Ascend NPU to fix DMA hang — vllm-project/vllm2 reactions · 3 comments
- Jun 17, 2026[ROCm][Perf] Fix Qwen3-vLLM audio encoder TP when heads are not divisible by TP size — vllm-project/vllm2 reactions · 14 comments
- Jun 16, 2026[RFC]: Qwen3-Omni RL Support on FSDP with vLLM-Omni rollout — verl-project/verl-omni2 reactions · 1 comment
- May 11, 2026[Metrics] Scope unregister_vllm_metrics() to strictly "vllm:" metrics — vllm-project/vllm2 reactions · 10 comments
- Apr 14, 2026[RFC]: Support Mooncake Based ECConnector for EPD — vllm-project/vllm2 reactions · 3 comments
- Apr 7, 2026Collaboration: Integrate VoxCPM into vllm-omni — a710128/nanovllm-voxcpm2 reactions · 4 comments
- Feb 25, 2026vllm-omni is supporting the model inference — bytedance/mammothmoda2 reactions · 3 comments
- Jan 10, 2026[Models] Add `SharedFusedMoE` support to Qwen3MoE — vllm-project/vllm2 reactions · 2 comments
- Jan 9, 2026[Feature]: RL Integration: Support for the veRL + veOmni + vLLM-Omni stack. — JiusiServe/vllm-omni2 reactions · 3 comments
- Dec 20, 2025[RFC]: Optional Hidden States Transfer in `KVConnectorBase_V1` to Eliminate Decoder Prefix-Prefill in P/D Disaggregation — vllm-project/vllm2 reactions · 2 comments
- Nov 12, 2026Add opt-in single RTX 4090 hardware profile — vllm-project/recipes1 reaction · 1 comment
- Nov 10, 2026Sync recipes from vllm-project/vllm-omni@d8751cd14a85 — vllm-project/recipes1 reaction · 2 comments
- Oct 29, 2026Sync recipes from vllm-project/vllm@9a0f9b45df86, vllm-project/vllm-omni@46766eb27f3e — vllm-project/recipes1 reaction · 2 comments
- Oct 5, 2026feat: upgrade vllm from 0.30.0 to 0.31.0 for CUDA and ROCm backends — gpustack/runner1 reaction · 2 comments
- Oct 2, 2026[Model] Add Nemotron 3.5 ASR transcription support — vllm-project/vllm1 reaction · 9 comments
- Sep 23, 2026feat: upgrade vllm from 0.29.0 to 0.30.0 for CUDA and ROCm backends — gpustack/runner1 reaction · 2 comments
- Sep 21, 2026feat(voice): add VoxCPM as a local TTS provider — moltis-org/moltis1 reaction · 1 comment
- Sep 3, 2026Add FastH3 VSA recipe — vllm-project/recipes1 reaction · 1 comment
- Jul 23, 2026[Bug]: HunyuanImage-3.0-Instruct (AR+DiT) startup hangs when waitting engine core process start on Ascend A3 — vllm-project/vllm-ascend1 reaction · 4 comments
- Jul 21, 2026[vLLM backend][Feature Request] Support vLLM-Omni for native audio/video input — triton-inference-server/server1 reaction · 3 comments
- Jul 9, 2026[RFC] Refactor experimental step execution support by reusing vLLM-Omni interfaces — verl-project/verl-omni1 reaction · 7 comments
- Jun 16, 2026Add user guide for v2v and transfer for vllm-omni — NVIDIA/cosmos1 reaction · 5 comments
- Jun 15, 2026[Quantization] Extend ModelOpt mixed precision and NVFP4 runtime formats — vllm-project/vllm1 reaction · 6 comments
- Jun 12, 2026[Draft]Support Megatron LoRA adapter export for rollout — verl-project/verl1 reaction · 1 comment
- Jun 12, 2026[Draft]Support Qwen3-Omni Megatron LoRA rollout — verl-project/verl-omni1 reaction · 1 comment
- Jun 11, 2026310p 部署 Qwen3-TTS — vllm-project/vllm-ascend1 reaction · 11 comments
- Jun 9, 2026docs(diffusion): add Qwen-Image v0.20→v0.22 H200 retro — hsliuustc0106/vllm-omni-cookbook1 reaction · 6 comments
- Jun 6, 2026[Kernel] FlashInfer FP8 scaled_mm: restore N-D output shape — vllm-project/vllm1 reaction · 3 comments
- May 22, 2026[feat]: Cosmos3 port (Tier-A scaffold; SSIM + Tier-B deferred pending weight publish) — hao-ai-lab/FastVideo1 reaction · 8 comments
- May 20, 2026[RFC] Q3 Road Map (welcome contributions) — verl-project/verl-omni1 reaction · 30 comments
- May 15, 2026Add extra_labelnames to PrometheusStatLogger for extensibility — vllm-project/vllm1 reaction · 3 comments
- May 14, 2026Mxfp4 support — yiliu30/vllm-omni-fork1 reaction
- Apr 20, 2026Custom pipeline class fix — knlnguyen1901/vllm-omni1 reaction
- Mar 18, 2026Add SoulX-Duplug turn-serving support — nemoramo/vllm-omni1 reaction · 1 comment
- Jan 12, 2026vLLM/vLLM-Omni integration — NAVER-Cloud-HyperCLOVA-X/OmniServe1 reaction
- Jan 9, 2026[Feature][Community]: Model Support: Add support for Bagel and HunyuanImage3.0 models in disaggregated setups. — JiusiServe/vllm-omni1 reaction
- Oct 6, 2026[Feature] Support Irodori-TTS Japanese-focused TTS in SGLang-Omni — sgl-project/sglang-omni0 reactions
- Aug 5, 2026[Feature]: Diffusion kv block manager — JiusiServe/vllm-omni0 reactions
- Aug 2, 2026[Bug]: Shared prompt_token_ids causes stale M-RoPE positions — vllm-project/vllm0 reactions · 1 comment
- Apr 1, 2026[RFC]: HunyuanImage DIT Fa3-fp8 quantization — JiusiServe/vllm-omni0 reactions · 1 comment
- Feb 27, 2026Support realtime API for audio streaming — EricLBuehler/mistral.rs0 reactions · 1 comment
On Hacker News · 1
- Dec 2, 2025VLLM-Omni: A framework for efficient model inference with Omni-modality models2 points · 1 comment
In this repository29 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- 3d-animation-short-generatorSkill
- brand-promo-video-generatorSkill
- co-op-game-intro-generatorSkill
- h3-prompt-writingSkill
- handdrawn-live-video-generatorSkill
- minimalist-product-ad-generatorSkill
- music-video-subtitle-generatorSkill
- paper-collage-explainer-generatorSkill
- papercraft-stop-motion-explainerSkill
- vllm-omni-h3-character-replacementSkill
- add-diffusion-modelSkill
- add-omni-modelSkill
- add-recipeSkill
- add-tts-modelSkill
- diffusion-accuracy-testSkill
- diffusion-perf-optSkill
- diffusion-perf-testSkill
- dit-npu-fusion-analysisSkill
- find-simplificationsSkill
- omni-perf-ceilingSkill
- precheck-prSkill
- production-add-diffusion-modelSkill
- quantizationSkill
- review-prSkill
- vllm-omni-npu-model-runner-upgradeSkill
- vllm-omniSkill
- vllm-omni-testSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 8 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈8
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 8
Run a disciplined loop: freeze commands, measure a low-overhead baseline, attribute the gap to one stage or hop, change one lever, and …
- Distinctive: line 15
Ask for anything not already known:
- Distinctive: line 34
Freeze three commands: Write them as scripts with every variable explicit:
- Distinctive: line 44
Collect a real baseline: No torch profiler, no orchestrator monitor.
- Distinctive: line 55
Pick a lever: Use references/optimization-playbook.md.
- Very specific: line 70
Keep operator/shape traces and host-stack traces separate.
- Distinctive: line 91
Every optimization needs correctness evidence on the same workload, including "math-preserving" changes.
- Distinctive: line 132
When the talker saturates, add talker replicas before raising thinker batch limits.
- Distinctive: line 152
Never report profiler-enabled latency as a result.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Omni Model Performance Optimization
Run a disciplined loop: freeze commands, measure a low-overhead baseline, attribute the gap to one stage or hop, change one lever, and prove the gain without losing quality. Profiler traces diagnose; they never supply the final latency number.
First questions
Ask for anything not already known:
- Hardware: accelerator model and count, interconnect, device sharing, and backend software stack (for Ascend: CANN, torch-npu, and vllm-ascend).
- Model, checkpoint, vLLM and vLLM-Omni commits, and the deploy YAML actually
loaded (default or
--deploy-config, itsbase_configchain, platform section,--stage-overrides). - Session mode:
turn(HTTP) orduplex(realtime). They have different defaults and different MRv2 support. - Workload: input modalities and sizes, output modalities (text only or text plus audio), output length, concurrency or request rate, streaming or not.
- Target: TTFT, AUDIO_TTFP, AUDIO_RTF, audio continuity, throughput at a given concurrency, GPU count, or memory. Name one primary metric and its percentile (mean, P90, P99).
- Quality tolerance: bitwise, within run-to-run variance, or a stated accuracy and WER budget. Default to quality-preserving.
Workflow
- Freeze three commands. Write them as scripts with every variable
explicit:
- server:
vllm serve <model> --omni --deploy-config ... [--stage-overrides ...] - single request: one fixed
/v1/chat/completions(or/v1/audio/speech,/v1/realtime) payload with fixed media, seed, andmodalities - benchmark:
vllm bench serve --omni ...with fixed dataset, prompt count, concurrency or rate,--extra-body,--percentile-metrics,--print-stage,--save-resultTemplates: references/measurement-protocol.md.
- server:
- Collect a real baseline. No torch profiler, no orchestrator monitor.
Warm up (graph capture and first-request compilation distort small runs).
Sweep concurrency at 1, a middle value, and the target. Record mean and
P99 TTFT, AUDIO_TTFP, AUDIO_RTF, E2EL, throughput, failures, and
audio_continuity_ok_rate. Run the accuracy baseline on the same build. - Attribute the gap to a stage or hop. Re-run with
--log-stats(per-stage wall time,stage_gen_time_ms, connectortx_time_ms,rx_decode_time_ms,in_flight_time_ms) and--print-stage. Classify with the bottleneck table below before touching code. - Pick a lever. Use references/optimization-playbook.md. Try configuration-only levers before code changes, and existing model options before new kernels.
- A/B one variable at a time. Same hardware, commits, dataset, seed,
prompt count, concurrency, and
--extra-body. When the change is behind a switch, run every column on the same head and toggle the switch. Stacked changes get one column each. - Pass the quality gate. See below. A failed or inconclusive gate blocks a ready claim.
- Trace only after narrowing. Profile the stage you already suspect:
per-stage
profiler_configthrough--stage-overrides, warm up, thenPOST /start_profileandPOST /stop_profilewith{"stages": [...]}. Keep operator/shape traces and host-stack traces separate. For timelines, reuse.claude/skills/diffusion-perf-opt/scripts/trace_analyzer.py. - Write the plan. Classify candidates P0/P1/P2, tie each to evidence, and let the user choose before implementing anything above P0.
Bottleneck table
| Evidence | Likely bottleneck | First levers |
| High TTFT, thinker prefill dominates | Thinker prefill or multimodal encoding | Encoder CUDA graphs, max_num_batched_tokens, thinker TP, dedicated thinker GPU |
| Good TTFT, high AUDIO_TTFP | Talker first chunk or code2wav first decode | initial_codec_chunk_frames, MRv2 talker_first_audio + codec_chunk_ramp, first-chunk express |
| AUDIO_RTF ≥ 1 as concurrency rises | Talker or code2wav saturated | Stage max_num_seqs, CUDA graphs, talker replicas, code2wav graph batch sizes |
Good mean RTF, low audio_continuity_ok_rate | Bursty chunk delivery | codec_chunk_frames, codec_left_context_frames, first-chunk express slack |
High connector in_flight_time_ms or rx_decode_time_ms | Transport or payload size | Connector type, payload trimming, co-location |
| High latency while GPUs idle | Orchestrator loop or queue backlog | --enable-orch-monitor, event-driven orchestrator, replica balance |
| OOM or cannot co-locate | Memory split | gpu_memory_utilization, kv_cache_memory_bytes, graph budgets, MPS |
| Client time far above server time | Client, streaming encode, or network | Response format, chunk size, bench client placement |
Quality gate
Every optimization needs correctness evidence on the same workload, including "math-preserving" changes.
- Text: thinker greedy output unchanged, or accuracy (Daily-Omni, Video-MME) within baseline run-to-run variance.
- Speech: Seed-TTS WER (
--seed-tts-wer-eval) within variance; for bitwise-claimed changes, identical codec ids and waveform for a fixed seed. - Streaming: same duration, no truncation or repetition, and
audio_continuity_ok_ratenot worse. - Concurrency: per-request outputs at concurrency 2+ equal single-request outputs.
- Graphs: graph mode matches a control with outer and model-internal graphs explicitly disabled for touched stages; verify the effective switches.
- Precision changes (TF32, fused kernels, cuDNN algorithm search, short KV, quantization) must state that rounding changes and show the metric delta.
Priority rules
- P0: measurement fixes and configuration-only changes with low risk (warmup, frozen commands, batch limits, placement, replicas, existing opt-in deploy profiles, chunk sizes).
- P1: contained code changes (graph capture for a new stage path, host sync removal, payload trimming, scheduler batching, fused existing ops).
- P2: numerics-changing or invasive work (new custom kernels, TF32 or FP8 paths, short-KV attention, algorithm changes, MRv2 enablement for a new model).
Present the plan as a table (priority, stage, lever, evidence, expected gain, validation, quality risk), then ask which item to implement. Proceed alone only on P0 measurement or instrumentation fixes.
Omni heuristics
async_chunktrades total compute for first-audio latency. It cuts AUDIO_TTFP sharply but can raise E2EL and RTF; offline batch jobs may be faster without it. Judge it against the stated primary metric.- Streaming output and
async_chunkdepend on each other: withoutasync_chunkstreaming cannot start early, and without streaming the client does not see the earlier first packet. - For the three-stage CUDA examples, start with a dedicated thinker GPU; verify stage costs before applying this placement to a different topology. When the talker saturates, add talker replicas before raising thinker batch limits.
- Prefix caching is stage- and profile-specific. Default
qwen3_omni_moe.yamluses MRv2 on CUDA with Thinker prefix caching on; Talker and Code2Wav keep it off. The specializedqwen3_omni_moe_mrv2.yamloverlay and non-CUDA platform sections disable Thinker caching. Talker first-frame and fused MTP fast paths require caching off on the Talker, independently of Thinker caching. MiniCPM-o's default duplex deploy keeps caching off; its CUDA turn profile enables it forllm(stage 0), while the turn MRv2 overlay disables it again. Inspect the effective YAML before changing it. - MRv2 output is not bitwise identical to V1. Re-measure MRv2 numbers after any runner change; MiniCPM-o's earlier MRv2 numbers were invalidated. MRv2 downstream stages support turn mode; duplex needs its own profile and its own measurement.
- Stages sharing one GPU under MRv2 should run under NVIDIA MPS.
- Graph caches have fixed budgets. When a budget fills, uncached shapes run eagerly: correct but slower. Check startup capture logs after changing batch sizes or graph caps.
- The orchestrator is event-driven by default (
VLLM_OMNI_EVENT_DRIVEN_ORCHunset). Orchestrator monitor counts are not comparable across loop modes. - Never report profiler-enabled latency as a result.
Report template
## Summary
[Outcome, primary metric delta, and the stage it came from]
## Setup
- vllm-omni <sha>, vllm <sha>, <accelerator x N>, driver/backend versions
- Deploy: <yaml> + <overrides>; session mode <turn|duplex>
- Commands: serve / request / bench (scripts or exact lines)
## Results (warmup <w>, measured <n> requests, concurrency <c>)
| Metric | Baseline | Candidate | Delta |
|--------|----------|-----------|-------|
| TTFT mean / P99 (ms) | | | |
| AUDIO_TTFP mean / P99 (ms) | | | |
| AUDIO_RTF mean / P99 | | | |
| Throughput (req/s or audio-s/s) | | | |
| audio_continuity_ok_rate | | | |
| Failures | | | |
## Stage attribution
| Row | Baseline | Candidate |
|-----|----------|-----------|
| Stage 0 gen (ms) | | |
| Stage 1 gen (ms) | | |
| Stage 2 gen (ms) | | |
| Connector in-flight (ms) | | |
Isolation: [metric that stayed unchanged, isolating the gain to <stage>]
## Quality
[Accuracy / WER / parity result versus baseline variance]
## Next steps
[Candidate table or chosen item]References
- references/measurement-protocol.md: command templates, metric definitions, diagnostics, A/B rules
- references/optimization-playbook.md: levers by layer with knobs, evidence, and validation
.claude/skills/vllm-omni/references/models/qwen3-omni.md,.claude/skills/vllm-omni/references/models/minicpm-o.md: model profilesdocs/design/qwen3_omni_tts_performance_optimization.md,docs/design/qwen3_omni_mrv2_performance.md,docs/design/minicpm_o45_mrv2_performance.md: design evidencedocs/contributing/profiling.md(sections on Omni AR/TTS stages and the orchestrator monitor),docs/contributing/metrics.md.claude/skills/review-pr/references/checks/perf-verification.md: what reviewers will ask foromni-perf-ceiling: reachable ceiling and per-stage headroom before choosing leversquantizationskill for precision work;vllm-omni-testfor perf JSON rows undertests/dfx/perf/tests/
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

