omni-perf-ceiling
Skill7.8k starsChanged 3 days agoApache-2.0View on GitHub
Estimate idealized serving performance of a vLLM-Omni multi-stage omni model (Qwen3-Omni, MiniCPM-o, and similar thinker/talker/code2wav pipelines) and how much headroom the current implementation leaves. Use when asked for the lowest reachable AUDIO_RTF, TTFT or first-audio latency (AUDIO_TTFP) at a given concurrency, the maximum audio throughput on given hardware, whether a model "still has optimization room", or which stage bounds a target. Produces measured and calibrated roofline estimates with a per-stage gap ledger, on CUDA GPUs or Ascend NPUs. Use `omni-perf-opt` afterwards to implement optimizations.
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Who's talking about vllm-project/vllm-omni
- verl-project/verl“[RFC] Multi-modal Generation RL 2026Q2 Roadmap”GitHub · Mar 26, 2026 · 16 reactions
- vllm-project/vllm“[Model][VLM] Add Qwen2.5-Omni model support (end-to-end full support)”GitHub · Apr 9, 2025 · 11 reactions
- verl-project/verl“[RFC] Support Qwen-Image Flow-GRPO Training based on vLLM-Omni”GitHub · Dec 22, 2025 · 10 reactions
- verl-project/verl“[RFC] Support Diffusion Generative Models and Async Reward during Rollout”GitHub · Dec 17, 2025 · 8 reactions
- vllm-project/vllm“[Migration] Migrate GGUF quantization support to plugin”GitHub · Apr 12, 2026 · 7 reactions
The biggest 5 of 51 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 51
Talked about on GitHub and Hacker News (51 times).
On GitHub · 50
- Mar 26, 2026[RFC] Multi-modal Generation RL 2026Q2 Roadmap — verl-project/verl16 reactions · 21 comments
- Apr 9, 2025[Model][VLM] Add Qwen2.5-Omni model support (end-to-end full support) — vllm-project/vllm11 reactions · 11 comments
- Dec 22, 2025[RFC] Support Qwen-Image Flow-GRPO Training based on vLLM-Omni — verl-project/verl10 reactions · 8 comments
- Dec 17, 2025[RFC] Support Diffusion Generative Models and Async Reward during Rollout — verl-project/verl8 reactions · 3 comments
- Apr 12, 2026[Migration] Migrate GGUF quantization support to plugin — vllm-project/vllm7 reactions · 16 comments
45 more on GitHub
- Aug 14, 2026[RFC] Tracking v1 upgrade for verl-omni v0.3.0: remaining recipe coverage — verl-project/verl-omni5 reactions · 19 comments
- Apr 29, 2026在vLLM-Omni中支持SenseNova-U1 — OpenSenseNova/SenseNova-U14 reactions · 6 comments
- Dec 7, 2025RL post training for omni-modality models — ByteDance-Seed/VeOmni4 reactions · 2 comments
- Oct 16, 2026Add MOSS-Image Base and SFT recipe — vllm-project/recipes2 reactions · 2 comments
- Aug 3, 2026[Draft] [Bugfix] Clone safetensors tensors before H2D copy on Ascend NPU to fix DMA hang — vllm-project/vllm2 reactions · 3 comments
- Jun 17, 2026[ROCm][Perf] Fix Qwen3-vLLM audio encoder TP when heads are not divisible by TP size — vllm-project/vllm2 reactions · 14 comments
- Jun 16, 2026[RFC]: Qwen3-Omni RL Support on FSDP with vLLM-Omni rollout — verl-project/verl-omni2 reactions · 1 comment
- May 11, 2026[Metrics] Scope unregister_vllm_metrics() to strictly "vllm:" metrics — vllm-project/vllm2 reactions · 10 comments
- Apr 14, 2026[RFC]: Support Mooncake Based ECConnector for EPD — vllm-project/vllm2 reactions · 3 comments
- Apr 7, 2026Collaboration: Integrate VoxCPM into vllm-omni — a710128/nanovllm-voxcpm2 reactions · 4 comments
- Feb 25, 2026vllm-omni is supporting the model inference — bytedance/mammothmoda2 reactions · 3 comments
- Jan 10, 2026[Models] Add `SharedFusedMoE` support to Qwen3MoE — vllm-project/vllm2 reactions · 2 comments
- Jan 9, 2026[Feature]: RL Integration: Support for the veRL + veOmni + vLLM-Omni stack. — JiusiServe/vllm-omni2 reactions · 3 comments
- Dec 20, 2025[RFC]: Optional Hidden States Transfer in `KVConnectorBase_V1` to Eliminate Decoder Prefix-Prefill in P/D Disaggregation — vllm-project/vllm2 reactions · 2 comments
- Nov 12, 2026Add opt-in single RTX 4090 hardware profile — vllm-project/recipes1 reaction · 1 comment
- Nov 10, 2026Sync recipes from vllm-project/vllm-omni@d8751cd14a85 — vllm-project/recipes1 reaction · 2 comments
- Oct 29, 2026Sync recipes from vllm-project/vllm@9a0f9b45df86, vllm-project/vllm-omni@46766eb27f3e — vllm-project/recipes1 reaction · 2 comments
- Oct 5, 2026feat: upgrade vllm from 0.30.0 to 0.31.0 for CUDA and ROCm backends — gpustack/runner1 reaction · 2 comments
- Oct 2, 2026[Model] Add Nemotron 3.5 ASR transcription support — vllm-project/vllm1 reaction · 9 comments
- Sep 23, 2026feat: upgrade vllm from 0.29.0 to 0.30.0 for CUDA and ROCm backends — gpustack/runner1 reaction · 2 comments
- Sep 21, 2026feat(voice): add VoxCPM as a local TTS provider — moltis-org/moltis1 reaction · 1 comment
- Sep 3, 2026Add FastH3 VSA recipe — vllm-project/recipes1 reaction · 1 comment
- Jul 23, 2026[Bug]: HunyuanImage-3.0-Instruct (AR+DiT) startup hangs when waitting engine core process start on Ascend A3 — vllm-project/vllm-ascend1 reaction · 4 comments
- Jul 21, 2026[vLLM backend][Feature Request] Support vLLM-Omni for native audio/video input — triton-inference-server/server1 reaction · 3 comments
- Jul 9, 2026[RFC] Refactor experimental step execution support by reusing vLLM-Omni interfaces — verl-project/verl-omni1 reaction · 7 comments
- Jun 16, 2026Add user guide for v2v and transfer for vllm-omni — NVIDIA/cosmos1 reaction · 5 comments
- Jun 15, 2026[Quantization] Extend ModelOpt mixed precision and NVFP4 runtime formats — vllm-project/vllm1 reaction · 6 comments
- Jun 12, 2026[Draft]Support Megatron LoRA adapter export for rollout — verl-project/verl1 reaction · 1 comment
- Jun 12, 2026[Draft]Support Qwen3-Omni Megatron LoRA rollout — verl-project/verl-omni1 reaction · 1 comment
- Jun 11, 2026310p 部署 Qwen3-TTS — vllm-project/vllm-ascend1 reaction · 11 comments
- Jun 9, 2026docs(diffusion): add Qwen-Image v0.20→v0.22 H200 retro — hsliuustc0106/vllm-omni-cookbook1 reaction · 6 comments
- Jun 6, 2026[Kernel] FlashInfer FP8 scaled_mm: restore N-D output shape — vllm-project/vllm1 reaction · 3 comments
- May 22, 2026[feat]: Cosmos3 port (Tier-A scaffold; SSIM + Tier-B deferred pending weight publish) — hao-ai-lab/FastVideo1 reaction · 8 comments
- May 20, 2026[RFC] Q3 Road Map (welcome contributions) — verl-project/verl-omni1 reaction · 30 comments
- May 15, 2026Add extra_labelnames to PrometheusStatLogger for extensibility — vllm-project/vllm1 reaction · 3 comments
- May 14, 2026Mxfp4 support — yiliu30/vllm-omni-fork1 reaction
- Apr 20, 2026Custom pipeline class fix — knlnguyen1901/vllm-omni1 reaction
- Mar 18, 2026Add SoulX-Duplug turn-serving support — nemoramo/vllm-omni1 reaction · 1 comment
- Jan 12, 2026vLLM/vLLM-Omni integration — NAVER-Cloud-HyperCLOVA-X/OmniServe1 reaction
- Jan 9, 2026[Feature][Community]: Model Support: Add support for Bagel and HunyuanImage3.0 models in disaggregated setups. — JiusiServe/vllm-omni1 reaction
- Oct 6, 2026[Feature] Support Irodori-TTS Japanese-focused TTS in SGLang-Omni — sgl-project/sglang-omni0 reactions
- Aug 5, 2026[Feature]: Diffusion kv block manager — JiusiServe/vllm-omni0 reactions
- Aug 2, 2026[Bug]: Shared prompt_token_ids causes stale M-RoPE positions — vllm-project/vllm0 reactions · 1 comment
- Apr 1, 2026[RFC]: HunyuanImage DIT Fa3-fp8 quantization — JiusiServe/vllm-omni0 reactions · 1 comment
- Feb 27, 2026Support realtime API for audio streaming — EricLBuehler/mistral.rs0 reactions · 1 comment
On Hacker News · 1
- Dec 2, 2025VLLM-Omni: A framework for efficient model inference with Omni-modality models2 points · 1 comment
In this repository29 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- 3d-animation-short-generatorSkill
- brand-promo-video-generatorSkill
- co-op-game-intro-generatorSkill
- h3-prompt-writingSkill
- handdrawn-live-video-generatorSkill
- minimalist-product-ad-generatorSkill
- music-video-subtitle-generatorSkill
- paper-collage-explainer-generatorSkill
- papercraft-stop-motion-explainerSkill
- vllm-omni-h3-character-replacementSkill
- add-diffusion-modelSkill
- add-omni-modelSkill
- add-recipeSkill
- add-tts-modelSkill
- diffusion-accuracy-testSkill
- diffusion-perf-optSkill
- diffusion-perf-testSkill
- dit-npu-fusion-analysisSkill
- find-simplificationsSkill
- omni-perf-optSkill
- precheck-prSkill
- production-add-diffusion-modelSkill
- quantizationSkill
- review-prSkill
- vllm-omni-npu-model-runner-upgradeSkill
- vllm-omniSkill
- vllm-omni-testSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 5 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 40
Do not use one model for both.
- Very specific: line 71
Freeze the workload and commands: Use the omni-perf-opt measurement templates.
- Very specific: line 75
Check admission before measuring capacity: Every stage's max_num_seqs, session caps (max_sessions for duplex), and per-stage graph batch …
- Very specific: line 92
Build the ledger: Run python .claude/skills/omni-perf-ceiling/scripts/ceiling_report.py on the result JSON and bench log of each cell.
- Very specific: line 113
Compute calibrated estimates: Run scripts/measure_peaks.py on the target device for memory bandwidth, dense matmul throughput and …
- Distinctive: line 127
Write each term's formula with the numbers substituted so it can be re-checked, and state which inputs are measured and which analytic.
- Distinctive: line 140
Treat kernel efficiency, extra work, host, transfer and queueing as hypotheses.
- Distinctive: line 155
Do not quote a ceiling without the measured cell beside it on the same commit and hardware.
- Distinctive: line 187
Keep its estimate tied to its own concurrency and workload.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Omni Performance Ceiling
Answer "how fast can this get, and where is the remaining gap" with two numbers per target, not one:
- Measured: the shipped configuration on the target hardware.
- Calibrated estimate: reference work priced with device microbenchmarks. It is conditional on the workload, precision, cache and serial-path assumptions. It is neither a strict physical bound nor an achievable performance target.
measured - estimate is an unattributed gap, not guaranteed optimization
headroom. Report it per stage and require profiling before assigning a cause.
The legacy CLI --*-bound and JSON *_bound names remain for compatibility;
bound_kind=calibrated_estimate identifies their semantics.
First questions
Fill in what is not already known; the omni-perf-opt first questions apply.
In addition, fix:
- Target cells, for example
AUDIO_RTFat concurrency 1 and 64, and TTFT orAUDIO_TTFPat concurrency 1. - RTF definition: per-request mean (
mean_audio_rtf) or aggregate (sum of generation time over sum of audio). They differ under load. - Output mode: streaming chunks or a single final audio. Streaming adds first-chunk work; compare engines only within one mode.
- Device budget: one shared device, or one device per stage.
- Accelerator: CUDA GPU or Ascend NPU. On Ascend read references/npu-ceiling.md first; MPS and per-process SM attribution are not available there.
Two regimes
Concurrency 1 is a latency problem; high concurrency is a capacity problem. Do not use one model for both. Derivations: references/bound-models.md.
- Low concurrency: account for serial stage work and the gates before each stage starts. Stage wall time includes upstream waits, so it does not establish exclusive kernel time or exclude scheduling overhead.
- High concurrency: Little's law relates full request E2EL/audio duration to audio throughput while concurrency is held. Aggregate audio RTF ends at the last audio packet and excludes any request tail. Accumulate shared demand per resource (memory and each compute dtype), take the maximum, then combine with each stage's serial constraint. Do not sum durations of stages that can overlap via MPS or independent streams. Serial rows inside one stage may still be summed.
Fixed request work joins the matching resource before the maximum. The
aggregate shared estimate is C/N * max_q(D_q + W_q/mean(d)); D is streaming
resource demand and W is fixed demand. Different dtype constraints remain
separate unless a hardware resource mapping proves they share a serial pipe.
The fitted T_r = a + b*d_r split is descriptive and assumes common a and b.
Its coefficients do not identify causal fixed/streaming overhead. Workload
averages do not bound individual requests: the current report leaves mean
RTF unavailable, even at C1, because it lacks per-request work counts.
First-token and first-packet latency estimates describe the head of a. TTFT includes
the work before the Thinker's first token (encoders, prefill on uncached
tokens); TTFP adds every hand-off before the first audio packet (Thinker steps
the Talker needs, the first codec chunk, Code2Wav on that chunk). The estimate
has no queueing term, so judge it at concurrency 1.
Workflow
- Freeze the workload and commands. Use the
omni-perf-optmeasurement templates. Record commit, hardware, deploy YAML plus overlay, output mode, dataset, prompt count, and--extra-body. - Check admission before measuring capacity. Every stage's
max_num_seqs, session caps (max_sessionsfor duplex), and per-stage graph batch limits must be at least the target concurrency. Otherwise the high-concurrency cell measures queueing, not compute. Read startup logs for graph capture budgets and eager fallbacks at the target batch. Check KV capacity at the workload too (--kv-capacity): each stage'sGPU KV cache sizedivided by its largest request's tokens must be at least the target concurrency. - Measure the cells. Warm up with at least
Crequests, then run10 x C(minimum 128) prompts per cell with--print-stage,--save-result --save-detailed, and device utilisation sampling (nvidia-smi --query-gpu=utilization.gpu -lms 500, ornpu-smi info watchon Ascend). Keep profilers off. Usebenchmark_end_timeplusdurationto align samples; resultdateincludes postprocessing and is not a measurement timestamp. Keep client/server sampling clocks synchronized. - Build the ledger. Run
python .claude/skills/omni-perf-ceiling/scripts/ceiling_report.pyon the result JSON and bench log of each cell. It checks Little's law, derives aggregate audio RTF from complete clientrequest_audio_metrics(unavailable for legacy or partial data), per-stage time shares, GPU idle share, and the implied throughput floor. A Little's-law mismatch above 10% means the run did not hold concurrency (ramp, failures, or too few prompts); fix that before reading anything else. Pass--serieswith the cells of one configuration. Decreases in throughput, aggregate RTF, TTFT or TTFP are diagnostics to investigate; batching can change efficiency with C. - Attribute device time per stage. Pick one:
- one device per stage (
*_3gpu.yaml): per-device utilisation is the stage'sg_idirectly; - CUDA only: one shared device with MPS disabled
(
platforms.cuda.cuda_mps: false) andnvidia-smi pmon -s u, giving per-process SM share. Ascend has no per-process AI Core counter; - stage isolation microbenchmarks at the target batch. Experiment recipes: references/ceiling-experiments.md.
- one device per stage (
- Compute calibrated estimates. Run
scripts/measure_peaks.pyon the target device for memory bandwidth, dense matmul throughput and per-kernel launch latency (eager and graph replay). Runscripts/count_reference_work.pyon the checkpoint for work counts: it builds the reference transformers model on the meta device and counts each decoder's prefill and decode FLOPs per token, streamed weight bytes and KV bytes per token, plus Code2Wav FLOPs per chunk from the reference streaming path, and prints the bound flags. Give each stage three terms: compute (FLOPs per audio second), memory (weights plus KV read per decode step, or bytes moved), and launch (kernels per step times steps per audio second, at the stage's eager or graph mode). Split a stage into kernel groups with different bound classes, one flag each; the serial stage estimate issum over groups of max(compute, memory)andmax(device, launch)in its own process; the larger term names the stage's bound class. Write each term's formula with the numbers substituted so it can be re-checked, and state which inputs are measured and which analytic. Then add the per-request terms:--fixed-boundfor work every request adds (Thinker prefill on uncached tokens, encoders) and--fill-boundfor latency before a downstream stage can start (first Thinker step before the Talker, the first codec chunk before Code2Wav, and@outfor Code2Wav on that chunk). The report produces an aggregate RTF estimate, a TTFT estimate (terms gating stage 0) and a TTFP estimate (all serial gates). Mean RTF estimation requires per-request work and is unavailable. - Compare measurement and estimate. Pass
NAME@STAGEso the report joins the calibration to measured stage busy time. Estimate/measured is a diagnostic ratio; values above 100% can reflect calibration limits. Treat kernel efficiency, extra work, host, transfer and queueing as hypotheses. Use nsys/msprof timelines and kernel counters to identify the critical path. Slow kernels can cause queueing and downstream idle time. The fitted a/b split and an idle percentage cannot establish causality.--efficiency-target compute=F,memory=Fprojects shared resource demand at a hypothetical fraction of calibration. It does not measure achieved efficiency or attribute the residual.--profiled NAME=SECONDScompares a supplied implementation duration with its calibrated roofline. Neither comparison is a guarantee of an attainable speedup. - Report. Fill the template below. Hand the largest gap to
omni-perf-optas its first candidate.
Rules
- Do not quote a ceiling without the measured cell beside it on the same commit and hardware.
- Historical numbers are context, not evidence for the current revision.
- Label empirical rooflines as calibrated estimates. Strict hardware bounds require justified resource limits and unavoidable work; this tool does not certify them. Name perfect batching, cache, precision and serial assumptions.
- Calibrate on the serving device and host with
measure_peaks.py. A single GEMM shape or PyTorch tiny-op launch cost is not a universal hardware limit. - Shared devices use separate resource constraints; summing serial stage durations across independent streams overstates the shared requirement.
- Work counts (FLOPs, streamed weight bytes, KV bytes) come from the checkpoint's reference implementation, never from the vLLM-Omni code path being measured: counting the engine's own path folds its waste (padding, recomputed overlap, extra CFG rows) into the bound and hides the headroom. Take only FLOPs, weights and KV from the reference, not activation bytes; the reference does not fuse. Workload (tokens per audio second, context, uncached tokens) comes from the result JSON.
- Numerics and algorithm constants follow the reference unless the measured
deployment differs. Then price the deployment's dtype in the bound, and its
chunk size and steps as an algorithm lever, with the reference's setting
reported beside it.
A term that cannot be grounded is written as "not bounded:
<reason>", never estimated to fill the column. - Investigate ratios above 100%: calibration may be exceeded, or work coverage and attribution may differ. Do not declare physical impossibility.
- Report aggregate and mean RTF separately. Workload-average work supports only aggregate estimates. Mean RTF stays unavailable without per-request counts.
- TTFT/TTFP estimates omit queueing. At every C their residual may also include inefficient kernels or extra work; profile before assigning host/scheduler cost.
- Report the regime for each cell. Do not extrapolate a high-concurrency RTF from low-concurrency runs, or the reverse.
- A cell whose throughput is still rising has not established saturation. Keep its estimate tied to its own concurrency and workload.
- A ceiling belongs to one accelerator. Bounds scale with that device's measured peaks, so never carry a CUDA ceiling to Ascend or back.
Report template
## Ceiling summary
| Cell | Measured | Calibrated estimate | Estimate/measured | Binding term |
|------|----------|-------|----------|--------------|
| Aggregate RTF C1 | | | | |
| Mean RTF C1 | | unavailable: per-request work missing | | |
| Aggregate RTF C64 | | | | |
| Mean RTF C64 | | unavailable: per-request work missing | | |
| Audio throughput C64 (audio-s/s) | | | | |
| TTFT C1 (ms, mean) | | | | |
| AUDIO_TTFP C1 (ms, mean) | | | | |
## Setup
- vllm-omni <sha>, vllm <ver>, <GPU>, driver
- Deploy: <yaml> + <overlay>; session mode; output mode
- Workload: dataset, prompts, warmup, extra body
## Gap ledger (C<high>)
| Stage | Time share | g_i (GPU-s per audio-s) | Estimated g_i | Estimate/measured | Resource class | Launch estimate | Next lever |
|-------|-----------|-------------------------|-----------|----------|-------------|--------------|------------|
| Stage 0 | | | | | | | |
| Stage 1 | | | | | | | |
| Stage 2 | | | | | | | |
| GPU idle (cause unassigned) | | | | | | | |
## End-to-end split (T = a + b * d)
| Cell | Fitted a (ms) | Estimate a (ms) | Fitted b | Estimate b | Fixed share of RTF |
|------|-----------------|--------------|------------|---------|--------------------|
| C1 | | | | | |
| C<high> | | | | | |
## First-packet path (C1)
| Term | Gates | Latency estimate (ms) | Formula |
|------|-------|--------------------|---------|
| Thinker prefill | stage 0 (TTFT) | | |
| Thinker steps before Talker | stage 1 | | |
| First codec chunk | stage 2 | | |
| Code2Wav first chunk | first packet | | |
| **TTFP estimate** / measured | | | |
## Estimate arithmetic
[Per stage: each term's formula with numbers substituted, its source
(measured peak, checkpoint, result JSON, FLOP counter, analytic), and any
term left "not bounded: <reason>"]
## Checks
- Little's law: C / audio_throughput vs request_e2el_rtf, separately from audio RTF
- Admission: stage capacities and KV capacity >= C; graph fallbacks at C
- Sweep: investigate changes in throughput, aggregate RTF, TTFT and TTFP
- Attribution: trace evidence, or gap remains unattributed
- Interpretation: calibrated estimates, no guaranteed speedup
- Quality: <metric> within run-to-run variance, or not measured
## Not measured
[Bounds left analytic, cells skipped, and why]References
- references/bound-models.md: pipeline and capacity models, Little's law, roofline bounds for decode and compute stages
- references/ceiling-experiments.md: admission overlays, per-stage device attribution, peak measurement
- references/npu-ceiling.md: what the method keeps
on Ascend NPU, what MPS and pmon are replaced by, reservation and
npu-smisampling, and how the bound terms shift omni-perf-opt: measurement protocol, quality gate, optimization levers.claude/skills/vllm-omni/references/models/: per-model stage maps and deploy YAMLs
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

