modelopt-benchmark-model-kernels
Skill5.6k starsChanged todayApache-2.0View on GitHub
Inspect Hugging Face decoder layers on meta tensors and plan or run per-rank BF16, FP8, and NVFP4 GEMM or fused-MoE microbenchmarks with the bundled scripts and a local FlashInfer checkout. Use when choosing a model, GPU, TP, EP, or M/token-concurrency sweep; deriving common fused QKV and gate/up shapes without loading checkpoint weights; or using FlashInfer benchmark utilities. Do not use for end-to-end server throughput or request latency.
1 other project uses a version of this.
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Most recently taken up by
- NVIDIA/Model-Optimizerword for word, first seen Nov 15, 2026
Who's talking about NVIDIA/TensorRT-Model-Optimizer
- NVIDIA/Model-Optimizer“Replace ONNX simplification package from onnxsim to onnxslim”GitHub · Oct 28, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“Fix FLOPs calculation”GitHub · Sep 29, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case casting input directly to output”GitHub · Sep 9, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case with cast producing network output”GitHub · Sep 8, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“"Unknown quantization method" when loading nvfp4 checkpoint with vLLM”GitHub · Aug 19, 2025 · 1 reactions
The biggest 5 of 45 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 45
Talked about on GitHub (45 times).
On GitHub · 45
- Oct 28, 2025Replace ONNX simplification package from onnxsim to onnxslim — NVIDIA/Model-Optimizer1 reaction · 23 comments
- Sep 29, 2025Fix FLOPs calculation — NVIDIA/Model-Optimizer1 reaction · 11 comments
- Sep 9, 2025[Autocast] Fix edge case casting input directly to output — NVIDIA/Model-Optimizer1 reaction · 10 comments
- Sep 8, 2025[Autocast] Fix edge case with cast producing network output — NVIDIA/Model-Optimizer1 reaction · 6 comments
- Aug 19, 2025"Unknown quantization method" when loading nvfp4 checkpoint with vLLM — NVIDIA/Model-Optimizer1 reaction · 3 comments
40 more on GitHub
- Mar 30, 2025[Quantization] Add TRT-ModelOpt as a Backend — huggingface/diffusers1 reaction · 50 comments
- Mar 11, 2025✨[Feature] Supporting Blackwell FP4 — pytorch/TensorRT1 reaction
- Sep 16, 2026推論最適化ツール・OSS — warabinbin/papers0 reactions
- Aug 10, 2026fold_weight crashes with AttributeError: 'NoneType' object has no attribute 'data' on Megatron models with tied word embeddings — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 5, 2025Kimi-k2 calib+export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 4, 2025Quantization — HashCitrine/sample-to-make-llm-service0 reactions
- Dec 4, 2025Diffusion Rename [TRT MO -> MO] — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Dec 3, 2025[5704162] Create a copy to avoid leaking ProcessGroup into state dict — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Dec 2, 2025fix: Fix auto quantize multi-gpu tests — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Dec 2, 2025qwen3_8b eagle3 model train and vllm ckpt export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 27, 2025Add all example e2e tests for github PR merge / nightly — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 26, 2025[Bug Fix]: Correct block_size in NVFP4_MLP_WEIGHT_ONLY_CFG to support vLLM inference — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 26, 2025fix: relax sympy version — inisis/OnnxSlim0 reactions · 1 comment
- Nov 24, 2025[None][doc] Deprecate --export_fmt flag when using ModelOpt — NVIDIA/TensorRT-LLM0 reactions · 4 comments
- Nov 24, 2025[Help wanted] CNN QAT: How to correctly export ONNX and use DeviceModel for inference — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 22, 2025fix: issues related to TensorRT-Model-Optimizer — inisis/OnnxSlim0 reactions · 2 comments
- Nov 20, 2025[2/N] Added KDLoss based AutoQuantize — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 20, 2025[1/N] Refactored AutoQuantizeSearcher to _AutoQuantizeBaseSearcher & AutoQuantizeGradientSearcher; seperated quant modules and score modules — NVIDIA/Model-Optimizer0 reactions · 11 comments
- Nov 17, 2025failed to quant the Qwen3-VL-30B-A3B — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 13, 2025Support Wan2.2 t2v diffusers quantization — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 3, 2025PyTorch geometric quantization support — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 3, 2025Failed to save quantized model if some tensors offload to CPU — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Oct 28, 2025[OMNIML-2917] export layer config using actual prefix instead of hard… — NVIDIA/Model-Optimizer0 reactions · 9 comments
- Oct 27, 2025Add convert_llama3_config_to_decilm_config + unit test — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025CodeRabbit Generated Unit Tests: Add GPU tests for Megatron quantization; review/refactor unit tests — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025📝 Add docstrings to `kaix/kvcache_mcore` — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 15, 2025FP8 Block quantize onnx export support — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Sep 15, 2025add end_process in deepseek ptq — NVIDIA/Model-Optimizer0 reactions · 10 comments
- Jul 13, 2025feat: Simplify and Improve Whisper Example — NVIDIA/TensorRT-LLM0 reactions · 3 comments
- Jul 3, 2025[future request] Support more quant for torch backend — NVIDIA/TensorRT-LLM0 reactions · 2 comments
- Jul 3, 2025🐛 [Bug] examples/dynamo/torch_compile_gpt2.py run failed — pytorch/TensorRT0 reactions · 2 comments
- Jun 15, 2025INT8 to FP8 scale conversion — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Jun 13, 2025Fix speculative decoding example — NVIDIA/Model-Optimizer0 reactions · 6 comments
- May 21, 2025`get_modelike_from_algo_cfg` doesn't accept `QuantizeAlgorithmConfig`, but it's typing says it should — NVIDIA/Model-Optimizer0 reactions · 3 comments
- May 17, 2025Decode latency of FP4 LLM is higher than expected — NVIDIA/TensorRT-LLM0 reactions · 1 comment
- May 7, 2025Error in torch_compile.ipynb — Project-MONAI/tutorials0 reactions · 1 comment
- May 6, 2025当我使用增量训练的时候,只改变修改一个feature的zch_size的大小,其他所有配置都不改变,会出错。 — alibaba/TorchEasyRec0 reactions
- Apr 30, 2025INT8 quantization for HiFi-GAN vocoder -- performance issue — NVIDIA/TensorRT0 reactions · 7 comments
- Mar 24, 2025bert model split into many layers after int8 quantization — NVIDIA/TensorRT0 reactions · 13 comments
- Sep 8, 2024urlopen error retrieval incomplete in modules/autoencoder_mednist.ipynb — Project-MONAI/tutorials0 reactions
Other versions1 project uses a version of this.
Word for word · 1
- NVIDIA/Model-Optimizer★ 5.6k100% same
Edited · 0
None found.
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository33 other files it gives its agents.
- AGENTS.mdAGENTS.md
- tools/debugger/CLAUDE.mdCLAUDE.md
- tools/launcher/CLAUDE.mdCLAUDE.md
- accessing-mlflowSkill
- benchmark-model-kernelsSkill
- commonSkill
- compare-resultsSkill
- day0-releaseSkill
- debugSkill
- deploymentSkill
- evaluationSkill
- launching-evalsSkill
- modelopt-accessing-mlflowSkill
- modelopt-compare-resultsSkill
- modelopt-day0-releaseSkill
- modelopt-debugSkill
- modelopt-deploymentSkill
- modelopt-evaluationSkill
- modelopt-launching-evalsSkill
- modelopt-monitorSkill
- modelopt-ptqSkill
- modelopt-qadSkill
- modelopt-quant-recipe-searchSkill
- modelopt-release-cherry-pickSkill
- modelopt-run-agentxSkill
- modelopt-speculative-decodingSkill
- monitorSkill
- ptqSkill
- qadSkill
- quant-recipe-searchSkill
- release-cherry-pickSkill
- run-agentxSkill
- speculative-decodingSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 3 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 20
Ask exactly one unresolved decision per message with two or three concrete options, state the default, and wait for the answer.
- Very specific: line 71
Run a short plumbing check first (append --dry_run_iters 1 --num_iters 3 --no_autotune, throwaway workdir), especially after changing …
- Distinctive: line 79
Do not restate, re-derive, or override these — the code enforces them:
- Distinctive: line 103
Do not benchmark a padded shape to dodge these limits and call it vLLM-equivalent.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Benchmark Model Kernels
Plan a single-GPU microbenchmark for the per-rank shapes of an intended deployment. The scripts never load checkpoint weights, launch distributed workers, or measure collectives, serving throughput, or request latency.
Interview, one decision per message
Ask exactly one unresolved decision per message with two or three concrete options, state the default, and wait for the answer. Recommend an option only when it is substantively better. Skip decisions already answered. Follow this order:
- Model — a local directory, a
config.json, or a Hub ID (equivalent; no default). The script builds the model on meta tensors from configuration only. Do not enable--trust-remote-codewithout explicit approval; pin--revision <commit>when it is needed. - TP and EP — accept both directly, or derive them from the intended
deployment's GPU model and count and confirm. Defaults:
TP=1,EP=1. Derive parallelism from the intended deployment, never from the GPU that runs the microbenchmark. The script validates every sharding rule (divisibility, GQA replication, expert partitioning) and errors loudly. - M sweep — balanced default (recommended):
1 8 64 512; decode-focused:1 4 16 32; throughput-focused:64 256 1024 4096. M is roughly the tokens scheduled per step (decode: active sequences), not endpoint concurrency. With EP, an MoE row models one rank's share of the global batch: a global batch of B tokens corresponds to the column M = B/EP, so do not compare different EP values at the same M. Shape preview — always run this before anything else; no FlashInfer or GPU needed:
python "$SKILL_DIR/scripts/benchmark_model.py" <model> \ --tp <tp> --ep <ep> --ms <m1> <m2> ... --print_onlyReview the printed shapes and the MoE tuple with the user.
# unsupported:lines mean the list is partial and the script exits nonzero — handle those via Manual supplements below.- FlashInfer checkout and GPU — the full benchmark needs a FlashInfer
source checkout containing
benchmarks/flashinfer_benchmark.py; the installed wheel alone is not enough. Prefer a clean checkout matching the installedflashinferversion; ask before cloning or installing anything. On the benchmark machine, checknvidia-smiand package versions, verify the target GPU is idle (concurrent work on the same GPU skews timings), and verify CUPTI timing with a tinybench_gpu_time(..., enable_cupti=True)probe — a warning that falls back to CUDA events is a failure (thecupti-python/nvidia-cuda-cuptipackages must match PyTorch's CUDA major). Ask for a GPU index only when several are visible. Pick a fresh workdir and state it. Full benchmark:
CUDA_VISIBLE_DEVICES=<gpu-index> \ python "$SKILL_DIR/scripts/benchmark_model.py" <model> \ --tp <tp> --ep <ep> --ms <m1> <m2> ... \ --flashinfer_repo <flashinfer-repo> --workdir <workdir>Run a short plumbing check first (append
--dry_run_iters 1 --num_iters 3 --no_autotune, throwaway workdir), especially after changing FlashInfer versions or shape logic; never present it as a performance result. Then run with defaults. The first fused-MoE build or autotune can take several minutes; its cache is reused.
Rules the scripts own
Do not restate, re-derive, or override these — the code enforces them:
benchmark_model.pyderives--nks(asN,K,module-nametriples) and all--moe_*arguments; overrides are rejected.- Row labels are logical shapes; the runner applies vLLM's physical padding. Report both shapes whenever they differ.
- A failed case writes FlashInfer's error message (or a pointer to
driver.log) into itscombined_results.csvcell, and the command exits nonzero after the table is written. Never present a partial table as a successful benchmark; readdriver.logbefore rerunning anything.
Known backend and driver limits
mm_fp4TensorRT-LLM needsN % 128 == 0(shuffled weight layout): its cell reports no result row, or on stock 0.6.x drivers an empty assertion that fails everymm_fp4backend for that shape.mm_fp8(trtllm_low_latency) needsK % 128 == 0.- MoE rows cover FlashInfer's CUTLASS fused MoE, the trtllm-gen NVFP4 and
per-tensor FP8 MoE, and the CuteDSL NVFP4 MoE. The CuteDSL MoE row appears
only for Swiglu models (the kernel supports nothing else); the
cutedslandtrtllmbackends require recent GPUs and report per-case errors elsewhere. - Gated NVFP4 CUTLASS MoE with
2F % 128 != 0per rank fails — vLLM raises instead of padding — often as aCUDA error: misaligned address. Prefer EP over TP for the experts to keep the per-rank width legal. - Do not benchmark a padded shape to dodge these limits and call it vLLM-equivalent; report the limit instead.
Manual supplements
For each # unsupported: layout from the preview: inspect that module's
forward path and the intended runtime's TP/EP sharding, derive the per-rank
shape (never guess sharding from a weight shape alone), and benchmark only the
missing shape:
CUDA_VISIBLE_DEVICES=<gpu-index> \
python "$SKILL_DIR/scripts/benchmark_via_builtin.py" \
--flashinfer_repo <flashinfer-repo> --ms <m1> <m2> ... \
--nks <n>,<k>,<name> --workdir <workdir>Label these rows as manual supplements; this is not a substitute for running
benchmark_model.py first. Shell-quote user-supplied paths.
Report
Report the command, GPU, versions, TP/EP, M values, shapes (logical and
physical where they differ), warnings, and the artifacts: testlist.txt,
driver.log (full driver output), builtin_results.csv (milliseconds,
success-only), and combined_results.csv (long form, microseconds: columns module_name,
M, N, K, backend, with_quant, runtime in GEMM and MoE sections; modules
fused into one GEMM are joined with | in module_name, while distinct
same-shape modules appear as duplicated rows sharing one measurement; MoE
rows keep the H= F= E= top_k= parameter line and leave N/K empty). In quantization-recipe terms:
bf16 rows are the unquantized W16A16 baseline, fp8 rows are per-tensor
W8A8, and nvfp4 rows are W4A4. Plain quantized rows time the kernel with
pre-quantized activations; *_with_quant rows add a separately measured
activation-quantization time in the scale-factor layout that backend
consumes, except the NVFP4 CUTLASS MoE row, which is a single fused
measurement. MoE routing is synthetic: uniform expert
distribution everywhere (real skewed routing is slower), and the trtllm-gen
rows, which route in-kernel, use a fixed renormalize method regardless of
the model's routing scheme. CUTLASS and CuteDSL MoE rows receive precomputed
expert indices and exclude routing-selection cost entirely. No row includes
the router GEMM itself. These are kernel times
only — never describe them as end-to-end latency or throughput; they omit
weights, layer frequency, communication, KV cache, and scheduling.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

