modelopt-ptq
Skill5.6k starsChanged todayApache-2.0View on GitHub
Use when the user asks to "quantize a model", "run PTQ", "post-training quantization", "NVFP4 quantization", "FP8 quantization", "INT8 quantization", "INT4 AWQ", "quantize LLM", "quantize MoE", "quantize VLM", or needs to produce a quantized HuggingFace checkpoint from a pretrained model using ModelOpt. Do NOT use for multi-candidate recipe exploration or optimization (use modelopt-quant-recipe-search).
- Installs packages
1 other project uses a version of this.
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Most recently taken up by
- NVIDIA/Model-Optimizerword for word, first seen Nov 15, 2026
Who's talking about NVIDIA/TensorRT-Model-Optimizer
- NVIDIA/Model-Optimizer“Replace ONNX simplification package from onnxsim to onnxslim”GitHub · Oct 28, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“Fix FLOPs calculation”GitHub · Sep 29, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case casting input directly to output”GitHub · Sep 9, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case with cast producing network output”GitHub · Sep 8, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“"Unknown quantization method" when loading nvfp4 checkpoint with vLLM”GitHub · Aug 19, 2025 · 1 reactions
The biggest 5 of 45 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 45
Talked about on GitHub (45 times).
On GitHub · 45
- Oct 28, 2025Replace ONNX simplification package from onnxsim to onnxslim — NVIDIA/Model-Optimizer1 reaction · 23 comments
- Sep 29, 2025Fix FLOPs calculation — NVIDIA/Model-Optimizer1 reaction · 11 comments
- Sep 9, 2025[Autocast] Fix edge case casting input directly to output — NVIDIA/Model-Optimizer1 reaction · 10 comments
- Sep 8, 2025[Autocast] Fix edge case with cast producing network output — NVIDIA/Model-Optimizer1 reaction · 6 comments
- Aug 19, 2025"Unknown quantization method" when loading nvfp4 checkpoint with vLLM — NVIDIA/Model-Optimizer1 reaction · 3 comments
40 more on GitHub
- Mar 30, 2025[Quantization] Add TRT-ModelOpt as a Backend — huggingface/diffusers1 reaction · 50 comments
- Mar 11, 2025✨[Feature] Supporting Blackwell FP4 — pytorch/TensorRT1 reaction
- Sep 16, 2026推論最適化ツール・OSS — warabinbin/papers0 reactions
- Aug 10, 2026fold_weight crashes with AttributeError: 'NoneType' object has no attribute 'data' on Megatron models with tied word embeddings — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 5, 2025Kimi-k2 calib+export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 4, 2025Quantization — HashCitrine/sample-to-make-llm-service0 reactions
- Dec 4, 2025Diffusion Rename [TRT MO -> MO] — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Dec 3, 2025[5704162] Create a copy to avoid leaking ProcessGroup into state dict — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Dec 2, 2025fix: Fix auto quantize multi-gpu tests — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Dec 2, 2025qwen3_8b eagle3 model train and vllm ckpt export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 27, 2025Add all example e2e tests for github PR merge / nightly — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 26, 2025[Bug Fix]: Correct block_size in NVFP4_MLP_WEIGHT_ONLY_CFG to support vLLM inference — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 26, 2025fix: relax sympy version — inisis/OnnxSlim0 reactions · 1 comment
- Nov 24, 2025[None][doc] Deprecate --export_fmt flag when using ModelOpt — NVIDIA/TensorRT-LLM0 reactions · 4 comments
- Nov 24, 2025[Help wanted] CNN QAT: How to correctly export ONNX and use DeviceModel for inference — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 22, 2025fix: issues related to TensorRT-Model-Optimizer — inisis/OnnxSlim0 reactions · 2 comments
- Nov 20, 2025[2/N] Added KDLoss based AutoQuantize — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 20, 2025[1/N] Refactored AutoQuantizeSearcher to _AutoQuantizeBaseSearcher & AutoQuantizeGradientSearcher; seperated quant modules and score modules — NVIDIA/Model-Optimizer0 reactions · 11 comments
- Nov 17, 2025failed to quant the Qwen3-VL-30B-A3B — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 13, 2025Support Wan2.2 t2v diffusers quantization — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 3, 2025PyTorch geometric quantization support — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 3, 2025Failed to save quantized model if some tensors offload to CPU — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Oct 28, 2025[OMNIML-2917] export layer config using actual prefix instead of hard… — NVIDIA/Model-Optimizer0 reactions · 9 comments
- Oct 27, 2025Add convert_llama3_config_to_decilm_config + unit test — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025CodeRabbit Generated Unit Tests: Add GPU tests for Megatron quantization; review/refactor unit tests — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025📝 Add docstrings to `kaix/kvcache_mcore` — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 15, 2025FP8 Block quantize onnx export support — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Sep 15, 2025add end_process in deepseek ptq — NVIDIA/Model-Optimizer0 reactions · 10 comments
- Jul 13, 2025feat: Simplify and Improve Whisper Example — NVIDIA/TensorRT-LLM0 reactions · 3 comments
- Jul 3, 2025[future request] Support more quant for torch backend — NVIDIA/TensorRT-LLM0 reactions · 2 comments
- Jul 3, 2025🐛 [Bug] examples/dynamo/torch_compile_gpt2.py run failed — pytorch/TensorRT0 reactions · 2 comments
- Jun 15, 2025INT8 to FP8 scale conversion — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Jun 13, 2025Fix speculative decoding example — NVIDIA/Model-Optimizer0 reactions · 6 comments
- May 21, 2025`get_modelike_from_algo_cfg` doesn't accept `QuantizeAlgorithmConfig`, but it's typing says it should — NVIDIA/Model-Optimizer0 reactions · 3 comments
- May 17, 2025Decode latency of FP4 LLM is higher than expected — NVIDIA/TensorRT-LLM0 reactions · 1 comment
- May 7, 2025Error in torch_compile.ipynb — Project-MONAI/tutorials0 reactions · 1 comment
- May 6, 2025当我使用增量训练的时候,只改变修改一个feature的zch_size的大小,其他所有配置都不改变,会出错。 — alibaba/TorchEasyRec0 reactions
- Apr 30, 2025INT8 quantization for HiFi-GAN vocoder -- performance issue — NVIDIA/TensorRT0 reactions · 7 comments
- Mar 24, 2025bert model split into many layers after int8 quantization — NVIDIA/TensorRT0 reactions · 13 comments
- Sep 8, 2024urlopen error retrieval incomplete in modules/autoencoder_mednist.ipynb — Project-MONAI/tutorials0 reactions
Other versions1 project uses a version of this.
Word for word · 1
- NVIDIA/Model-Optimizer★ 5.6k100% same
Edited · 0
None found.
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository33 other files it gives its agents.
- AGENTS.mdAGENTS.md
- tools/debugger/CLAUDE.mdCLAUDE.md
- tools/launcher/CLAUDE.mdCLAUDE.md
- accessing-mlflowSkill
- benchmark-model-kernelsSkill
- commonSkill
- compare-resultsSkill
- day0-releaseSkill
- debugSkill
- deploymentSkill
- evaluationSkill
- launching-evalsSkill
- modelopt-accessing-mlflowSkill
- modelopt-benchmark-model-kernelsSkill
- modelopt-compare-resultsSkill
- modelopt-day0-releaseSkill
- modelopt-debugSkill
- modelopt-deploymentSkill
- modelopt-evaluationSkill
- modelopt-launching-evalsSkill
- modelopt-monitorSkill
- modelopt-qadSkill
- modelopt-quant-recipe-searchSkill
- modelopt-release-cherry-pickSkill
- modelopt-run-agentxSkill
- modelopt-speculative-decodingSkill
- monitorSkill
- ptqSkill
- qadSkill
- quant-recipe-searchSkill
- release-cherry-pickSkill
- run-agentxSkill
- speculative-decodingSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
18 rules. About 0 are common, 5 are distinctive, and 13 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
13
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 16
Use modelopt-quant-recipe-search for multi-candidate recipe exploration or optimization.
- Very specific: line 21
Use the common skill's environment-setup.md and workspace-management.md.
- Distinctive: line 22
After completing them you should know:
- Very specific: line 32
Check the support table in examples/hf_ptq/README.md for verified HF models.
- Very specific: line 39
If the model uses trust_remote_code (check config.json for auto_map), inspect its custom Python files for imports not present in the …
- Distinctive: line 51
If extra deps are needed:
- Very specific: line 64
If a model-specific recipe exists, prefer --recipe <path> — but inspect its include/exclude patterns rather than assuming (e.g.
- Very specific: line 71
Use --qformat <name> (e.g., --qformat nvfp4).
- Very specific: line 90
Text-only LLM PTQ: Prefer the representative nemotron-post-training-v3 blend.
- Very specific: line 130
Run --help for all options.
- Very specific: line 137
Write a YAML config using common/hf/ptq.sh.
- Very specific: line 151
Follow references/unsupported-models.md.
- Distinctive: line 158
After job submission, register the job and set up monitoring per the modelopt-monitor skill.
- Very specific: line 173
Read references/checkpoint-validation.md and perform all four validation groups on the exact checkpoint path that will be …
- Distinctive: line 175
Check output size and estimated bits per weight against the baseline/source checkpoint.
- Very specific: line 176
Check quantized-weight coverage against the requested qformat/recipe/config.
- Distinctive: line 177
Check metadata consistency against the baseline/source model.
- Very specific: line 185
Call mto.enable_huggingface_checkpointing() before quantization
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
ModelOpt Post-Training Quantization
Produce a quantized checkpoint from a pretrained model. Read examples/hf_ptq/README.md first — it has the support matrix, CLI flags, and accuracy guidance.
Use modelopt-quant-recipe-search for multi-candidate recipe exploration or
optimization. Use this skill for each selected recipe's PTQ run.
Step 1 — Environment
Use the common skill's environment-setup.md and workspace-management.md.
After completing them you should know:
- ModelOpt source is available
- Local or remote (+ cluster config if remote)
- SLURM / Docker+GPU / bare GPU
- Launcher available?
- Which workspace to use
Step 2 — Is the model supported?
Check the support table in examples/hf_ptq/README.md for verified HF models.
- Listed → supported, use
hf_ptq.py(step 4A/4B) - Not listed → read
references/unsupported-models.mdto determine ifhf_ptq.pycan still work or if a custom script is needed (step 4C)
Step 2.5 — Check for model-specific dependencies
If the model uses trust_remote_code (check config.json for auto_map), inspect its custom Python files for imports not present in the container:
grep -h "^from \|^import " <model_path>/modeling_*.py | sort -uKnown dependency patterns:
| Import found | Packages to install |
from mamba_ssm / from causal_conv1d | mamba-ssm causal-conv1d (Mamba/hybrid models: NemotronH, Jamba) |
If extra deps are needed:
- Launcher (4B): set
EXTRA_PIP_DEPSin the task'senvironmentsection —ptq.shinstalls them automatically - Manual (4A):
unset PIP_CONSTRAINT && pip install <deps>before runninghf_ptq.py
Step 3 — Choose quantization format
First, check for a model-specific recipe:
ls modelopt_recipes/models/ 2>/dev/null
ls modelopt_recipes/model_type/<model_type>/ptq/ 2>/dev/null # per-arch; <model_type> from local config.json (Hub ID: AutoConfig.from_pretrained)If a model-specific recipe exists, prefer --recipe <path> — but inspect its include/exclude patterns rather than assuming (e.g. for VLMs, confirm the vision tower is actually excluded).
If no model-specific recipe, choose a format based on GPU (details in examples/hf_ptq/README.md):
- Blackwell (B100/B200/GB200):
nvfp4variants - Hopper (H100/H200) or older:
fp8orint4_awq
Use --qformat <name> (e.g., --qformat nvfp4). Format definitions: modelopt/torch/quantization/config.py. General PTQ recipes in modelopt_recipes/general/ptq/ correspond to the same formats — --qformat is the simpler way to use them.
Before running PTQ, sanity-check the selected qformat/recipe against the model structure. Inspect the recipe's include/exclude patterns and summarize which layer groups will be quantized and approximately how many modules/layers match (attention projections, MLP projections, experts, etc.). If the match count is 0, or far smaller than expected for the model, stop and fix the recipe or ask the user before launching calibration.
VLMs: generic *mlp*/*experts* recipes also match the vision tower (model.visual.*); quantizing the ViT silently breaks image benchmarks. Use the model_type/<model_type>/ptq/ recipe or add *visual*/*vision_tower* excludes, then verify in Step 5 — see references/checkpoint-validation.md.
If the source checkpoint is already quantized and the requested recipe/config reduces quantization coverage, confirm that intent with the user before running. For example, if an FP8 checkpoint is used as input and the recipe excludes some layers so they would fall back to BF16 instead of staying quantized, call out the affected layer groups and ask whether that FP8-to-BF16 fallback is intended.
NVFP4 can be calibrated on Hopper but requires Blackwell for inference.
Step 4 — Run PTQ
Goal: checkpoint on disk (.safetensors + config.json).
For listed models (4A/4B): run full calibration directly (--calib_size 512).
For unlisted models (4C): run a smoke test first (--calib_size 4), wait for success, then full calibration.
Recommended calibration datasets
Text-only LLM PTQ: Prefer the representative
nemotron-post-training-v3blend.modelopt/torch/utils/dataset_utils.pyexpands it to seven registered Nemotron SFT domains. Configure Hugging Face credentials where required.python examples/hf_ptq/hf_ptq.py ... \ --dataset nemotron-post-training-v3VLM PTQ: Include image-text calibration with
--calib_with_images. This path usesnemotron_vlm_dataset_v2with the current default subsetssparsetables,plotqa_cot, andwiki_en;examples/hf_ptq/hf_ptq.pyandmodelopt/torch/utils/vlm_dataset_utils.pyare the source of truth.python examples/hf_ptq/hf_ptq.py ... \ --calib_with_images
--dataset selects text-only calibration and cannot substitute for multimodal examples. Use --dataset cnn_dailymail only as a fallback when representative data is unavailable, such as without gated-data access or with only a local public cache.
Which path?
In README table? ─→ YES ──→ SLURM (local or remote)? ──→ LAUNCHER (4B)
│ Local Docker + GPU? ────────→ LAUNCHER (4B)
│ Remote Docker (no SLURM)? ──→ MANUAL (4A)
│ Bare GPU (local or remote)? → MANUAL (4A)
│
└→ NOT LISTED ──→ UNLISTED MODEL (4C)4A — Direct: supported model, manual execution
pip install --no-build-isolation "nvidia-modelopt[hf]"
pip install -r examples/hf_ptq/requirements.txt
python examples/hf_ptq/hf_ptq.py \
--pyt_ckpt_path <model> \
--qformat <format> \
--calib_size 512 \
--export_path <output>Run --help for all options.
For remote, use remote_run from the common skill's remote_exec.sh; see its
remote-execution.md.
4B — Launcher: supported model on SLURM or local Docker
Write a YAML config using common/hf/ptq.sh. See references/launcher-guide.md for the full template.
cd tools/launcher
# SLURM (remote or local):
SLURM_HOST=<host> SLURM_ACCOUNT=<acct> uv run launch.py --yaml <config.yaml> user=<ssh_user> identity=<ssh_key> --yes
# Local Docker:
uv run launch.py --yaml <config.yaml> hf_local=<hf_cache> --yesThe launcher blocks and tails logs until the job completes. If the launcher fails (missing deps, config errors), fall back to path 4A (manual execution).
4C — Unlisted model
Follow references/unsupported-models.md. It walks through investigating the model, patching ModelOpt if needed, and running hf_ptq.py. Run manually (like 4A) for easier monitoring and debugging.
For SLURM, see the common skill's slurm-setup.md and this skill's
references/slurm-setup-ptq.md.
Monitoring
After job submission, register the job and set up monitoring per the modelopt-monitor skill.
Step 5 — Verify output
ls -lh <output_path>/
# Expect: config.json, tokenizer files, model-*.safetensorsReport the path and size to the user.
Post-quantization validation
This is a required gate before any deployment or evaluation submission. Do not submit an eval, start a production serving job, or hand off the checkpoint as ready until the gate, including its serving canary, has passed.
Read references/checkpoint-validation.md and perform all four validation groups on the exact checkpoint path that will be deployed/evaluated:
- Check output size and estimated bits per weight against the baseline/source checkpoint.
- Check quantized-weight coverage against the requested qformat/recipe/config.
- Check metadata consistency against the baseline/source model.
- Complete the required downstream handoff and serving-readiness validation.
Report the gate result before moving on. Follow the canonical report format and all blocking conditions in references/checkpoint-validation.md; do not hand off a checkpoint unless every required check passes.
Key API Rules
mtq.register()classes must define_setup()and call it from__init__- Call
mto.enable_huggingface_checkpointing()before quantization - Wildcard
*gate*matches too broadly — use*mlp.gate*or*router* - VLMs:
hf_ptq.pyauto-extracts the language model viaextract_and_prepare_language_model_from_vl()— no manual VLM handling needed in most cases - FP8 checkpoints: prefer
_QuantFP8Linear(lazy dequant) overFineGrainedFP8Config(dequantize=True)which wastes ~2x memory. Seereferences/unsupported-models.mdfor details - Custom quantizer names must end with
_input_quantizeror_weight_quantizer
Common Pitfalls
- Model-specific dependencies: Models with
trust_remote_codemay import packages not in the container (e.g.,mamba-ssmfor hybrid Mamba models). See Step 2.5. UseEXTRA_PIP_DEPSenv var with the launcher, or install manually before runninghf_ptq.py - Linear-attention models (Qwen3-Next/3.5/3.8, GLM-5.x Flash, Kimi Linear, OLMo Hybrid) fall back to slow, memory-hungry PyTorch kernels without
pip install "fla-core>=0.5.2"; install it before calibrating on GPU (it breaks CPU runs of these models) - Transformers version: New models may need a newer version of transformers than what's installed. Check
config.jsonfortransformers_version. In containers, beware ofPIP_CONSTRAINTblocking upgrades — seereferences/slurm-setup-ptq.mdfor workarounds - Gated datasets: Some calibration datasets require HF authentication. Set
HF_TOKENin the job environment. Use--dataset cnn_dailymailonly as the constrained-environment fallback described in Step 4, not as the preferred calibration set - NFS root_squash + Docker: See the common skill's
slurm-setup.mdsection 5
References
| Reference | When to read |
common skill: environment-setup.md | Step 1: always |
common skill: workspace-management.md | Step 1: always |
references/launcher-guide.md | Step 4B only (launcher path) |
tools/launcher/CLAUDE.md | Step 4B only, if you need more launcher detail |
references/unsupported-models.md | Step 4C only (unlisted model) |
references/checkpoint-validation.md | Step 5: mandatory post-PTQ gate before deployment/evaluation |
common skill: remote-execution.md | Step 4A/4C only, if target is remote |
common skill: slurm-setup.md | Step 4A/4C only, if using SLURM manually (not launcher) |
references/slurm-setup-ptq.md | Step 4A/4C only, PTQ-specific SLURM (container, GPU sizing, FSDP2) |
examples/hf_ptq/README.md | Step 3: support matrix, CLI flags, accuracy |
modelopt/torch/quantization/config.py | Step 3: format definitions |
modelopt/torch/export/model_utils.py | Step 4C: TRT-LLM export type mapping |
modelopt_recipes/ | Step 3: pre-built recipes |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

