modelopt-qad
Skill5.6k starsChanged todayApache-2.0View on GitHub
Run explicitly requested ModelOpt Quantization-Aware Distillation (QAD) through Megatron Bridge (recommended) or Hugging Face Transformers to recover a measured BF16-to-PTQ accuracy gap. Use only when the user explicitly asks for QAD, including its topology, data preparation, launch, resume, checkpoint export, or recovery decisions.
1 other project uses a version of this.
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Most recently taken up by
- NVIDIA/Model-Optimizerword for word, first seen Nov 15, 2026
Who's talking about NVIDIA/TensorRT-Model-Optimizer
- NVIDIA/Model-Optimizer“Replace ONNX simplification package from onnxsim to onnxslim”GitHub · Oct 28, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“Fix FLOPs calculation”GitHub · Sep 29, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case casting input directly to output”GitHub · Sep 9, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case with cast producing network output”GitHub · Sep 8, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“"Unknown quantization method" when loading nvfp4 checkpoint with vLLM”GitHub · Aug 19, 2025 · 1 reactions
The biggest 5 of 45 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 45
Talked about on GitHub (45 times).
On GitHub · 45
- Oct 28, 2025Replace ONNX simplification package from onnxsim to onnxslim — NVIDIA/Model-Optimizer1 reaction · 23 comments
- Sep 29, 2025Fix FLOPs calculation — NVIDIA/Model-Optimizer1 reaction · 11 comments
- Sep 9, 2025[Autocast] Fix edge case casting input directly to output — NVIDIA/Model-Optimizer1 reaction · 10 comments
- Sep 8, 2025[Autocast] Fix edge case with cast producing network output — NVIDIA/Model-Optimizer1 reaction · 6 comments
- Aug 19, 2025"Unknown quantization method" when loading nvfp4 checkpoint with vLLM — NVIDIA/Model-Optimizer1 reaction · 3 comments
40 more on GitHub
- Mar 30, 2025[Quantization] Add TRT-ModelOpt as a Backend — huggingface/diffusers1 reaction · 50 comments
- Mar 11, 2025✨[Feature] Supporting Blackwell FP4 — pytorch/TensorRT1 reaction
- Sep 16, 2026推論最適化ツール・OSS — warabinbin/papers0 reactions
- Aug 10, 2026fold_weight crashes with AttributeError: 'NoneType' object has no attribute 'data' on Megatron models with tied word embeddings — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 5, 2025Kimi-k2 calib+export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 4, 2025Quantization — HashCitrine/sample-to-make-llm-service0 reactions
- Dec 4, 2025Diffusion Rename [TRT MO -> MO] — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Dec 3, 2025[5704162] Create a copy to avoid leaking ProcessGroup into state dict — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Dec 2, 2025fix: Fix auto quantize multi-gpu tests — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Dec 2, 2025qwen3_8b eagle3 model train and vllm ckpt export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 27, 2025Add all example e2e tests for github PR merge / nightly — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 26, 2025[Bug Fix]: Correct block_size in NVFP4_MLP_WEIGHT_ONLY_CFG to support vLLM inference — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 26, 2025fix: relax sympy version — inisis/OnnxSlim0 reactions · 1 comment
- Nov 24, 2025[None][doc] Deprecate --export_fmt flag when using ModelOpt — NVIDIA/TensorRT-LLM0 reactions · 4 comments
- Nov 24, 2025[Help wanted] CNN QAT: How to correctly export ONNX and use DeviceModel for inference — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 22, 2025fix: issues related to TensorRT-Model-Optimizer — inisis/OnnxSlim0 reactions · 2 comments
- Nov 20, 2025[2/N] Added KDLoss based AutoQuantize — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 20, 2025[1/N] Refactored AutoQuantizeSearcher to _AutoQuantizeBaseSearcher & AutoQuantizeGradientSearcher; seperated quant modules and score modules — NVIDIA/Model-Optimizer0 reactions · 11 comments
- Nov 17, 2025failed to quant the Qwen3-VL-30B-A3B — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 13, 2025Support Wan2.2 t2v diffusers quantization — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 3, 2025PyTorch geometric quantization support — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 3, 2025Failed to save quantized model if some tensors offload to CPU — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Oct 28, 2025[OMNIML-2917] export layer config using actual prefix instead of hard… — NVIDIA/Model-Optimizer0 reactions · 9 comments
- Oct 27, 2025Add convert_llama3_config_to_decilm_config + unit test — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025CodeRabbit Generated Unit Tests: Add GPU tests for Megatron quantization; review/refactor unit tests — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025📝 Add docstrings to `kaix/kvcache_mcore` — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 15, 2025FP8 Block quantize onnx export support — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Sep 15, 2025add end_process in deepseek ptq — NVIDIA/Model-Optimizer0 reactions · 10 comments
- Jul 13, 2025feat: Simplify and Improve Whisper Example — NVIDIA/TensorRT-LLM0 reactions · 3 comments
- Jul 3, 2025[future request] Support more quant for torch backend — NVIDIA/TensorRT-LLM0 reactions · 2 comments
- Jul 3, 2025🐛 [Bug] examples/dynamo/torch_compile_gpt2.py run failed — pytorch/TensorRT0 reactions · 2 comments
- Jun 15, 2025INT8 to FP8 scale conversion — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Jun 13, 2025Fix speculative decoding example — NVIDIA/Model-Optimizer0 reactions · 6 comments
- May 21, 2025`get_modelike_from_algo_cfg` doesn't accept `QuantizeAlgorithmConfig`, but it's typing says it should — NVIDIA/Model-Optimizer0 reactions · 3 comments
- May 17, 2025Decode latency of FP4 LLM is higher than expected — NVIDIA/TensorRT-LLM0 reactions · 1 comment
- May 7, 2025Error in torch_compile.ipynb — Project-MONAI/tutorials0 reactions · 1 comment
- May 6, 2025当我使用增量训练的时候,只改变修改一个feature的zch_size的大小,其他所有配置都不改变,会出错。 — alibaba/TorchEasyRec0 reactions
- Apr 30, 2025INT8 quantization for HiFi-GAN vocoder -- performance issue — NVIDIA/TensorRT0 reactions · 7 comments
- Mar 24, 2025bert model split into many layers after int8 quantization — NVIDIA/TensorRT0 reactions · 13 comments
- Sep 8, 2024urlopen error retrieval incomplete in modules/autoencoder_mednist.ipynb — Project-MONAI/tutorials0 reactions
Other versions1 project uses a version of this.
Word for word · 1
- NVIDIA/Model-Optimizer★ 5.6k100% same
Edited · 0
None found.
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository33 other files it gives its agents.
- AGENTS.mdAGENTS.md
- tools/debugger/CLAUDE.mdCLAUDE.md
- tools/launcher/CLAUDE.mdCLAUDE.md
- accessing-mlflowSkill
- benchmark-model-kernelsSkill
- commonSkill
- compare-resultsSkill
- day0-releaseSkill
- debugSkill
- deploymentSkill
- evaluationSkill
- launching-evalsSkill
- modelopt-accessing-mlflowSkill
- modelopt-benchmark-model-kernelsSkill
- modelopt-compare-resultsSkill
- modelopt-day0-releaseSkill
- modelopt-debugSkill
- modelopt-deploymentSkill
- modelopt-evaluationSkill
- modelopt-launching-evalsSkill
- modelopt-monitorSkill
- modelopt-ptqSkill
- modelopt-quant-recipe-searchSkill
- modelopt-release-cherry-pickSkill
- modelopt-run-agentxSkill
- modelopt-speculative-decodingSkill
- monitorSkill
- ptqSkill
- qadSkill
- quant-recipe-searchSkill
- release-cherry-pickSkill
- run-agentxSkill
- speculative-decodingSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
13 rules. About 0 are common, 8 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈8
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 36
Use the common skill's environment-setup.md and workspace-management.md.
- Very specific: line 39
Treat the example README and --help output as authoritative for mutable flags, commands, containers, and checkpoint formats.
- Very specific: line 83
Prepare the full capped dataset once: Use suitable user-provided data, or copy examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml …
- Distinctive: line 90
Run and monitor QAD: Run one training job at a time per QAD run.
- Distinctive: line 134
Start benchmark evaluation at the knee of the KD loss curve, where its steep drop ends, rounded to the next checkpoint, then evaluate every …
- Distinctive: line 140
Run noise is 2x the standard error of a benchmark's gap to BF16: paired per question when both sides score the same questions, otherwise …
- Distinctive: line 151
Pass: every benchmark of the full original PTQ suite is within 1% of BF16, each at its compare-results run floor (such as SciCode's 8 runs …
- Distinctive: line 158
After a duration exit, resume the latest QAD checkpoint in the same output directory with unchanged prepared data paths/cache, seed, …
- Very specific: line 175
On a miss, change one QAD knob: Read each missed benchmark against PTQ: worse than PTQ is drift (lower the peak LR or add matching data).
- Very specific: line 191
Before constructing commands, read examples/llm_qat/README.md, its ARGUMENTS.md, selected training/dataset/Accelerate configs, and …
- Distinctive: line 194
Follow its quantize → distill → export flow: the PTQ checkpoint is the student.
- Distinctive: line 198
Use the HF tutorial's evaluation and checkpoint resume guidance.
- Distinctive: line 203
Use HF evaluation/checkpoint intervals.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
ModelOpt Quantization-Aware Distillation
QAD is expensive. Run it only when the user explicitly authorizes QAD for the target model or run, with a loop or GPU-hour budget; ask for the budget when it is missing. A Day-0, PTQ, evaluation, comparison, or recipe-search request alone is not authorization to start QAD. The authorization covers new recipes from a recipe search only when the user says so.
Confirm the gap
For both frameworks, reuse only validated, comparable BF16/PTQ results and the exact benchmark configuration from preceding evaluation or recipe search; run missing, invalid, or non-comparable baselines. Confirm the target benchmarks and their context-length needs. Stop if the PTQ gap to BF16 is already below 1%.
Follow the supported workflow
Read docs/source/guides/quantization_aware_training_and_distillation.rst
for QAD background and framework selection.
Recommend Megatron Bridge for scalable training and its model-specific tutorials.
Choose the applicable tutorial from examples/megatron_bridge/tutorials/README.md.
Honor an explicit HF choice.
Use the common skill's environment-setup.md and workspace-management.md;
for Slurm, also read slurm-setup.md and, when remote, remote-execution.md.
Treat the example README and --help output as authoritative for mutable flags,
commands, containers, and checkpoint formats. Both frameworks can run on local
GPUs or Slurm; use the user's requested environment.
Megatron Bridge execution and training policy
For Megatron Bridge, before constructing commands, read:
examples/megatron_bridge/README.md, especially PTQ, data preparation, QAD, export, and Slurm usageexamples/megatron_bridge/{quantize.py,distill.py}via--help
Execute in this order
- Reproduce PTQ and verify compatibility. In the target runtime, require
AutoBridge.can_handle()for the target model and PTQ throughquantize.pyto succeed while preserving the exact preceding PTQ config or recipe: format, layer selection, calibration data/count, sequence length, and seed. A changed quantization setting is a new PTQ candidate and must be evaluated before QAD. In the master-rank.quant_summary.txt, require finite positiveamaxfor enabled static quantizers; acceptdynamic/format-definedNoneonly when the recipe intends it. Treat the summary as rank-local under model parallelism. Choose topology explicitly. Derive the smallest fitting node count and TP/PP/CP/EP from student and teacher architecture, the chosen sequence length, and available GPU memory. Prefer CP before TP for small long-context models; keep EP=1 for dense models and ETP=1 because the current
distill.pyworkflow does not support expert tensor parallelism. For MoE require:DP = world_size / (TP * PP * CP)EDP = world_size / (EP * PP)- integral DP/EDP,
num_experts % EP == 0, andGBS % (MBS * DP) == 0
Static-block NVFP4 recipes require TP=1; use PP for memory. With PP, balance stages with
--num_layers_in_{first,last}_pipeline_stage(the first and last stages also hold the embedding and output layer) and recompute only where memory needs it (--recompute_num_layers_per_stage). In the first job's allocation, time a few iterations of each candidate layout with short launches before the run starts, and keep the fastest. Give each timing launch a scratch--output_dir:distill.pyresumes from<output_dir>/checkpoints, so a timing checkpoint there would become the run's starting point.- Prepare the full capped dataset once. Use suitable user-provided data, or
copy
examples/megatron_bridge/data/nemotron-cascade-2-blend.yamlas the default. Set the target tokenizer and workspace path, then materialize the randomly sampled subset before training. Pack the chosen sequence length; Megatron's99,1,0split creates the 1% validation holdout from the same data. Keep every source mixed across training, never ordered as a curriculum, so early stops see the whole blend. - Run and monitor QAD. Run one training job at a time per QAD run; separate runs may train in parallel. Fold startup validation into the first job; do not submit separate GPU preflight jobs or split at recovery iterations. Let training continue while evaluating saved checkpoints, and cancel it when a stop condition below is met.
Default training policy
| Setting | Default |
| Sequence length | 32768; adjust for target benchmarks |
| Peak / minimum LR | 1e-5 / 1e-6 |
| LR schedule | cosine |
LR horizon (train_iters) | 1000 iterations |
| Global batch size | 512 |
| Dataset | nvidia/Nemotron-Cascade-2-SFT-Data by default |
| Materialized token budget | 17.3B at 32K; cover train_iters at the chosen length |
| Training validation | every 25 iterations; deterministic 1% holdout; 2 batches |
| Checkpoint interval | 50 iterations |
| Loss logging | every 10 iterations |
| Recovery benchmark | from the KD-loss knee, then every checkpoint |
| Slurm duration exit | 220 minutes for a 4-hour allocation |
Run policy
train_iterssets the cosine LR horizon and is a loop knob: with 1000, the LR stays near its peak through iteration 150.distill.pyalso stops training there, with or withoutexit_interval. Stop runs by the decisions below and leaveexit_intervalunset. A run that reachestrain_iterswhile the decision is still to continue ends as a Miss from under-recovery.- On Slurm, use initial step timing to submit only enough sequential jobs to reach
a few checkpoints past the expected KD-loss knee; never submit through
train_itersupfront. At each recovery checkpoint, submit to the next only after its targeted evaluation and any triggered full suite, and only while the decision is to continue. - On Slurm, give all training jobs the same run-specific job name and
--dependency=singleton; record job IDs and, on any stop, cancel pending jobs before the active job. - Cancel on non-finite loss, repeated skipped iterations, or a sustained spike. At iteration 50, require the loss aggregate to be lower than at iteration 10.
- At each recovery checkpoint, first evaluate the one to three benchmarks with the largest PTQ drops or that failed in earlier loops. Run the remaining original PTQ suite at that checkpoint once the targeted benchmarks are within run noise of 1%, or after recovery beyond run noise.
- Start benchmark evaluation at the knee of the KD loss curve, where its steep drop ends, rounded to the next checkpoint, then evaluate every checkpoint while the scores still change. Before the knee, use loss only to catch drift; after it, judge and pick checkpoints by benchmarks alone, since KD loss keeps falling after they peak. Between near ties, the benchmark most sensitive to quantization decides.
- Run noise is 2x the standard error of a benchmark's gap to BF16: paired per question when both sides score the same questions, otherwise from both sides' per-sample standard errors, and pooled across runs once repeats exist. Repeat a benchmark only when its gap is within run noise of 1%, up to 8 runs per side; the compare-results run floors still apply before a Pass.
- Export each checkpoint to evaluate and pass the ptq skill's
references/checkpoint-validation.mdgate on it before evaluation. Run the serving canary on the run's first export only; later exports change only weights, so on those the gate passes without the canary: record its serving-canary row asskipped; first export <path> passed. - At each evaluated checkpoint, decide:
- Pass: every benchmark of the full original PTQ suite is within 1% of BF16, each at its compare-results run floor (such as SciCode's 8 runs per side). Keep the checkpoint and stop the run.
- Miss: two evaluated checkpoints past the knee bring no gain beyond run noise, or two consecutive checkpoints decline beyond it. Stop and change one QAD knob in the recipe loop.
- Otherwise continue to the next checkpoint.
- After a duration exit, resume the latest QAD checkpoint in the same output directory with unchanged prepared data paths/cache, seed, topology, optimizer, scheduler, iteration, and consumed-sample state; do not restart from PTQ.
- Report, per loop, the PTQ recipe and calibration, data sample, QAD settings, training topology, loss/state, checkpoints, and comparable BF16/PTQ/QAD results.
Recipe loop
Each run trains one PTQ recipe and differs from that recipe's control in one QAD knob. Start from the best PTQ recipe of the preceding PTQ evaluation or recipe search; its control uses the default policy. When GPUs allow, start one-knob variants as separate runs alongside the control instead of waiting for its miss. This loop chooses the QAD settings; the quant-recipe-search skill chooses the next recipe from its outcome. Stop when the QAD budget is spent and report the best result so far.
- On a miss, change one QAD knob. Read each missed benchmark against PTQ:
worse than PTQ is drift (lower the peak LR or add matching data); better
than PTQ but short is under-recovery (keep the LR high longer with a larger
train_iters). Knobs, in order of priority when GPUs are limited: peak LR,train_iters, data blend, then teacher-generated data. A changedtrain_itersis a fresh run from PTQ, with data materialized to cover it: resuming would change the decay length the checkpoint's LR scheduler expects. - Compare a knob with its control at the same checkpoints past the knee, on benchmark scores pooled over fresh runs. A rejection holds only for that PTQ recipe: rerun the control when the recipe changes.
- End the loop on a recipe with a pass, or with a clear miss once every knob has missed or as soon as QAD stays flat against PTQ or far from passing. Return the outcome, the best checkpoint, and its scores.
Hugging Face execution and training policy
Before constructing commands, read examples/llm_qat/README.md, its
ARGUMENTS.md, selected training/dataset/Accelerate configs, and
{quantize.py,train.py,export.py} --help.
Follow its quantize → distill → export flow: the PTQ checkpoint is the student;
the original unquantized model is the teacher. Adapt the QAD training and
Accelerate configs to the model and budget.
Resolve tutorial-relative commands and config paths from examples/llm_qat/.
Use the HF tutorial's evaluation and checkpoint resume guidance.
Reuse the Bridge procedure's PTQ-recipe preservation and one-training-job-at-a-time guidance. Apply the Run policy's loss-stability and benchmark-recovery stop conditions and report the same fields. Use HF evaluation/checkpoint intervals; Bridge-specific topology, data defaults, iteration thresholds, and Slurm job chaining are not HF requirements.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

