modelopt-speculative-decoding
Skill5.6k starsChanged todayApache-2.0View on GitHub
Train, debug, and validate a speculative decoding draft model (EAGLE3, DFlash, DSpark, Domino) through the ModelOpt launcher pipeline. Use when the user wants to add a new model to a draft-training pipeline, asks why a pipeline run failed, wants experiment logs reviewed, or wants to check whether a run's acceptance rate meets threshold. Triggers on "EAGLE3", "DFlash", "DSpark", "Domino", "draft model", "acceptance rate", "speculative decoding pipeline". Do NOT use for quantizing a model (use modelopt-ptq) or serving a checkpoint (use modelopt-deployment).
1 other project uses a version of this.
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Most recently taken up by
- NVIDIA/Model-Optimizerword for word, first seen Nov 15, 2026
Who's talking about NVIDIA/TensorRT-Model-Optimizer
- NVIDIA/Model-Optimizer“Replace ONNX simplification package from onnxsim to onnxslim”GitHub · Oct 28, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“Fix FLOPs calculation”GitHub · Sep 29, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case casting input directly to output”GitHub · Sep 9, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case with cast producing network output”GitHub · Sep 8, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“"Unknown quantization method" when loading nvfp4 checkpoint with vLLM”GitHub · Aug 19, 2025 · 1 reactions
The biggest 5 of 45 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 45
Talked about on GitHub (45 times).
On GitHub · 45
- Oct 28, 2025Replace ONNX simplification package from onnxsim to onnxslim — NVIDIA/Model-Optimizer1 reaction · 23 comments
- Sep 29, 2025Fix FLOPs calculation — NVIDIA/Model-Optimizer1 reaction · 11 comments
- Sep 9, 2025[Autocast] Fix edge case casting input directly to output — NVIDIA/Model-Optimizer1 reaction · 10 comments
- Sep 8, 2025[Autocast] Fix edge case with cast producing network output — NVIDIA/Model-Optimizer1 reaction · 6 comments
- Aug 19, 2025"Unknown quantization method" when loading nvfp4 checkpoint with vLLM — NVIDIA/Model-Optimizer1 reaction · 3 comments
40 more on GitHub
- Mar 30, 2025[Quantization] Add TRT-ModelOpt as a Backend — huggingface/diffusers1 reaction · 50 comments
- Mar 11, 2025✨[Feature] Supporting Blackwell FP4 — pytorch/TensorRT1 reaction
- Sep 16, 2026推論最適化ツール・OSS — warabinbin/papers0 reactions
- Aug 10, 2026fold_weight crashes with AttributeError: 'NoneType' object has no attribute 'data' on Megatron models with tied word embeddings — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 5, 2025Kimi-k2 calib+export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 4, 2025Quantization — HashCitrine/sample-to-make-llm-service0 reactions
- Dec 4, 2025Diffusion Rename [TRT MO -> MO] — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Dec 3, 2025[5704162] Create a copy to avoid leaking ProcessGroup into state dict — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Dec 2, 2025fix: Fix auto quantize multi-gpu tests — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Dec 2, 2025qwen3_8b eagle3 model train and vllm ckpt export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 27, 2025Add all example e2e tests for github PR merge / nightly — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 26, 2025[Bug Fix]: Correct block_size in NVFP4_MLP_WEIGHT_ONLY_CFG to support vLLM inference — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 26, 2025fix: relax sympy version — inisis/OnnxSlim0 reactions · 1 comment
- Nov 24, 2025[None][doc] Deprecate --export_fmt flag when using ModelOpt — NVIDIA/TensorRT-LLM0 reactions · 4 comments
- Nov 24, 2025[Help wanted] CNN QAT: How to correctly export ONNX and use DeviceModel for inference — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 22, 2025fix: issues related to TensorRT-Model-Optimizer — inisis/OnnxSlim0 reactions · 2 comments
- Nov 20, 2025[2/N] Added KDLoss based AutoQuantize — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 20, 2025[1/N] Refactored AutoQuantizeSearcher to _AutoQuantizeBaseSearcher & AutoQuantizeGradientSearcher; seperated quant modules and score modules — NVIDIA/Model-Optimizer0 reactions · 11 comments
- Nov 17, 2025failed to quant the Qwen3-VL-30B-A3B — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 13, 2025Support Wan2.2 t2v diffusers quantization — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 3, 2025PyTorch geometric quantization support — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 3, 2025Failed to save quantized model if some tensors offload to CPU — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Oct 28, 2025[OMNIML-2917] export layer config using actual prefix instead of hard… — NVIDIA/Model-Optimizer0 reactions · 9 comments
- Oct 27, 2025Add convert_llama3_config_to_decilm_config + unit test — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025CodeRabbit Generated Unit Tests: Add GPU tests for Megatron quantization; review/refactor unit tests — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025📝 Add docstrings to `kaix/kvcache_mcore` — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 15, 2025FP8 Block quantize onnx export support — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Sep 15, 2025add end_process in deepseek ptq — NVIDIA/Model-Optimizer0 reactions · 10 comments
- Jul 13, 2025feat: Simplify and Improve Whisper Example — NVIDIA/TensorRT-LLM0 reactions · 3 comments
- Jul 3, 2025[future request] Support more quant for torch backend — NVIDIA/TensorRT-LLM0 reactions · 2 comments
- Jul 3, 2025🐛 [Bug] examples/dynamo/torch_compile_gpt2.py run failed — pytorch/TensorRT0 reactions · 2 comments
- Jun 15, 2025INT8 to FP8 scale conversion — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Jun 13, 2025Fix speculative decoding example — NVIDIA/Model-Optimizer0 reactions · 6 comments
- May 21, 2025`get_modelike_from_algo_cfg` doesn't accept `QuantizeAlgorithmConfig`, but it's typing says it should — NVIDIA/Model-Optimizer0 reactions · 3 comments
- May 17, 2025Decode latency of FP4 LLM is higher than expected — NVIDIA/TensorRT-LLM0 reactions · 1 comment
- May 7, 2025Error in torch_compile.ipynb — Project-MONAI/tutorials0 reactions · 1 comment
- May 6, 2025当我使用增量训练的时候,只改变修改一个feature的zch_size的大小,其他所有配置都不改变,会出错。 — alibaba/TorchEasyRec0 reactions
- Apr 30, 2025INT8 quantization for HiFi-GAN vocoder -- performance issue — NVIDIA/TensorRT0 reactions · 7 comments
- Mar 24, 2025bert model split into many layers after int8 quantization — NVIDIA/TensorRT0 reactions · 13 comments
- Sep 8, 2024urlopen error retrieval incomplete in modules/autoencoder_mednist.ipynb — Project-MONAI/tutorials0 reactions
Other versions1 project uses a version of this.
Word for word · 1
- NVIDIA/Model-Optimizer★ 5.6k100% same
Edited · 0
None found.
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository33 other files it gives its agents.
- AGENTS.mdAGENTS.md
- tools/debugger/CLAUDE.mdCLAUDE.md
- tools/launcher/CLAUDE.mdCLAUDE.md
- accessing-mlflowSkill
- benchmark-model-kernelsSkill
- commonSkill
- compare-resultsSkill
- day0-releaseSkill
- debugSkill
- deploymentSkill
- evaluationSkill
- launching-evalsSkill
- modelopt-accessing-mlflowSkill
- modelopt-benchmark-model-kernelsSkill
- modelopt-compare-resultsSkill
- modelopt-day0-releaseSkill
- modelopt-debugSkill
- modelopt-deploymentSkill
- modelopt-evaluationSkill
- modelopt-launching-evalsSkill
- modelopt-monitorSkill
- modelopt-ptqSkill
- modelopt-qadSkill
- modelopt-quant-recipe-searchSkill
- modelopt-release-cherry-pickSkill
- modelopt-run-agentxSkill
- monitorSkill
- ptqSkill
- qadSkill
- quant-recipe-searchSkill
- release-cherry-pickSkill
- run-agentxSkill
- speculative-decodingSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 0 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 35
Read the stage file for *what to do*, and the algorithm sheet for *the values to plug in*.
- Very specific: line 62
All recipes live in modelopt_recipes/general/speculative_decoding/<algorithm>.yaml.
- Very specific: line 81
If the user's algorithm has no sheet yet, the stage procedures still apply — derive the missing values from an existing launcher example …
- Very specific: line 98
Review logs: when it finishes (references/stages/review-logs.md).
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Speculative Decoding Draft-Model Training
Everything needed to take a target model from "no draft head" to "validated acceptance rate" lives in this directory. Work through the stages below in order for a new model; jump straight to a stage when you already know which one you need.
Two axes: stage and algorithm
The pipeline is the same shape for every draft-model algorithm — synthesize data, dump base-model hidden states, train the draft, benchmark acceptance rate. What changes between algorithms is which training script and recipe run, which knobs matter, and which failures are typical.
So this skill is split along those two axes, and you almost always read one file from each:
| Axis | Directory | What it holds |
| Stage | references/stages/ | The procedure — algorithm-independent |
| Algorithm | references/algorithms/ | The data sheet — scripts, recipe, knobs, thresholds, known failures |
Read the stage file for what to do, and the algorithm sheet for the values to
plug in. When a stage file says "see the algorithm sheet", it means the section of
references/algorithms/<algorithm>.md with the matching heading.
Where the code lives
The launcher scripts in tools/launcher/common/{eagle3,specdec}/ are thin wrappers around
examples/speculative_decoding/ — main.py (training), collect_hidden_states/ (dumps),
scripts/{export_hf_checkpoint,ar_validate,quantize_drafter}.py. Read a wrapper for the
flags it passes, that directory for what they do. Its README.md is also the non-launcher
path: training invoked directly, without Slurm.
Stages
| Stage | Reference | Use when |
| 1. Configure | references/stages/configure.md | Adding a model that has no pipeline YAML yet |
| 2. Review logs | references/stages/review-logs.md | A run finished (or died) and you want a pass/fail summary |
| 3. Triage | references/stages/triage.md | A task failed and you need root cause plus a fix |
| 4. Validate | references/stages/validate.md | All tasks passed and you need to confirm the acceptance rate gate |
Review-logs and triage overlap by design: review-logs is the fast sweep across all tasks, triage is the deep dive into one failing task. Start with review-logs unless the user already knows which task broke.
Algorithms
All recipes live in modelopt_recipes/general/speculative_decoding/<algorithm>.yaml.
| Algorithm | Sheet | Family |
| EAGLE3 | references/algorithms/eagle3.md | Autoregressive draft head |
| DFlash | references/algorithms/dflash.md | Block diffusion |
| DSpark | references/algorithms/dspark.md | DFlash backbone + Markov head + optional confidence head |
| Domino | references/algorithms/domino.md | DFlash backbone + GRU causal correction head |
| DFlash2 | references/algorithms/dflash2.md | DFlash backbone + sublayer convolutions + candidate selector |
| LiLiCorr | references/algorithms/lilicorr.md | DFlash backbone + candidate-lattice reranker (covers lilicorr_conv.yaml) |
Everything below EAGLE3 is a DFlash variant, not a separate pipeline: same
recipe_type: speculative_dflash, same training script, same dflash.* config
namespace, selected by dflash_architecture_config.projector_type. Read
references/algorithms/dflash.md first, then the variant's sheet for the delta.
LiLiCorr ships two recipes — lilicorr.yaml and lilicorr_conv.yaml, the latter
adding DFlash2's sublayer convolutions — but one sheet covers both.
If the user's algorithm has no sheet yet, the stage procedures still apply — derive
the missing values from an existing launcher example for that algorithm
(tools/launcher/examples/*/*/hf_*_<algorithm>.yaml) and its recipe, then write the
sheet as you go. references/algorithms/README.md defines what a sheet must contain.
End-to-end: a new model
- Confirm the algorithm and find the closest existing launcher example.
- Configure — write the pipeline YAML (
references/stages/configure.md). Preview with
--dryrun, then submit:cd tools/launcher uv run launch.py --yaml examples/<Org>/<Model>/<config>.yaml --yes- Register the job and set up monitoring per the modelopt-monitor skill.
- Review logs when it finishes (
references/stages/review-logs.md). - Triage anything that failed (
references/stages/triage.md), fix, re-run only the failed tasks onward viapipeline.task_N.skip=true. - Validate once all tasks pass (
references/stages/validate.md).
Model-support gaps that need code changes land in modelopt/torch/speculative/ and
require a separate ModelOpt PR — the pipeline YAML alone cannot fix an unrecognized
architecture.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

