modelopt-compare-results
Skill5.6k starsChanged todayApache-2.0View on GitHub
Establish baseline-vs-candidate evaluation plans, delegate missing evaluations, compare validated results, and decide quantization feasibility. Use when the user asks to compare baseline vs quantized runs, explain an accuracy drop/regression, verify whether a quantized checkpoint is acceptable, or compare NEL/MLflow evaluation outputs. Do NOT use for generic single-model evaluation without comparison intent (use modelopt-evaluation), live NEL status/debugging (use modelopt-launching-evals), or generic MLflow browsing without a comparison goal (use modelopt-accessing-mlflow).
1 other project uses a version of this.
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Most recently taken up by
- NVIDIA/Model-Optimizerword for word, first seen Nov 15, 2026
Who's talking about NVIDIA/TensorRT-Model-Optimizer
- NVIDIA/Model-Optimizer“Replace ONNX simplification package from onnxsim to onnxslim”GitHub · Oct 28, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“Fix FLOPs calculation”GitHub · Sep 29, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case casting input directly to output”GitHub · Sep 9, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case with cast producing network output”GitHub · Sep 8, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“"Unknown quantization method" when loading nvfp4 checkpoint with vLLM”GitHub · Aug 19, 2025 · 1 reactions
The biggest 5 of 45 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 45
Talked about on GitHub (45 times).
On GitHub · 45
- Oct 28, 2025Replace ONNX simplification package from onnxsim to onnxslim — NVIDIA/Model-Optimizer1 reaction · 23 comments
- Sep 29, 2025Fix FLOPs calculation — NVIDIA/Model-Optimizer1 reaction · 11 comments
- Sep 9, 2025[Autocast] Fix edge case casting input directly to output — NVIDIA/Model-Optimizer1 reaction · 10 comments
- Sep 8, 2025[Autocast] Fix edge case with cast producing network output — NVIDIA/Model-Optimizer1 reaction · 6 comments
- Aug 19, 2025"Unknown quantization method" when loading nvfp4 checkpoint with vLLM — NVIDIA/Model-Optimizer1 reaction · 3 comments
40 more on GitHub
- Mar 30, 2025[Quantization] Add TRT-ModelOpt as a Backend — huggingface/diffusers1 reaction · 50 comments
- Mar 11, 2025✨[Feature] Supporting Blackwell FP4 — pytorch/TensorRT1 reaction
- Sep 16, 2026推論最適化ツール・OSS — warabinbin/papers0 reactions
- Aug 10, 2026fold_weight crashes with AttributeError: 'NoneType' object has no attribute 'data' on Megatron models with tied word embeddings — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 5, 2025Kimi-k2 calib+export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 4, 2025Quantization — HashCitrine/sample-to-make-llm-service0 reactions
- Dec 4, 2025Diffusion Rename [TRT MO -> MO] — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Dec 3, 2025[5704162] Create a copy to avoid leaking ProcessGroup into state dict — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Dec 2, 2025fix: Fix auto quantize multi-gpu tests — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Dec 2, 2025qwen3_8b eagle3 model train and vllm ckpt export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 27, 2025Add all example e2e tests for github PR merge / nightly — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 26, 2025[Bug Fix]: Correct block_size in NVFP4_MLP_WEIGHT_ONLY_CFG to support vLLM inference — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 26, 2025fix: relax sympy version — inisis/OnnxSlim0 reactions · 1 comment
- Nov 24, 2025[None][doc] Deprecate --export_fmt flag when using ModelOpt — NVIDIA/TensorRT-LLM0 reactions · 4 comments
- Nov 24, 2025[Help wanted] CNN QAT: How to correctly export ONNX and use DeviceModel for inference — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 22, 2025fix: issues related to TensorRT-Model-Optimizer — inisis/OnnxSlim0 reactions · 2 comments
- Nov 20, 2025[2/N] Added KDLoss based AutoQuantize — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 20, 2025[1/N] Refactored AutoQuantizeSearcher to _AutoQuantizeBaseSearcher & AutoQuantizeGradientSearcher; seperated quant modules and score modules — NVIDIA/Model-Optimizer0 reactions · 11 comments
- Nov 17, 2025failed to quant the Qwen3-VL-30B-A3B — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 13, 2025Support Wan2.2 t2v diffusers quantization — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 3, 2025PyTorch geometric quantization support — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 3, 2025Failed to save quantized model if some tensors offload to CPU — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Oct 28, 2025[OMNIML-2917] export layer config using actual prefix instead of hard… — NVIDIA/Model-Optimizer0 reactions · 9 comments
- Oct 27, 2025Add convert_llama3_config_to_decilm_config + unit test — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025CodeRabbit Generated Unit Tests: Add GPU tests for Megatron quantization; review/refactor unit tests — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025📝 Add docstrings to `kaix/kvcache_mcore` — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 15, 2025FP8 Block quantize onnx export support — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Sep 15, 2025add end_process in deepseek ptq — NVIDIA/Model-Optimizer0 reactions · 10 comments
- Jul 13, 2025feat: Simplify and Improve Whisper Example — NVIDIA/TensorRT-LLM0 reactions · 3 comments
- Jul 3, 2025[future request] Support more quant for torch backend — NVIDIA/TensorRT-LLM0 reactions · 2 comments
- Jul 3, 2025🐛 [Bug] examples/dynamo/torch_compile_gpt2.py run failed — pytorch/TensorRT0 reactions · 2 comments
- Jun 15, 2025INT8 to FP8 scale conversion — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Jun 13, 2025Fix speculative decoding example — NVIDIA/Model-Optimizer0 reactions · 6 comments
- May 21, 2025`get_modelike_from_algo_cfg` doesn't accept `QuantizeAlgorithmConfig`, but it's typing says it should — NVIDIA/Model-Optimizer0 reactions · 3 comments
- May 17, 2025Decode latency of FP4 LLM is higher than expected — NVIDIA/TensorRT-LLM0 reactions · 1 comment
- May 7, 2025Error in torch_compile.ipynb — Project-MONAI/tutorials0 reactions · 1 comment
- May 6, 2025当我使用增量训练的时候,只改变修改一个feature的zch_size的大小,其他所有配置都不改变,会出错。 — alibaba/TorchEasyRec0 reactions
- Apr 30, 2025INT8 quantization for HiFi-GAN vocoder -- performance issue — NVIDIA/TensorRT0 reactions · 7 comments
- Mar 24, 2025bert model split into many layers after int8 quantization — NVIDIA/TensorRT0 reactions · 13 comments
- Sep 8, 2024urlopen error retrieval incomplete in modules/autoencoder_mednist.ipynb — Project-MONAI/tutorials0 reactions
Other versions1 project uses a version of this.
Word for word · 1
- NVIDIA/Model-Optimizer★ 5.6k100% same
Edited · 0
None found.
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository33 other files it gives its agents.
- AGENTS.mdAGENTS.md
- tools/debugger/CLAUDE.mdCLAUDE.md
- tools/launcher/CLAUDE.mdCLAUDE.md
- accessing-mlflowSkill
- benchmark-model-kernelsSkill
- commonSkill
- compare-resultsSkill
- day0-releaseSkill
- debugSkill
- deploymentSkill
- evaluationSkill
- launching-evalsSkill
- modelopt-accessing-mlflowSkill
- modelopt-benchmark-model-kernelsSkill
- modelopt-day0-releaseSkill
- modelopt-debugSkill
- modelopt-deploymentSkill
- modelopt-evaluationSkill
- modelopt-launching-evalsSkill
- modelopt-monitorSkill
- modelopt-ptqSkill
- modelopt-qadSkill
- modelopt-quant-recipe-searchSkill
- modelopt-release-cherry-pickSkill
- modelopt-run-agentxSkill
- modelopt-speculative-decodingSkill
- monitorSkill
- ptqSkill
- qadSkill
- quant-recipe-searchSkill
- release-cherry-pickSkill
- run-agentxSkill
- speculative-decodingSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 6 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 9
Use this to plan and complete a baseline-vs-candidate comparison.
- Distinctive: line 19
If a required baseline or candidate evaluation is missing, delegate to the evaluation skill to create, run, and verify it.
- Very specific: line 34
Use the evaluation skill's references/run-validation.md to perform the
- Distinctive: line 41
Follow Assess results to compute deltas and uncertainty.
- Distinctive: line 50
Before treating a baseline-vs-quantized delta as a model quality result, verify the validated runs are comparable:
- Distinctive: line 76
If the requested gate is relative to BF16 and no BF16 baseline is available, report that gate as inconclusive.
- Distinctive: line 83
If any item differs, either rerun with matched settings or label the result as not an apples-to-apples quantization comparison.
- Very specific: line 112
Never report acceptable when external baseline sanity failed.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Compare Results
Use this to plan and complete a baseline-vs-candidate comparison. The baseline is the reference checkpoint, and the candidate is the checkpoint whose accuracy change is being measured, typically a further quantized version of the baseline.
Workflow
- Establish the candidate checkpoint/run and the matching baseline. Infer the baseline from the PTQ source model/checkpoint in the workspace or config used to create the candidate. If it cannot be inferred, ask the user for the baseline checkpoint or an existing baseline invocation/run path.
- If a required baseline or candidate evaluation is missing, delegate to the evaluation skill to create, run, and verify it. The companion evaluation config should match benchmark versions, task configs, serving args, token limits, dataset setup, credentials, cluster, and container as closely as possible; change only the model/checkpoint and checkpoint-specific serving or quantization flags.
- Fetch the baseline and candidate task list, configs, score artifacts, and logs. If the user provides MLflow runs or invocation IDs, use the accessing-mlflow skill to fetch configs and artifacts.
- Confirm each run passed evaluation Step 9, "Verify completed evaluation run", before comparing scores. If not, validate logs, server health, judge/code-execution status, sample accounting, and reasoning parsing before computing deltas.
- For each task, use the canonical score field from the matching evaluation
skill task recipe,
recipes/tasks/<task>.md, under Score Extraction. - Use the evaluation skill's
references/run-validation.mdto perform the External Baseline Sanity Check. Record each source URL, protocol difference, and task status before applying the candidate-delta gate. A failed baseline blocks a success verdict; correct and rerun it first. If no credible comparable reference exists, label the baseline externally unverified rather than claiming the check passed, then continue using the validated measured baseline. - Follow Assess results to compute deltas and uncertainty; mark unavailable uncertainty explicitly.
- Report comparability, external baseline sanity, and quantized-feasibility verdicts before interpreting the delta as model quality. If the user did not provide an acceptance threshold, report feasibility as inconclusive instead of inventing one.
Comparability Checklist
Before treating a baseline-vs-quantized delta as a model quality result, verify the validated runs are comparable:
- Prompt text, system prompt, chat template, and rendered messages match.
- Task name, benchmark version, dataset split, container, harness, and task fragment match.
- Generation settings match, including temperature, top_p, top_k, max tokens, stop strings, repetition detection, chat-template kwargs, reasoning mode/budget, and task-specific overrides.
- Reasoning traces are enabled, disabled, parsed, stripped, or ignored consistently between runs.
- The number of evaluated and scored samples/repeats matches for each task and split.
- Judge-backed or simulator-backed tasks use the same judge/user model, endpoint class, prompt, and scoring config.
- The same accuracy metric and score field is used for both runs.
- Timeout policies, effective limits, and failure scoring/exclusions match.
Apply Timeout and Output-Limit Accounting in the evaluation skill's
references/run-validation.mdto both runs; matching limits alone cannot rule out serving-speed effects on scores. - Baseline precision matches the gate. A
<1pp vs BF16gate requires a true full-precision (BF16) baseline. Many models ship natively quantized (e.g. INT4W4A16or block-wise FP8) with no BF16 release — a quant-to-quant comparison against the released precision (e.g. INT4 vs NVFP4, as for Kimi-K2.6) is still a valid result. State which precision the baseline is and apply only an acceptance criterion explicitly defined for that precision. If the requested gate is relative to BF16 and no BF16 baseline is available, report that gate as inconclusive; do not reinterpret it as an FP8/INT4 gate.
For SciCode, keep num_repeats: 1 and require at least 8 runs per side, comparing
the two means — see the evaluation skill's recipes/tasks/aa/scicode.md. Fewer
than 8 valid runs on a side is INDETERMINATE, not a delta.
If any item differs, either rerun with matched settings or label the result as not an apples-to-apples quantization comparison.
These checks compare the baseline and candidate to each other. The external
baseline check in the evaluation skill's references/run-validation.md
separately tests whether the baseline's absolute score is credible; both guards
must be reported.
Report Format
Include:
- Baseline and candidate identifiers.
- Per-task metric path, baseline and candidate scores with SEs, signed loss
(
Candidate - Baselinefor accuracy; negative is worse), confidence intervals and method, sample/run counts, and an Assessment column. Mark unavailable uncertainty explicitly; keep the assessment concise, not a statistics tutorial. - Per-task external reference score, source URL, known protocol differences,
percentage-point difference, and sanity status (
verified,failed, orexternally unverified). - Comparability status for prompt/template, generation settings, sample counts, reasoning handling, judge/simulator setup, and score field.
- Per-task timeout and output-limit counts/rates for both runs, explicit
denominators, telemetry coverage, effective limits, and unresolved effects on
the score. Unknown accounting or unresolved infrastructure effects prevent an
acceptablequantization-feasibility verdict. - Comparability verdict: comparable, not comparable, or inconclusive.
- Quantization feasibility verdict: acceptable, not acceptable, or inconclusive.
Never report
acceptablewhen external baseline sanity failed. An externally unverified baseline does not blockacceptable; apply the candidate-delta gate and report the missing external corroboration.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

