modelopt-evaluation
Skill5.6k starsChanged todayApache-2.0View on GitHub
Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "run nel". Handles deployment, config generation, and evaluation execution. Always load when validating completed runs or reporting evaluation scores, including standalone modelopt-launching-evals analysis. Not for quantizing models (use modelopt-ptq), standalone serving (use modelopt-deployment), or comparing completed baseline-vs-quantized results (use modelopt-compare-results).
- Reads credentials
1 other project uses a version of this.
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Most recently taken up by
- NVIDIA/Model-Optimizerword for word, first seen Nov 15, 2026
Who's talking about NVIDIA/TensorRT-Model-Optimizer
- NVIDIA/Model-Optimizer“Replace ONNX simplification package from onnxsim to onnxslim”GitHub · Oct 28, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“Fix FLOPs calculation”GitHub · Sep 29, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case casting input directly to output”GitHub · Sep 9, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“[Autocast] Fix edge case with cast producing network output”GitHub · Sep 8, 2025 · 1 reactions
- NVIDIA/Model-Optimizer“"Unknown quantization method" when loading nvfp4 checkpoint with vLLM”GitHub · Aug 19, 2025 · 1 reactions
The biggest 5 of 45 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 45
Talked about on GitHub (45 times).
On GitHub · 45
- Oct 28, 2025Replace ONNX simplification package from onnxsim to onnxslim — NVIDIA/Model-Optimizer1 reaction · 23 comments
- Sep 29, 2025Fix FLOPs calculation — NVIDIA/Model-Optimizer1 reaction · 11 comments
- Sep 9, 2025[Autocast] Fix edge case casting input directly to output — NVIDIA/Model-Optimizer1 reaction · 10 comments
- Sep 8, 2025[Autocast] Fix edge case with cast producing network output — NVIDIA/Model-Optimizer1 reaction · 6 comments
- Aug 19, 2025"Unknown quantization method" when loading nvfp4 checkpoint with vLLM — NVIDIA/Model-Optimizer1 reaction · 3 comments
40 more on GitHub
- Mar 30, 2025[Quantization] Add TRT-ModelOpt as a Backend — huggingface/diffusers1 reaction · 50 comments
- Mar 11, 2025✨[Feature] Supporting Blackwell FP4 — pytorch/TensorRT1 reaction
- Sep 16, 2026推論最適化ツール・OSS — warabinbin/papers0 reactions
- Aug 10, 2026fold_weight crashes with AttributeError: 'NoneType' object has no attribute 'data' on Megatron models with tied word embeddings — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 5, 2025Kimi-k2 calib+export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Dec 4, 2025Quantization — HashCitrine/sample-to-make-llm-service0 reactions
- Dec 4, 2025Diffusion Rename [TRT MO -> MO] — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Dec 3, 2025[5704162] Create a copy to avoid leaking ProcessGroup into state dict — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Dec 2, 2025fix: Fix auto quantize multi-gpu tests — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Dec 2, 2025qwen3_8b eagle3 model train and vllm ckpt export — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 27, 2025Add all example e2e tests for github PR merge / nightly — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 26, 2025[Bug Fix]: Correct block_size in NVFP4_MLP_WEIGHT_ONLY_CFG to support vLLM inference — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 26, 2025fix: relax sympy version — inisis/OnnxSlim0 reactions · 1 comment
- Nov 24, 2025[None][doc] Deprecate --export_fmt flag when using ModelOpt — NVIDIA/TensorRT-LLM0 reactions · 4 comments
- Nov 24, 2025[Help wanted] CNN QAT: How to correctly export ONNX and use DeviceModel for inference — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 22, 2025fix: issues related to TensorRT-Model-Optimizer — inisis/OnnxSlim0 reactions · 2 comments
- Nov 20, 2025[2/N] Added KDLoss based AutoQuantize — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 20, 2025[1/N] Refactored AutoQuantizeSearcher to _AutoQuantizeBaseSearcher & AutoQuantizeGradientSearcher; seperated quant modules and score modules — NVIDIA/Model-Optimizer0 reactions · 11 comments
- Nov 17, 2025failed to quant the Qwen3-VL-30B-A3B — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 13, 2025Support Wan2.2 t2v diffusers quantization — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Nov 3, 2025PyTorch geometric quantization support — NVIDIA/Model-Optimizer0 reactions · 1 comment
- Nov 3, 2025Failed to save quantized model if some tensors offload to CPU — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Oct 28, 2025[OMNIML-2917] export layer config using actual prefix instead of hard… — NVIDIA/Model-Optimizer0 reactions · 9 comments
- Oct 27, 2025Add convert_llama3_config_to_decilm_config + unit test — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025CodeRabbit Generated Unit Tests: Add GPU tests for Megatron quantization; review/refactor unit tests — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 25, 2025📝 Add docstrings to `kaix/kvcache_mcore` — NVIDIA/Model-Optimizer0 reactions · 2 comments
- Sep 15, 2025FP8 Block quantize onnx export support — NVIDIA/Model-Optimizer0 reactions · 3 comments
- Sep 15, 2025add end_process in deepseek ptq — NVIDIA/Model-Optimizer0 reactions · 10 comments
- Jul 13, 2025feat: Simplify and Improve Whisper Example — NVIDIA/TensorRT-LLM0 reactions · 3 comments
- Jul 3, 2025[future request] Support more quant for torch backend — NVIDIA/TensorRT-LLM0 reactions · 2 comments
- Jul 3, 2025🐛 [Bug] examples/dynamo/torch_compile_gpt2.py run failed — pytorch/TensorRT0 reactions · 2 comments
- Jun 15, 2025INT8 to FP8 scale conversion — NVIDIA/Model-Optimizer0 reactions · 6 comments
- Jun 13, 2025Fix speculative decoding example — NVIDIA/Model-Optimizer0 reactions · 6 comments
- May 21, 2025`get_modelike_from_algo_cfg` doesn't accept `QuantizeAlgorithmConfig`, but it's typing says it should — NVIDIA/Model-Optimizer0 reactions · 3 comments
- May 17, 2025Decode latency of FP4 LLM is higher than expected — NVIDIA/TensorRT-LLM0 reactions · 1 comment
- May 7, 2025Error in torch_compile.ipynb — Project-MONAI/tutorials0 reactions · 1 comment
- May 6, 2025当我使用增量训练的时候,只改变修改一个feature的zch_size的大小,其他所有配置都不改变,会出错。 — alibaba/TorchEasyRec0 reactions
- Apr 30, 2025INT8 quantization for HiFi-GAN vocoder -- performance issue — NVIDIA/TensorRT0 reactions · 7 comments
- Mar 24, 2025bert model split into many layers after int8 quantization — NVIDIA/TensorRT0 reactions · 13 comments
- Sep 8, 2024urlopen error retrieval incomplete in modules/autoencoder_mednist.ipynb — Project-MONAI/tutorials0 reactions
Other versions1 project uses a version of this.
Word for word · 1
- NVIDIA/Model-Optimizer★ 5.6k100% same
Edited · 0
None found.
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository33 other files it gives its agents.
- AGENTS.mdAGENTS.md
- tools/debugger/CLAUDE.mdCLAUDE.md
- tools/launcher/CLAUDE.mdCLAUDE.md
- accessing-mlflowSkill
- benchmark-model-kernelsSkill
- commonSkill
- compare-resultsSkill
- day0-releaseSkill
- debugSkill
- deploymentSkill
- evaluationSkill
- launching-evalsSkill
- modelopt-accessing-mlflowSkill
- modelopt-benchmark-model-kernelsSkill
- modelopt-compare-resultsSkill
- modelopt-day0-releaseSkill
- modelopt-debugSkill
- modelopt-deploymentSkill
- modelopt-launching-evalsSkill
- modelopt-monitorSkill
- modelopt-ptqSkill
- modelopt-qadSkill
- modelopt-quant-recipe-searchSkill
- modelopt-release-cherry-pickSkill
- modelopt-run-agentxSkill
- modelopt-speculative-decodingSkill
- monitorSkill
- ptqSkill
- qadSkill
- quant-recipe-searchSkill
- release-cherry-pickSkill
- run-agentxSkill
- speculative-decodingSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 2 are distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 21
If MODELOPT_WORKSPACE_ROOT is set, use the common skill's workspace-management.md and reuse existing workspaces (PTQ → checkpoint smoke …
- Very specific: line 49
Read references/nel-next.md (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe …
- Distinctive: line 50
Read model-card research for generation settings.
- Very specific: line 52
Run modelopttools:eval-config (Step 3b) to write the AWS-sandbox creds + harbor infra rows (${NEL_NEXT_EVAL_IMAGE}, …
- Very specific: line 54
Before reporting a full-run score, apply references/run-validation.md, including its benchmark-specific checks.
- Very specific: line 69
Read references/gym.md (pinned launcher, gym prepare/reap machinery, pin↔container coupling, preflight gaps, failure modes) + …
- Very specific: line 90
Run both dry-run and launch through "$SKILL_DIR/scripts/nel-gym.sh".
- Distinctive: line 99
Read only the sections needed for the current stage.
- Very specific: line 131
If the user asks to avoid doom loops (looping reasoning), or a canary under a long max_new_tokens runs far longer than expected, read …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
NeMo Evaluator Launcher Assistant
Guide the user through creating NEL YAML configs, running evaluations, and monitoring progress. NEL manages serving for canaries and full runs; see Step 3's serving-handoff check for external endpoints.
Completed-run analysis
For completed runs, skip launch/config; use launching-evals for artifacts/analysis and apply run-validation.md before scores. Its policy overrides blanket uncapping and fixed token/context limits.
Workspace integration
If MODELOPT_WORKSPACE_ROOT is set, use the common skill's workspace-management.md and reuse existing workspaces (PTQ → checkpoint smoke test → NEL evaluation; carry compatibility requirements into NEL's deployment config).
Workflow
- [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT set)
- [ ] Step 1: Check `nel` install + existing config; set up `.env` (+ `modelopttools:eval-config` for judge-scored runs)
- [ ] Step 2: Build base config (5-question flow OR shortcut)
- [ ] Step 3: Configure deployment (model path, params, cross-check)
- [ ] Step 4: Fill remaining ??? values
- [ ] Step 5: Confirm tasks (iterative)
- [ ] Step 6: Multi-node (if needed)
- [ ] Step 7: Interceptors (if needed)
- [ ] Step 7.5: Container auth (SLURM private images)
- [ ] Step 8: Dry-run → canary → full run
- [ ] Step 9: Verify completed runnel-next path (Terminal-Bench 2.x, SWE-bench, …) — branch here FIRST
A few agentic AA benchmarks do not run on the currently validated
nemo-evaluator-launcher 0.2.6 path (Steps 1–9 don't apply). They run on nel-next
(nemo-evaluator[harbor] 0.4.x) — a separate package, CLI (nel eval run), -O
overrides, and services/benchmarks/cluster/output schema. If the user asks
for one, do not add it to a 0.2.6 evaluation.tasks list — instead:
- Read
references/nel-next.md(shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark reciperecipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md; start fromrecipes/examples/example_eval_next.yaml. Read model-card research for generation settings; consult task-specific policy before selecting published budgets or fallbacks. - Isolated nel-next venv:
"$SKILL_DIR/scripts/nel-next.sh" --setup-only(keeps 0.2.6neluntouched). - Run
modelopttools:eval-config(Step 3b) to write the AWS-sandbox creds + harbor infra rows (${NEL_NEXT_EVAL_IMAGE},${HARBOR_*_ECR_REPOSITORY},${HARBOR_ECS_REGION}) into.env; always include theoutput.export_config.mlflowblock. - Dry-run → canary → full (
nel-next.sh eval run), then push to MLflow — SLURM doesn't auto-export, so runnel-next.sh mlflow-push -r <run_id> -c <cfg>after (config-driven; seereferences/nel-next.md). - Before reporting a full-run score, apply
references/run-validation.md, including its benchmark-specific checks.
Steps 1–9 below are currently validated with 0.2.6 — use them for everything else.
MRCR (NeMo Gym simple_agent) path — branch here too
MRCR does run on the currently validated 0.2.6 nel launcher (as a nemo_gym
task, not nel-next), so Steps 1–9 apply — but it is mechanically special and
standalone (one gym eval per config; never mix it with aa/ tasks). It is
simple as gym tasks go: simple_agent, no judge — deterministic prefix-gated
grading, HF_TOKEN the only secret. Not an AA benchmark — never generate it
for an "AA" request. If the user asks for MRCR:
- Read
references/gym.md(pinned launcher, gym prepare/reap machinery, pin↔container coupling, preflight gaps, failure modes) +recipes/tasks/gym/mrcr.md; start fromrecipes/examples/gym/example_mrcr.yaml(1M variant, like the golden). - Pick the variant first (
config_n3_1m/config_n3_128k/config) — it sets the context cap, dataset and metric prefix; the three are not comparable; set it in bothdata_prep_paramsandcollect_rollout_params. Take the largest variant within the checkpoint's trained context — see the recipe. .env:HF_TOKEN(dataset + n3 tokenizer are gated) plusNEMO_EVALUATOR_TRUST_PRE_CMD=1(thepre_cmdinstallstiktoken+transformers; prepare fails without it) andNEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1(nemo_gymis not in the FDF map).- MRCR needs a git-backed Gym image. The pin is newer than any image's baked
Gym and must apply, so the template's
container:is???and the bootstrap exits 1 on a non-git/opt/Gym(the publiceval-factory/nemo-gym:*images). NVIDIA-internal:modelopttools:eval-configStep 3d names a working image. - Long-context deploy: for 1M,
--max-model-len 1100000+VLLM_ALLOW_LONG_MAX_MODEL_LEN=1; for 128K,--max-model-lenat the checkpoint's trained context (room for the answer above 131,072) and no override. 1M also needsgpu_memory_utilization: 0.95. Both: multi-instance fan-out; never cap output tokens; report the needle-count strata alongsidepass@1/accuracy. - Run both dry-run and launch through
"$SKILL_DIR/scripts/nel-gym.sh"; it enforces the currently validated 0.2.6 launcher even ifnelon PATH is stale and avoids an unsetNEL_INVOCATION_IDfailure before client startup. Canary withlimit_samples— the MRCR template forwards it as++limit(see the recipe's Canary section); the prepare pass still runs in full.
Detailed launcher instructions live in launcher-workflow.md. Read only the sections needed for the current stage; retain the dry-run → canary → full-run gates. Existing configs must pass Step 3's serving-handoff check before Step 8. Paths in that reference are relative to this skill directory.
Step 1 — Prerequisites
Read Step 1.
Step 2 — Build base config (when not using shortcut)
Read Step 2.
Step 3 — Configure deployment
Read Step 3.
Step 4 — Fill remaining ??? values
Read Step 4.
Step 5 — Confirm tasks (iterative)
Read Step 5.
Step 6 — Multi-node
Read Step 6.
Step 7 — Interceptors
Read Step 7.
If the user asks to avoid doom loops (looping reasoning), or a canary under a long
max_new_tokens runs far longer than expected, read doom_loops.md.
Step 7.5 — Container registry auth (SLURM private images only)
Read Step 7.5.
Step 8 — Run evaluation (gated dry-run → canary → full)
Read Step 8.
Step 9 — Verify completed run
Apply run-validation.md's Bounded Evaluation-Failure
Policy and Timeout and Output-Limit Accounting before reporting scores. For
comparisons, also apply its External Baseline Sanity Check, then use
modelopt-compare-results.
Issues: https://github.com/NVIDIA-NeMo/Evaluator/issues · https://github.com/NVIDIA-NeMo/Evaluator/discussions
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

