nnpa-performance
Skill1.1k starsChanged 12 days agoApache-2.0View on GitHub
Use when a model is slow or not offloading on IBM Z NNPA (--maccel=NNPA): confirm the accelerator is active, see what offloaded and why ops stayed on CPU, check op limits, handle dynamic shapes, quantize, and tune placement and fusion.
Trending
Quiet since December.about the whole repository
Who's talking about onnx/onnx-mlir
- elixir-nx/nx“Import of MLIR modules”GitHub · Mar 5, 2024 · 4 reactions
- PyTorchKR/pytorchcore-kr“더 나은 질의 응답 구조를 위한 Discussion 활용 방안”GitHub · Jan 20, 2025 · 2 reactions
- llvm/torch-mlir“[RFC] Support for an embedded build of Torch-MLIR”GitHub · Sep 23, 2022 · 2 reactions
- Samsung/ONE“[circle-mlir] Update onnx-mlir to official release”GitHub · Jun 8, 2025 · 1 reactions
- Xilinx/onnx-mlir“ONNX to TOSA Legalization Upstream tracker”GitHub · Apr 20, 2023 · 1 reactions
The biggest 5 of 22 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 22
Talked about on GitHub (22 times).
On GitHub · 22
- Mar 5, 2024Import of MLIR modules — elixir-nx/nx4 reactions · 2 comments
- Jan 20, 2025더 나은 질의 응답 구조를 위한 Discussion 활용 방안 — PyTorchKR/pytorchcore-kr2 reactions
- Sep 23, 2022[RFC] Support for an embedded build of Torch-MLIR — llvm/torch-mlir2 reactions · 4 comments
- Jun 8, 2025[circle-mlir] Update onnx-mlir to official release — Samsung/ONE1 reaction · 4 comments
- Apr 20, 2023ONNX to TOSA Legalization Upstream tracker — Xilinx/onnx-mlir1 reaction
17 more on GitHub
- Dec 16, 2025[ttl] Automate elementwise op generation from specifications — tenstorrent/tt-lang0 reactions · 1 comment
- Jul 29, 2025Missing Symbols due to visibility setting change — onnx/onnx0 reactions · 2 comments
- Nov 19, 2024Add fancier CI badges showing number of tests and/or test coverage — iree-org/iree-test-suites0 reactions · 2 comments
- Aug 6, 2024Standardize rewriter tests; testing facilities — microsoft/onnxscriptsays it tested it0 reactions · 1 comment
- May 8, 2024Fix macos build by pinning version — Xilinx/onnx-mlir0 reactions
- Mar 30, 2024Include the ".onnxtext" extension in supported serialization format — onnx/onnx0 reactions · 1 comment
- Dec 20, 2023add filecheck test for add.onnx — xdslproject/xdsl0 reactions · 7 comments
- Dec 1, 2023Failed to build with the use of --cpp-implementation — protocolbuffers/protobuf0 reactions
- Oct 27, 2023Bring in the fix for empty slice in canonicalization — Xilinx/onnx-mlir0 reactions
- Oct 25, 2023Cherry-picked upstream PR: Add conversion for DivOp ONNX to TOSA (#2207) — Xilinx/onnx-mlir0 reactions
- Aug 26, 2023Build error on M1 MacOS machine — onnx/onnx0 reactions · 3 comments
- Oct 18, 2022ONNX MLIR exporter — nod-ai/onnx-mlir-exporter0 reactions · 1 comment
- Oct 9, 2022[Feature request] Machine Readable definition of Operations (Schema file instead of Markdown or cc file) — onnx/onnx0 reactions · 14 comments
- Jun 28, 2022Can we have newer opset of bidaf-9 model (including opset 9 or newer onnx.Scan op)? — onnx/models0 reactions · 1 comment
- Jun 5, 2022Broken Links — onnx/onnx.github.io0 reactions · 3 comments
- May 12, 2022Update the onnx-mlir version — DependableSystemsLab/LLTFI0 reactions · 2 comments
- Jun 30, 2020Error when trying to lower alex net onnx model to MLIR — onnx/models0 reactions · 1 comment
In this repository14 other files it gives its agents.
- AGENTS.mdAGENTS.md
- add-fusion-patternSkill
- add-new-onnx-operationSkill
- add-onnx-op-loweringSkill
- add-onnx-op-to-dialectSkill
- add-onnx-transform-passSkill
- coding-conventionsSkill
- debug-compilationSkill
- debug-type-catalogueSkill
- ground-lit-testsSkill
- nnpa-internalsSkill
- onnx-mlir-internalsSkill
- upgrade-llvm-mlirSkill
- using-torch-onnxmlir-with-huggingfaceSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 16
Do this before investigating any individual op, because getting it wrong rejects the *whole model* rather than one op, and it does so …
- Distinctive: line 29
Every NNPA compile already prints a fallback summary — nothing to turn on:
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
<!-- Generated from the onnx-mlir knowledge base (skills/nnpa-performance.md), verified against onnx-mlir@b505d64b. Do not edit here: see .agents/skills/README.md. -->Get Performance on NNPA
Ordered walkthrough for making a model run fast on IBM's NNPA accelerator. Getting an op onto NNPA is the first lever and the one that fails most often, so steps 1–5 cover offload; steps 6–8 cover the throughput levers that only matter once your ops are actually on the accelerator. Every check up to step 8 is compile-time — none of them need a run on real z hardware.
Step 1: Confirm the accelerator is actually active
Do this before investigating any individual op, because getting it wrong rejects the whole model rather than one op, and it does so silently:
onnx-mlir -O3 --march=z16 --maccel=NNPA mymodel.onnx # NNPA generation M14
onnx-mlir -O3 --march=z17 --maccel=NNPA mymodel.onnx # NNPA generation M15--maccel=NNPAselects the accelerator path;--marchselects the generation. Onlyz16/arch14(M14) andz17/arch15(M15) map to an NNPA level.--maccel=NNPAwith--marchomitted, or set to z14/z15, is not an error. The level resolves to none, every op fails its level check, and the entire model compiles to CPU with no diagnostic. See stack/compiler.md for what each value gets you.- NNPA flags exist in
--helpat all only in a build configured with-DONNX_MLIR_ACCELERATORS=NNPA. If--maccel=NNPAis rejected outright, the accelerator isn't compiled in — see stack/accelerators.md.
Step 2: See what offloaded
Every NNPA compile already prints a fallback summary — nothing to turn on:
[Warning] There are 2 onnx.Conv, 1 onnx.MatMul operations that run on CPU (not accelerated by NNPA). To get more information, recompile the model with the --onnx-op-stats option.Two limits on what that tells you: it counts only nine heavy op types (Conv, Gemm, GRU, LSTM, MatMul, MatMulInteger, QLinearMatMul, RNN, Softmax), and it does not say why. For the full picture — ops that offloaded appear as zhigh.*, ops that didn't stay onnx.*:
onnx-mlir -O3 --march=z16 --maccel=NNPA --onnx-op-stats TXT mymodel.onnx
onnx-mlir -O3 --march=z16 --maccel=NNPA --EmitZHighIR mymodel.onnx # writes .onnx.mlirStep 3: Ask why an op stayed on CPU
--opt-report=NNPAUnsupportedOps is the flag built for this question, and the one the [Warning] line above does not mention:
onnx-mlir -O3 --march=z16 --maccel=NNPA --opt-report=NNPAUnsupportedOps mymodel.onnx \
| grep NNPA-UNSUPPORTEDOPS-REPORTOne line per rejected op, four comma-separated fields:
==NNPA-UNSUPPORTEDOPS-REPORT==, onnx.MaxPoolSingleOut, resnetv18_pool0_fwd, ceil_mode not supported
==NNPA-UNSUPPORTEDOPS-REPORT==, onnx.Gemm, fc_1, `alpha` (2.000000) must be 1.0The third field is the node name from your own model (the onnx_node_name attribute), so it is searchable in Netron or against your graph — that is what turns "a Gemm was rejected" into "this Gemm was rejected." Many reasons also name the flag that relaxes them, so read the message text rather than just the category.
Two preconditions, one of which is a trap
--maccel=NNPAis required, and theNNPAUnsupportedOpsvalue only exists in an NNPA-enabled build. Otherwise--opt-reportoffers onlyParallelandSimd.- The report is silent below
--EmitMLIR. The reasons come from the ONNX-to-ZHigh lowering pass, which only runs at emission targets from--EmitMLIRup.--EmitONNXIRgives you no output at all — not "no rejections found," but nothing, which looks identical. The default target (--EmitLib),--EmitMLIR,--EmitLLVMIRand--EmitZHighIRall work.--onnx-op-statshas the same requirement and says so in its help text; this flag does not.
Not a problem, in case you go looking: device placement runs the same legality checks earlier with reporting deliberately switched off, and the lowering pass switches it back on — so a JSON device-placement config does not cost you the report, and you don't get duplicate lines. A silent report is always the emission-target trap above.
Step 4: Check the op's documented limits before changing anything
There is a per-op table with a Limitations column and a Minimum NNPA Level, which is the authoritative answer to "can this op offload at all, and under what conditions":
- repos/onnx-mlir/supported-ops-nnpa.md — start here. This wiki carries the full table, all ops and every limitation, at a pinned commit. It also explains something the generated table doesn't: a limitation reading "must be static" is a compile-time verifiability requirement of onnx-mlir's own legality check, not a hardware restriction — zDNN only ever sees concrete numbers at run time — which is precisely why step 5's escape hatch exists.
- docs/SupportedONNXOps-NNPA.md and docs/SupportedOps-NNPA-supplement.md — upstream on
main, generated from the source. Reach for these when you need something newer than the pin above, but note the direction of the mismatch:maincan be ahead of the onnx-mlir you actually have installed, so an op listed there may not offload in your build. The table shipped in your own install (docs/of the release you built from) is the version that matches your compiler.
In the level column, z16 - ^ means z16 and up; an op marked z17 needs Telum II.
Read your op's row before reaching for a flag. Conv, for example, documents six conditions: auto_pad must be NOTSET, VALID, or SAME_UPPER (and with NOTSET, pads must produce valid or same-upper padding); height and width dimensions must be static; group must be 1; dilations must be 1; input and output must be 4D (N×C×H×W); and kernel_shape must be static. Several of those are export-side facts about how the model was authored, not things a compile flag can change — a grouped or dilated convolution is outside zDNN's contract, and no flag makes it eligible.
Also note the hardware envelope: NNPA computes in DLFLOAT16, so F32/F16 tensors are converted and integer or F64 tensors are ineligible; rank must be 1–4; and each generation has dimension-index and tensor-size limits (large enough for normal models, but a model that exceeds them falls back to CPU).
Step 5: Make dynamic shapes acceptable
Dynamic dimensions are the most common fixable rejection. There are two different tools, and the difference matters:
--shapeInformation — pin the shape. Changes what gets compiled: the dimension becomes static, so the normal checks pass on their own merits.
onnx-mlir -O3 --march=z16 --maccel=NNPA --shapeInformation 0:1x3x224x224 mymodel.onnxThe syntax is INPUT_ID:D1xD2x...xDn, where the id is an input index from 0 (a range like 5-17, or -1 for all inputs) and -1 as a dimension leaves it dynamic. Re-run step 3 afterwards — pinning one input often clears several rejections at once, because the unknown dimension was propagating downstream.
--nnpa-disable-shape-restriction — keep the shape dynamic and assert it will be fine. The shape stays dynamic in the compiled model; the compiler skips the static height/width checks and defers validation to run time, on your word that the actual values will satisfy NNPA's constraints:
onnx-mlir -O3 --march=z16 --maccel=NNPA --nnpa-disable-shape-restriction mymodel.onnxThis is the flag the rejection message itself points you at:
==NNPA-UNSUPPORTEDOPS-REPORT==, onnx.Conv, conv_1, Height and/or width have dynamic dimensions. They are not supported for the VALID padding mode. Use --nnpa-disable-shape-restriction to assume dynamic shapes will satisfy NNPA constraints at runtime.It affects the dynamic height/width checks in the Conv2D parameter restrictions (so Conv) and in the common pooling legality check (so MaxPool/AveragePool). Use it when you know the spatial dimensions are fixed at inference even though the model declares them dynamic — a model exported with a symbolic H/W that you always call at one resolution is the standard case.
The trade: you are removing a check, not satisfying it. Nothing re-validates the assertion for you, so a run whose actual height/width violates the zDNN constraint is your problem, not the compiler's. Prefer --shapeInformation when the shape really is fixed — it makes the same fact explicit in the compiled model instead of implicit in a promise — and reach for --nnpa-disable-shape-restriction when the model must stay shape-dynamic for other reasons.
Step 6: Quantize (z17 only)
8-bit quantization is a throughput lever on Telum II: it requires -march=z17 -maccel=NNPA and is not available on z16. Two very different situations:
Your model is already quantized (by ONNX Runtime or similar, so it contains 8-bit ops): no flags needed. The compiler selects suitable 8-bit ops for NNPA on its own, and both statically and dynamically quantized models are supported.
Your model is float32: the compiler can quantize during compilation, dynamically only. It is off by default, so "disabling" it means omitting the flag:
# let the compiler choose scheme and op types
onnx-mlir -O3 --march=z17 --maccel=NNPA --nnpa-quant-dynamic mymodel.onnx
# choose the scheme yourself
onnx-mlir -O3 --march=z17 --maccel=NNPA --nnpa-quant-dynamic=asymActivation,symWeight mymodel.onnxWhich scheme to use — the four values are symWeight, asymWeight, symActivation, asymActivation, combined comma-separated:
- Symmetric is usually faster but less accurate; asymmetric is usually more accurate. The recommended practice is to try combinations on your own model rather than assume.
asymActivation,symWeightis the useful starting pair — and not only for the accuracy/speed balance: asymmetric quantization for weights is documented as not yet supported, even thoughasymWeightis accepted as a flag value. Passing it does not get you a working asymmetric-weight path.- Passing
--nnpa-quant-dynamicwith no value lets the compiler pick, and that empty value wins — once it is in the list the other options are ignored.
--nnpa-quant-op-types selects which op types get quantized. The published doc's example shows MatMul,Conv, but the flag's own help text says only MatMul is currently supported — so treat MatMul as the real answer and let the compiler decide by default. Two further flags move the dynamic-quantize-linear work off the accelerator when that turns out faster: --nnpa-cpu-dql (the whole DQL on CPU) and --nnpa-cpu-dql-scale (just the scale/offset computation).
Current limitations worth knowing before you invest: per-tensor only (no per-channel), 8-bit signed integer target, no blocked quantization, and the asymmetric-weight gap above. Full detail: docs/Quantization-NNPA.md, plus stack/accelerators.md for the selective nnpa_ops_config JSON route.
Step 7: Tune placement and data movement
Once ops are eligible, two things decide whether offloading them actually pays.
Placement. The default heuristic offloads everything eligible, even ops that would run faster on CPU:
--nnpa-placement-heuristic=QualifyingOps # default: all qualifying ops
--nnpa-placement-heuristic=FasterOps # only ops faster on NNPA
--nnpa-placement-heuristic=FasterOpsWSU # FasterOps, counting stick/unstick cost
--nnpa-placement-heuristic=MuchFasterOpsWSU # only when significantly fasterThe WSU variants price in stick/unstick, which is the point: moving a tensor into and out of NNPA's DLFLOAT16 format is not free, and a cheap op sandwiched between conversions can lose. If your model got slower after more ops became eligible, this is the first thing to change.
Data movement. Stick/unstick conversion is where offload wins get eaten. Fusion of ops with their surrounding stick/unstick is on by default; the relevant switches (--nnpa-disable-fusion-op-stick-unstick, --disable-compiler-stick-unstick, --nnpa-disable-saturation, --disable-zhigh-to-onnx) are documented with their interactions in stack/accelerators.md and concepts/stick-unstick-simd-codegen.md. Treat these as experiments to measure, not defaults to change.
Attention blocks. Two attention rewrites are on by default, so there is nothing to enable. An onnx.Attention op is decomposed into MatMul/Softmax/Add (plus Mul for the scale) before offload, so each piece can go to NNPA. The decomposition does not support GQA/MQA (kv_num_heads ≠ q_num_heads), softcap, qk_matmul_output_mode or softmax_precision; an onnx.Attention using any of them fails the compile with an op error rather than falling back to CPU. An attention mask that broadcasts into at least 4 MatMul → [Mul] → Add → Softmax chains of the same shape is expanded once, so those Adds no longer broadcast; broadcasting Adds are a common NNPA rejection. A broadcast scalar scale shared by at least 4 such chains is expanded the same way for the Muls. To measure what the mask expansion is worth, compare against a build with --nnpa-disable-expand-attention-mask. Details are in stack/accelerators.md.
Step 8: Measure — the only step that settles it
Everything above is a compile-time decision. Offload is not the goal; wall-clock time is, and more ops on NNPA is not automatically faster. Get per-op timings and compare:
onnx-mlir produces a shared library, not an executable, so a compiled model needs a driver to load it and call the entry point. RunONNXModel.py both compiles and runs, which is the shortest path:
python3 utils/RunONNXModel.py --model mymodel.onnx \
--compile-args="-O3 --march=z16 --maccel=NNPA --profile-ir=ZHigh" > profile.txt
grep "PERF-REPORT" profile.txt | sort -t, -k5 -rn | head -20--profile-ir=ZHigh is all you need: it instruments both onnx.* and zhigh.* ops, so one log shows offloaded and fallback ops side by side and you can see what the fallback actually costs. Use --profile-ir-with-sig=ZHigh instead to get each op's tensor shapes alongside the timings.
See debug-compilation.md for reading the profile and finding the slowest ops, and flows/instrumentation-and-profiling.md for the full instrumentation surface. Verify correctness alongside performance — utils/RunONNXModel.py --verify onnxruntime --verify-every-value — since --nnpa-disable-shape-restriction and quantization both trade guarantees for speed.
Related Pages
- stack/accelerators.md — the NNPA path itself: dialects, placement reporter, hardware levels, quantization, tuning flags
- repos/onnx-mlir/supported-ops-nnpa.md — which ops can offload
- concepts/nnpa-broadcasting.md — which broadcast shapes each op supports
- stack/compiler.md — what each
--marchvalue gets you, for SIMD and for the accelerator - skills/debug-compilation.md — general compilation diagnosis and op-level profiling
Verification
- Grounded against:
repo:onnx-mlir@b505d64:src/Accelerators/NNPA/Conversion/ONNXToZHigh/ONNXLegalityCheck.cpp,repo:onnx-mlir@b505d64:src/Accelerators/NNPA/Conversion/ONNXToZHigh/ONNXToZHighCommon.{cpp,hpp},repo:onnx-mlir@b505d64:src/Accelerators/NNPA/Conversion/ONNXToZHigh/ONNXToZHigh.cpp,repo:onnx-mlir@b505d64:src/Accelerators/NNPA/Conversion/ONNXToZHigh/DevicePlacement.cpp,repo:onnx-mlir@b505d64:src/Accelerators/NNPA/Conversion/ONNXToZHigh/RewriteONNXForZHigh.cpp,repo:onnx-mlir@b505d64:src/Accelerators/NNPA/Compiler/NNPACompilerOptions.{cpp,hpp},repo:onnx-mlir@b505d64:src/Accelerators/NNPA/Compiler/NNPACompilerUtils.cpp,repo:onnx-mlir@b505d64:src/Accelerators/NNPA/Support/NNPALimit.hpp,repo:onnx-mlir@b505d64:src/Compiler/CompilerOptions.cpp,repo:onnx-mlir@b505d64:src/Dialect/ONNX/ONNXOps/OpHelper.cpp,repo:onnx-mlir@b505d64:docs/SupportedONNXOps-NNPA.md,repo:onnx-mlir@b505d64:docs/Quantization-NNPA.md,repo:onnx-mlir@b505d64:src/Dialect/ONNX/Transforms/ExpandAttentionMask.cpp; the Attention-blocks paragraph re-checked againstrepo:onnx-mlir@b505d64(src/Dialect/ONNX/Transforms/ExpandAttentionMask.cpp,src/Dialect/ONNX/Transforms/AttentionToONNXOps.cpp,src/Accelerators/NNPA/Conversion/ONNXToZHigh/RewriteONNXForZHigh.cpp) - Files read:
ONNXLegalityCheck.cpp—onnxToZHighUnsupportedReport(theprintfformat) and the reason strings;onnxToZHighInCompatibilityReport;isValidElementTypeAndRank(F16/F32, rank 1–4); the threennpaDisableShapeRestrictionsites, which arecheckLegalityPoolOpsCommon(pooling H/W) andcheckConv2DParamRestrictions(Conv, called at theONNXConvOplegality check), including the two rejection messages that name the flagNNPACompilerOptions.cpp/.hpp—OPTREPORT_CL_ENUM_NNPA;nnpaDisableShapeRestrictiondesc;nnpaQuantDynamicOpt(the four scheme values plus theautoQuantOptempty-string default that suppresses the others,ValueOptional,CommaSeparated);nnpaQuantOpTypesOpt("Currently, only MatMul is supported");nnpaUseDynamicQuantizeLinearOnCPU{,ForScaleOffset};nnpaPlacementHeuristic(four values,QualifyingOpsdefault);nnpaDisableSaturation;nnpaDisableFusionOpStickUnstick;NNPAEmissionTargetTypeNNPACompilerUtils.cpp—ExpandAttentionMaskPassscheduled at the start ofaddONNXToZHighPassesunlessnnpaDisableExpandAttentionMask;RewriteONNXForZHigh.cpp—AttentionForZHighPattern(delegating tolowerONNXAttentionOp) and the always-illegalONNXAttentionOplegality rule;AttentionToONNXOps.cpp—checkSupportedAttentionAttributes;ExpandAttentionMask.cpp—minUses = 4, optional scalar-scaleMulNNPACompilerUtils.cpp—configureONNXToZHighLoweringPass(optReport == OptReport::NNPAUnsupportedOps, ...); pass order inaddPassesNNPA:createDevicePlacementPassunderemissionTarget >= EmitONNXIRbutaddONNXToZHighPassesunder>= EmitMLIR, which is what makes the report silent at--EmitONNXIRDevicePlacement.cpp/ONNXToZHigh.cpp—reportOnNNPAUnsupportedOpszeroed in the placement pass and restored inONNXToZHighLoweringPass::runOnOperationNNPALimit.hpp—NNPALevelM14/M15,MAXIMUM_NUM_HIDDEN_SIZE_LSTM8192 /_GRU10880,NNPAGetMaxForDim;RewriteONNXForZHigh.cpp— oversized MatMul dims are split, not rejectedCompilerOptions.cpp—--opt-reportvalues,--shapeInformationsyntax,--onnx-op-statsand its emission-target requirement;OpHelper.cpp—getNodeNameInPresenceOfOptresolving the report's third field fromonnx_node_namedocs/SupportedONNXOps-NNPA.md— theConvrow's six limitations, the level column convention, the DLFLOAT16/dimension-limit preamble;docs/Quantization-NNPA.md— z17 requirement, already-quantized vs float32 paths, per-tensor/i8/no-blocked/asymmetric-weight limitations, and the symmetric-faster/asymmetric-more-accurate guidance
- How to test: compile a model with
--maccel=NNPA --opt-report=NNPAUnsupportedOpsand confirm==NNPA-UNSUPPORTEDOPS-REPORT==lines appear at the default emission target and are absent at--EmitONNXIR; aConvwith a dynamic H or W should be rejected with a message naming--nnpa-disable-shape-restriction, and should offload either once that flag is passed or once--shapeInformationpins the input;--nnpa-quant-dynamicon a float32 model requires--march=z17to have any effect
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

