using-torch-onnxmlir-with-huggingface
Skill1.1k starsChanged 11 days agoApache-2.0View on GitHub
Use when compiling and running PyTorch or HuggingFace Transformers models through torch.compile() with the torch_onnxmlir backend: installation, encoder/decoder usage, configuration options, compiled-model caching, and debugging.
- Installs packages
Trending
Quiet since December.about the whole repository
Who's talking about onnx/onnx-mlir
- elixir-nx/nx“Import of MLIR modules”GitHub · Mar 5, 2024 · 4 reactions
- PyTorchKR/pytorchcore-kr“더 나은 질의 응답 구조를 위한 Discussion 활용 방안”GitHub · Jan 20, 2025 · 2 reactions
- llvm/torch-mlir“[RFC] Support for an embedded build of Torch-MLIR”GitHub · Sep 23, 2022 · 2 reactions
- Samsung/ONE“[circle-mlir] Update onnx-mlir to official release”GitHub · Jun 8, 2025 · 1 reactions
- Xilinx/onnx-mlir“ONNX to TOSA Legalization Upstream tracker”GitHub · Apr 20, 2023 · 1 reactions
The biggest 5 of 22 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 22
Talked about on GitHub (22 times).
On GitHub · 22
- Mar 5, 2024Import of MLIR modules — elixir-nx/nx4 reactions · 2 comments
- Jan 20, 2025더 나은 질의 응답 구조를 위한 Discussion 활용 방안 — PyTorchKR/pytorchcore-kr2 reactions
- Sep 23, 2022[RFC] Support for an embedded build of Torch-MLIR — llvm/torch-mlir2 reactions · 4 comments
- Jun 8, 2025[circle-mlir] Update onnx-mlir to official release — Samsung/ONE1 reaction · 4 comments
- Apr 20, 2023ONNX to TOSA Legalization Upstream tracker — Xilinx/onnx-mlir1 reaction
17 more on GitHub
- Dec 16, 2025[ttl] Automate elementwise op generation from specifications — tenstorrent/tt-lang0 reactions · 1 comment
- Jul 29, 2025Missing Symbols due to visibility setting change — onnx/onnx0 reactions · 2 comments
- Nov 19, 2024Add fancier CI badges showing number of tests and/or test coverage — iree-org/iree-test-suites0 reactions · 2 comments
- Aug 6, 2024Standardize rewriter tests; testing facilities — microsoft/onnxscriptsays it tested it0 reactions · 1 comment
- May 8, 2024Fix macos build by pinning version — Xilinx/onnx-mlir0 reactions
- Mar 30, 2024Include the ".onnxtext" extension in supported serialization format — onnx/onnx0 reactions · 1 comment
- Dec 20, 2023add filecheck test for add.onnx — xdslproject/xdsl0 reactions · 7 comments
- Dec 1, 2023Failed to build with the use of --cpp-implementation — protocolbuffers/protobuf0 reactions
- Oct 27, 2023Bring in the fix for empty slice in canonicalization — Xilinx/onnx-mlir0 reactions
- Oct 25, 2023Cherry-picked upstream PR: Add conversion for DivOp ONNX to TOSA (#2207) — Xilinx/onnx-mlir0 reactions
- Aug 26, 2023Build error on M1 MacOS machine — onnx/onnx0 reactions · 3 comments
- Oct 18, 2022ONNX MLIR exporter — nod-ai/onnx-mlir-exporter0 reactions · 1 comment
- Oct 9, 2022[Feature request] Machine Readable definition of Operations (Schema file instead of Markdown or cc file) — onnx/onnx0 reactions · 14 comments
- Jun 28, 2022Can we have newer opset of bidaf-9 model (including opset 9 or newer onnx.Scan op)? — onnx/models0 reactions · 1 comment
- Jun 5, 2022Broken Links — onnx/onnx.github.io0 reactions · 3 comments
- May 12, 2022Update the onnx-mlir version — DependableSystemsLab/LLTFI0 reactions · 2 comments
- Jun 30, 2020Error when trying to lower alex net onnx model to MLIR — onnx/models0 reactions · 1 comment
In this repository14 other files it gives its agents.
- AGENTS.mdAGENTS.md
- add-fusion-patternSkill
- add-new-onnx-operationSkill
- add-onnx-op-loweringSkill
- add-onnx-op-to-dialectSkill
- add-onnx-transform-passSkill
- coding-conventionsSkill
- debug-compilationSkill
- debug-type-catalogueSkill
- ground-lit-testsSkill
- nnpa-internalsSkill
- nnpa-performanceSkill
- onnx-mlir-internalsSkill
- upgrade-llvm-mlirSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 1 is distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 283
If the hash matches an existing cache entry, the compiled model is reused.
- Very specific: line 395
Use RunONNXModel.py to test the compiled model standalone:
- Very specific: line 401
Add -v to compile_options for verbose onnx-mlir output:
- Very specific: line 407
Add --onnx-op-stats TXT to see which ops are accelerated:
- Very specific: line 512
Debugging: Use keep_onnx_files, generate_test_data_set, and verbose compiler flags
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
<!-- Generated from the onnx-mlir knowledge base (skills/using-torch-onnxmlir-with-huggingface.md), verified against onnx-mlir@b505d64b. Do not edit here: see .agents/skills/README.md. -->Using torch_onnxmlir with HuggingFace Models
Tutorial for using the torch_onnxmlir package to compile and run HuggingFace Transformers models via torch.compile() with onnx-mlir as the backend.
Applies to: src/Runtime/python/torch_onnxmlir/, docs/RunTorchModel.md
Related pages: Runtime, Compiler
Overview
torch_onnxmlir provides a custom backend for PyTorch's torch.compile() that routes compilation through onnx-mlir instead of the default TorchInductor backend. This enables:
- Transparent acceleration: Compile PyTorch models (especially HuggingFace Transformers) with onnx-mlir optimizations without changing model code
- NNPA acceleration: Target IBM z16 NNPA hardware acceleration via
-march=z16 -maccel=NNPAcompiler flags - Persistent caching: Compiled models are cached on disk to avoid recompilation across runs
- HuggingFace integration: Works seamlessly with
transformers.AutoModel,AutoModelForCausalLM, and related classes
The backend sits between PyTorch's FX graph representation and onnx-mlir's compilation pipeline:
torch.nn.Module
→ torch.compile(backend="onnxmlir")
→ torch FX graph
→ torch_onnxmlir backend (backend.py)
→ ONNX export
→ om_pyrt.CompileSession (onnx-mlir compilation)
→ om_pyrt.InferenceSession (execution)
→ cached .so libraryInstallation
Prerequisites
- onnx-mlir built and installed: You need a working onnx-mlir build with Python bindings
- om_pyrt installed: The
torch_onnxmlirpackage depends onom_pyrt(see om_pyrt README)
Install torch_onnxmlir
From local onnx-mlir repository:
cd onnx-mlir
pip install -e src/Runtime/python/torch_onnxmlirOr from PyPI (when available):
pip install torch_onnxmlirVerify Installation
import torch_onnxmlir
print(torch_onnxmlir.__file__) # Should show installation pathBasic Usage Pattern
Minimal Example
import torch
import torch.nn as nn
import torch_onnxmlir # Registers "onnxmlir" backend
class AddModel(nn.Module):
def forward(self, x, y):
return x + y
model = AddModel()
# Compile with onnx-mlir backend
om_options = {
"compile_options": "-O3",
"compiler_path": "/workdir/onnx-mlir/build/Debug/bin/onnx-mlir",
}
compiled_model = torch.compile(model, backend="onnxmlir", options=om_options)
# Use normally
input1 = torch.randn(2)
input2 = torch.randn(2)
output = compiled_model(input1, input2)Backend Options
The options dict passed to torch.compile() supports:
compiler_path(str): Path toonnx-mlirbinary (e.g.,/workdir/onnx-mlir/build/Debug/bin/onnx-mlir)compiler_image_name(str or None): Docker image name for containerized compilation (e.g.,"onnxmlirczar/onnx-mlir:latest"). Set toNonefor local compilation.compile_options(str): Space-separated compiler flags passed to onnx-mlir (e.g.,"-O3 -march=z16 -maccel=NNPA")
Example with NNPA acceleration:
om_options = {
"compiler_image_name": None,
"compiler_path": "/workdir/onnx-mlir/build/Debug/bin/onnx-mlir",
"compile_options": "-O3 -march=z16 -maccel=NNPA",
}
compiled_model = torch.compile(model, backend="onnxmlir", options=om_options)HuggingFace Encoder Models
Encoder models (BERT-style, embedding models) process input sequences and return contextualized representations. Common use cases: sentence embeddings, classification, named entity recognition.
Example: Granite Embedding Model
import torch
import torch_onnxmlir
from transformers import AutoModel, AutoTokenizer
# Load model and tokenizer
model_path = "ibm-granite/granite-embedding-30m-english"
model = AutoModel.from_pretrained(model_path, dtype="float32")
tokenizer = AutoTokenizer.from_pretrained(model_path)
# Set to inference mode
model.eval()
# Compile with onnx-mlir backend
om_options = {
"compiler_image_name": None,
"compiler_path": "/workdir/onnx-mlir/build/Debug/bin/onnx-mlir",
"compile_options": "-O3 -march=z16 -maccel=NNPA",
}
compiled_model = torch.compile(model, backend="onnxmlir", options=om_options)
# Prepare inputs
text = "AI is fascinating."
inputs = tokenizer(text, return_tensors="pt")
# Run inference
with torch.no_grad():
outputs = compiled_model(**inputs)
# Extract CLS token embedding
cls_embedding = outputs.last_hidden_state[:, 0, :]
print(cls_embedding.shape) # [1, hidden_size]Supported Encoder Models
Tested models (from test_encoder_models.py):
ibm-granite/granite-embedding-30m-englishibm-granite/granite-embedding-278m-multilingualgoogle-bert/bert-base-uncased
Key Points for Encoders
- Set
dtype="float32": Ensures compatibility with onnx-mlir's type system - Use
model.eval(): Disables dropout and other training-specific behavior - Wrap in
torch.no_grad(): Disables gradient computation for inference - First run compiles: The first inference call triggers compilation and caching; subsequent calls use the cached model
- Variable sequence lengths: Different input lengths may trigger separate compilations (one per unique shape)
HuggingFace Decoder Models
Decoder models (GPT-style, causal language models) generate text autoregressively. They require special handling for the prefill and decode phases.
Example: GPT-2 Text Generation
import torch
import torch_onnxmlir
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model and tokenizer
model_path = "openai-community/gpt2"
model = AutoModelForCausalLM.from_pretrained(
model_path,
attn_implementation="eager", # Required: decomposes SDPA ops
dtype="float32",
)
tokenizer = AutoTokenizer.from_pretrained(model_path)
# Set to inference mode
model.eval()
# Compile the forward method (not the whole model)
om_options = {
"compiler_image_name": None,
"compiler_path": "/workdir/onnx-mlir/build/Debug/bin/onnx-mlir",
"compile_options": "-O3 -march=z16 -maccel=NNPA",
}
model.forward = torch.compile(model.forward, backend="onnxmlir", options=om_options)
# Prepare inputs
prompt = "Hello, I'm a language model,"
input_tokens = tokenizer(prompt, return_tensors="pt")
# Generate text
with torch.no_grad():
output = model.generate(**input_tokens, max_new_tokens=11)
# Decode output
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)Supported Decoder Models
Tested models (from test_decoder_models.py):
openai-community/gpt2Qwen/Qwen2-0.5B-Instructibm-granite/granite-3.3-2b-instructibm-granite/granite-4.0-350M
Key Points for Decoders
- Use
attn_implementation="eager": Required to decompose scaled dot-product attention (SDPA) ops into primitives that onnx-mlir can handle - Compile
model.forward, notmodel: Thegenerate()method orchestrates multiple forward passes; compile the forward method itself - Two compilations occur: One for the prefill phase (processing the prompt) and one for the decode phase (generating tokens one at a time with KV cache)
- KV cache handling: The backend automatically manages key-value cache tensors across generation steps
- Use
torch.no_grad(): Essential for generation to avoid memory overhead
Prefill vs. Decode Phases
Decoder models have two distinct execution patterns:
- Prefill: Process the entire prompt at once (shape:
[batch, prompt_length, ...]) - Decode: Generate one token at a time using cached keys/values (shape:
[batch, 1, ...])
The backend compiles these separately because they have different input shapes and computational patterns. You'll see two compilation messages in the logs on the first run.
Configuration Options
The torch_onnxmlir.config module provides runtime configuration knobs:
import torch_onnxmlir
# Custom cache directory (default: ~/.cache/torch_onnxmlir, changed from
# the bare ~/.cache in a 2026-07-29 update — see Caching Behavior below)
torch_onnxmlir.config.cache_dir = "/path/to/custom/cache"
# Maximum number of cached sessions (default: 3)
torch_onnxmlir.config.session_cache_limit = 5
# Keep ONNX files after compilation (default: False)
torch_onnxmlir.config.keep_onnx_files = True
# Generate test data sets for debugging (default: False)
torch_onnxmlir.config.generate_test_data_set = True
# Regenerate test data sets on every run (default: False)
torch_onnxmlir.config.regenerate_test_data_set = TrueConfiguration Details
cache_dir: Where compiled models are stored. Can also be set viaTORCHONNXMLIR_CACHE_DIRenvironment variable.session_cache_limit: LRU cache size for in-memory sessions. Older sessions are evicted when the limit is reached.keep_onnx_files: Useful for debugging; keeps intermediate ONNX files alongside compiled.solibraries.generate_test_data_set: Createstest_data_set/folder withinput_*.pbandoutput_*.pbfiles usable withRunONNXModel.pyfor standalone testing.regenerate_test_data_set: Overwrites test data on every run (slow; use only for debugging).
Hashing and Cache Key Generation
The backend uses a sophisticated hashing mechanism to determine if a model needs recompilation:
- Graph structure: FX graph topology and operations, including each node's literal arguments (reshape dims, axis indices, scalar thresholds), so two graphs that differ only in a constant argument get different keys
- Parameter values: Each parameter is reduced to a SHA-256 of its shape, dtype and data. A parameter with at most 1024 elements is hashed in full; a larger one is hashed from 1024 evenly strided samples. Nested sub-module parameters (e.g.
linear.weight) are included.config.sample_parameter_values_limitis still defined inconfig.pybut the hash no longer reads it - Input shapes: Each placeholder's shape and dtype are part of the cache key, with dynamic (
SymInt) dims renamed to stabledim_0,dim_1, ... - Compiler options: Different flags produce different cache entries
If the hash matches an existing cache entry, the compiled model is reused. Otherwise, a new compilation is triggered.
Caching Behavior
Cache Location
Default: ~/.cache/torch_onnxmlir/
Override via:
torch_onnxmlir.config.cache_dir = "/custom/path"Or environment variable:
export TORCHONNXMLIR_CACHE_DIR=/custom/pathCache Structure
~/.cache/torch_onnxmlir/
├── <hash1>/
│ ├── model.so # Compiled library
│ ├── om_backend_config.json # SHA-256 of model.so (and .constants.bin), signatures, compile info
│ ├── model.onnx # (if keep_onnx_files=True)
│ └── test_data_set/ # (if generate_test_data_set=True)
│ ├── input_0.pb
│ ├── output_0.pb
│ └── ...
├── <hash2>/
│ └── ...
└── ...Each unique model (by hash) gets its own subdirectory. Directories are created with mode 0o700 (owner only).
Cache Integrity
When a compiled model is cached, a SHA-256 hash of each .so/.constants.bin is written under artifact_hashes in that entry's om_backend_config.json. Before loading a cached .so, the backend recomputes the hashes. Any of these counts as a cache miss, and the model is recompiled instead of loaded:
- a hash does not match, or a hashed file is missing
- the entry has no stored hashes (written by a
torch_onnxmlirversion before this check), or no config file - a hash key is not a bare file name
So the first run after upgrading recompiles every cached model once. Keep the cache directory private to the account running inference: anyone who can write to it can replace both the .so and its stored hash.
Cache Invalidation
The cache is invalidated when:
- Model parameters change (weights updated)
- Graph structure changes (model architecture modified)
- Compiler options change (different flags)
- Input shapes change (for shape-dependent compilations)
Session Cache (In-Memory)
In addition to disk caching, the backend maintains an in-memory LRU cache of InferenceSession objects (default limit: 3). This avoids reloading the .so library from disk on repeated calls with the same model.
Testing and Debugging
Running Tests
The torch_onnxmlir package includes comprehensive tests:
cd onnx-mlir/src/Runtime/python/torch_onnxmlir/tests
mkdir build && cd build
cmake ..
ctest -j 8Or run individual tests with pytest:
python -m pytest test_add.py
python -m pytest test_encoder_models.py
python -m pytest test_decoder_models.pyTest Coverage
test_add.py: Simple element-wise additiontest_gemm.py: Matrix multiplicationtest_relu.py: Activation functiontest_encoder_models.py: BERT/Granite embedding modelstest_decoder_models.py: GPT-2/Qwen2/Granite causal LM modelstest_sessioncache.py: Cache behavior verificationtest_cache_integrity.py: Disk-cache hash check and directory mode (no compiler needed)test_hash_key.py: Cache-key generation: literal arguments and parameter values change the key (no compiler needed;conftest.pystubs outom_pyrtwhen no onnx-mlir build is installed)
The tests use the compiler at /workdir/onnx-mlir/build/Debug/bin/onnx-mlir (the container path) unless the COMPILER_PATH environment variable names another onnx-mlir binary.
Debugging Tips
- Enable logging:
import logging logging.basicConfig(level=logging.INFO) - Keep ONNX files:
Inspect the exported ONNX graph in the cache directory.torch_onnxmlir.config.keep_onnx_files = True - Generate test data sets:
Usetorch_onnxmlir.config.generate_test_data_set = TrueRunONNXModel.pyto test the compiled model standalone:python utils/RunONNXModel.py --load-model ~/.cache/torch_onnxmlir/<hash>/model.so - Check compiler output:
Add
-vtocompile_optionsfor verbose onnx-mlir output:om_options = {"compile_options": "-O3 -v"} - Verify NNPA usage:
Add
--onnx-op-stats TXTto see which ops are accelerated:om_options = {"compile_options": "-O3 -march=z16 -maccel=NNPA --onnx-op-stats TXT"} - Cache-hit/compile-time metrics with
explain_context()(added 2026-07-31, replacing an earlier one-shotexplain()/config.enable_explainAPI from two days prior — some upstream README sections still show the older snippet):with torch_onnxmlir.explain_context() as metrics: output = compiled_model(input) print(metrics) # formatted table: unique graphs, cache hit rate, # eager fallbacks, per-graph compile/inference timeformat="dict"gives programmatic access (metrics.data["summary"]["total_cache_hits"], etc.);sort_by="calls"reorders the per-graph table;export_to="path/to/file.json"(nested dirs auto-created) saves it instead of/alongside printing. This is the fastest way to confirm whether a slow first call was compilation (expected) versus a cache miss that shouldn't have happened (e.g. from an input shape that varies more than expected).
Common Issues
Issue: attn_implementation="eager" required for decoder models
Reason: PyTorch's fused SDPA op is not directly supported by ONNX export. Using "eager" decomposes it into primitive ops (matmul, softmax, etc.).
Issue: Different sequence lengths trigger recompilation
Reason: Input shapes are part of the cache key. Use padding to a fixed length if you want to avoid multiple compilations.
Issue: First run is slow
Reason: Compilation happens on the first inference call. Subsequent runs use the cached model.
Comparison with utils/python/transformers Scripts
The torch_onnxmlir package and the scripts in utils/python/transformers/ serve different purposes:
torch_onnxmlir (This Tutorial)
- Purpose: Transparent PyTorch integration via
torch.compile() - Interface: Python, PyTorch-native
- Use case: Drop-in replacement for default
torch.compile()backend - Model loading: Uses
transformers.AutoModeldirectly - Caching: Automatic, hash-based
- Best for: Iterative development, experimentation, HuggingFace workflows
utils/python/transformers Scripts
- Purpose: Standalone ONNX model execution (e.g.,
run_gpt2_from_huggingface.py) - Interface: Python, but uses
PyOMCompileandOMExecutionSessiondirectly - Use case: Pre-export ONNX models from HuggingFace, compile once, run many times
- Model loading: Downloads ONNX models from HuggingFace Hub (not PyTorch models)
- Caching: Manual (user manages downloaded ONNX files)
- Best for: Production deployments, benchmarking, quantization experiments
When to Use Which
| Scenario | Recommended Approach |
| Prototyping with HuggingFace models | torch_onnxmlir |
| Integrating into existing PyTorch training/inference code | torch_onnxmlir |
| Benchmarking onnx-mlir performance | utils/python/transformers scripts |
| Production deployment (pre-compiled models) | utils/python/transformers scripts or direct om_pyrt usage |
| Experimenting with quantization | utils/python/transformers scripts (see run_gpt2_from_huggingface.py with -march=z16 -maccel=NNPA) |
Both approaches ultimately use the same onnx-mlir compiler and runtime; the difference is in the level of abstraction and integration with PyTorch.
Advanced Topics
Custom FX Graph Transformations
The backend applies several FX graph transformations before ONNX export (in fx_utils.py):
- Freeze scalar constants: Convert scalar Python values to tensor constants
- Convert
SymIntto tensors: Handle dynamic shape symbols - Remove unused placeholders: Clean up graph inputs that aren't actually used
These transformations ensure the exported ONNX graph is compatible with onnx-mlir's expectations.
Session Cache Implementation
The SessionCache class (in sessioncache.py) implements an LRU cache with disk persistence:
- Cache key:
_om_followed by three SHA-256 digests: the FX graph (node args and parameter hashes included), the sorted compiler options, and the example inputs' ranks - Cache value:
CacheValueobject containing the compiled.sopath and metadata - Eviction policy: LRU (least recently used) when
session_cache_limitis reached - Persistence: Each entry's metadata (
artifact_hashes,example_inputs_indices,compilation_info,input_signature,output_signature) is stored inom_backend_config.jsoninside that entry's directory;load_from_diskverifies the hashes before creating theInferenceSession
Compiler Options Passthrough
The compile_options string is passed directly to onnx-mlir's CLI parser. All flags documented in docs/Options.md are available:
- Optimization levels:
-O0,-O1,-O2,-O3 - Target architecture:
-march=x86-64,-march=z16 - Accelerators:
-maccel=NNPA - Profiling:
--profile-ir=Onnx(timing) or--profile-ir-with-sig=Onnx(timing + shapes); the raw--instrument-*flags these subsume are deprecated for everyday use - Emission targets:
--EmitONNXIR,--EmitMLIR,--EmitLLVMIR(for debugging) - Op statistics:
--onnx-op-stats TXT
See Compiler Options for the full list.
Summary
torch_onnxmlir provides a seamless way to leverage onnx-mlir's optimizations and NNPA acceleration for HuggingFace Transformers models via PyTorch's torch.compile() API. Key takeaways:
- Installation: Requires
om_pyrtand a working onnx-mlir build - Basic usage:
torch.compile(model, backend="onnxmlir", options={...}) - Encoder models: Work out-of-the-box with
AutoModel - Decoder models: Require
attn_implementation="eager"and compilingmodel.forward - Caching: Automatic, hash-based, with configurable cache directory and limits; cached libraries are hash-checked before loading
- Debugging: Use
keep_onnx_files,generate_test_data_set, and verbose compiler flags - Comparison:
torch_onnxmlirfor PyTorch integration,utils/python/transformersscripts for standalone ONNX execution
Verification
This skill was grounded against the following sources:
Repositories and SHAs:
onnx-mlir@f4b8e8de: Main source for torch_onnxmlir implementation and documentationonnx-mlir@c92e126(2026-08-06 sync): re-verifiedcache_dirdefault andexplain_context()API against this later commit; commits94c6bc1/de23de78/7cea64fintroduced or changed theseonnx-mlir@b505d64: re-verified Hashing and Cache Key Generation, the cache-key bullet under Session Cache Implementation, and Test Coverage againstbackend.py(generate_hash_key,_stable_args_str,_tensor_sha256,_PARAM_FULL_HASH_LIMIT = 1024),config.py,tests/test_hash_key.pyandtests/conftest.py; also re-checked theconfigdefaults,TORCHONNXMLIR_CACHE_DIR, the tested encoder/decoder model lists and theENCODER_MODEL/DECODER_MODEL/COMPILER_PATHtest variablesonnx-mlir@5534bc7: re-verified the disk-cache layout,0o700directory mode, and hash check (Cache Structure, Cache Integrity, Session Cache Implementation) againstsessioncache.pyand the README's "Security note on the disk cache";tests/test_cache_integrity.pycovers them
Files Read:
src/Runtime/python/torch_onnxmlir/README.md— Installation, basic usage, caching,explain_context()usage/example outputsrc/Runtime/python/torch_onnxmlir/src/torch_onnxmlir/backend.py— Backend implementation, FX graph handling, cache-key generationsrc/Runtime/python/torch_onnxmlir/src/torch_onnxmlir/config.py— Configuration optionssrc/Runtime/python/torch_onnxmlir/src/torch_onnxmlir/sessioncache.py— Cache implementationsrc/Runtime/python/torch_onnxmlir/src/torch_onnxmlir/explain.py—explain_context()/is_enabled()implementationsrc/Runtime/python/torch_onnxmlir/tests/test_encoder_models.py— Encoder model examplessrc/Runtime/python/torch_onnxmlir/tests/test_decoder_models.py— Decoder model examplessrc/Runtime/python/torch_onnxmlir/tests/test_hash_key.py,conftest.py— Cache-key tests and theom_pyrtstubutils/python/transformers/run_gpt2_from_huggingface.py— Comparison approach
How to Test:
- Basic functionality test (from README.md):
python -c " import torch import torch.nn as nn import torch_onnxmlir class AddModel(nn.Module): def forward(self, x, y): return x + y model = AddModel() om_options = { 'compile_options': '-O3', 'compiler_path': '/workdir/onnx-mlir/build/Debug/bin/onnx-mlir', } compiled = torch.compile(model, backend='onnxmlir', options=om_options) x, y = torch.randn(2), torch.randn(2) result = compiled(x, y) print('Success:', result.shape) " - Run test suite:
cd onnx-mlir/src/Runtime/python/torch_onnxmlir/tests python -m pytest test_add.py - Encoder model test (requires HuggingFace transformers):
ENCODER_MODEL=bert-base-uncased python -m pytest test_encoder_models.py - Decoder model test:
DECODER_MODEL=gpt2 python -m pytest test_decoder_models.py
Expected: All tests pass, models compile and run with accuracy >98% similarity to PyTorch reference.
For more details on the runtime architecture, see Runtime. For compiler flag reference, see Compiler.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

