Build and Dependency

Quick Start

Clone and install:

git clone https://github.com/NVIDIA-NeMo/Automodel.git && cd Automodel
uv sync --locked --all-groups --extra all

Or use the NeMo-AutoModel 26.10 container from NVIDIA NGC:

docker pull nvcr.io/nvidia/nemo-automodel:26.10.00
docker run --gpus all -it nvcr.io/nvidia/nemo-automodel:26.10.00

Installation Options

Option 1: NeMo-AutoModel Container (NGC)

The container ships with all dependencies pre-installed at /opt/Automodel (WORKDIR) with the venv at /opt/venv. Run as-is:

docker run --gpus all --network=host -it --rm --shm-size=32g \
    nvcr.io/nvidia/nemo-automodel:26.10.00 /bin/bash

Mounting your local checkout into the container

To develop against your host checkout, bind-mount it over /opt/Automodel to override the installed source:

docker run --gpus all --network=host -it --rm --shm-size=32g \
    -v <local-Automodel-path>:/opt/Automodel \
    nvcr.io/nvidia/nemo-automodel:26.10.00 /bin/bash

Inside the container, patch pyproject.toml / uv.lock for the PyTorch base image, then re-sync:

cd /opt/Automodel
bash docker/common/update_pyproject_pytorch.sh /opt/Automodel
uv sync --locked --all-groups --extra all

Warning: the update_pyproject_pytorch.sh step is required. Without it, uv sync will try to reinstall torch, which leads to CUDA version mismatches and TE import failures — uv cannot recognize the torch baked into the PyTorch base container.

Option 2: uv (Recommended for Local Development)

--all-groups pulls the build, docs, and test dev groups (defined in pyproject.toml); drop it for a runtime-only install.

uv sync --locked --all-groups                          # base + dev groups
uv sync --locked --all-groups --extra cuda             # CUDA support
uv sync --locked --all-groups --extra fa               # flash-attention
uv sync --locked --all-groups --extra moe              # mixture-of-experts
uv sync --locked --all-groups --extra vlm              # vision-language models (core)
uv sync --locked --all-groups --extra vlm-media        # + video/Qwen/Mistral decode (opencv, decord, qwen-utils; FFmpeg-bearing)
uv sync --locked --all-groups --extra diffusion        # diffusion models
uv sync --locked --all-groups --extra diffusion-media  # + diffusion preprocessing/export (imageio-ffmpeg, opencv)
uv sync --locked --all-groups --extra media            # vlm-media + diffusion-media (union)
uv sync --locked --all-groups --extra delta-databricks # Delta Lake / Databricks
uv sync --locked --all-groups --extra msa              # MiniMax M3 MSA sparse prefill (official MiniMax-AI/MSA with the model-private compatibility patch; Linux + SM100; run via `uv run` so torch finds `ninja`)
uv sync --locked --all-groups --extra all              # all standard extras (EXCLUDES media — FFmpeg kept opt-in)

The media extras (vlm-media, diffusion-media, media) bundle FFmpeg and are deliberately excluded from all and from the container image — add them explicitly for video/image decode.

Option 3: uv pip

Full install (matches uv sync --extra all):

uv venv
source .venv/bin/activate
uv pip install -e ".[all]"

To add NeMo Run submission support to the base package:

uv venv
source .venv/bin/activate
uv pip install "nemo-automodel[cli]"

The cli extra is additive: it adds nemo-run but does not remove the base package's core training dependencies, including PyTorch.

Package Management

Always use uv. Do not introduce pip install commands in scripts or docs.

When upgrading dependencies, review compatibility backports and their version gates. Verify the candidate release contains the upstream fix before removing or bypassing a backport; never assume a future or patch release is fixed. If a backport remains necessary, validate it on the candidate release before extending its gate. Run the affected regression tests with the actual dependency and required backends; mocked version strings and skipped tests are not compatibility evidence. Record the release evidence and test results in the upgrade PR.

Environment Variables

export HF_TOKEN="hf_..."           # Hugging Face token for gated models
export WANDB_API_KEY="..."         # Weights & Biases logging
export HF_HOME="/path/to/hf_cache" # Hugging Face cache directory

CLI Usage

The entry point is automodel (defined at nemo_automodel.cli.app:main).

Pattern: uv run automodel <config.yaml> [--nproc-per-node N] [--key.subkey value ...]

# The YAML's recipe field selects LLM, VLM, diffusion, or retrieval behavior.
uv run automodel examples/llm_finetune/llama3_2/llama3_2_1b_squad.yaml --nproc-per-node 8

Override any config value from the CLI:

uv run automodel examples/llm_finetune/llama3_2/llama3_2_1b_squad.yaml \
    --model.pretrained_model_name_or_path meta-llama/Llama-3.2-1B

Common Pitfalls