ubuntu-nvidia-gpu-enablement
Skill3 starsChanged yesterdayMITView on GitHub
Enable NVIDIA GPUs on a Ubuntu server for compute/inference serving — install the open-kernel-module driver (required for Blackwell/Hopper), CUDA toolkit, turn on IOMMU (intel_iommu=on iommu=pt), set up nvidia-persistenced, and install/wire a container runtime (Docker + nvidia-container-toolkit, or the minimal CLI), then verify all GPUs, IOMMU groups, P2P, nvcc, and GPU containers. Use when asked to enable or set up NVIDIA GPUs, install the NVIDIA driver + CUDA on Ubuntu, install Docker + nvidia-container-toolkit for GPU containers, configure GPU IOMMU/passthrough, prepare a host for GPU serving (vLLM/PyTorch/TensorRT/NIM), or troubleshoot nouveau, persistence mode, GPU-in-container, or driver/CUDA/glibc problems.
- Installs packages
In this repository31 other files it gives its agents.
- affix-domainsSkill
- apple-designSkill
- apwcli-pairSkill
- ask-always-on-agentSkill
- asus-esc8000-firmware-upgradeSkill
- asus-esc8000-gpu-bios-tuningSkill
- build-stealth-rustdeskSkill
- chrome-cdp-setupSkill
- cua-macos-lume-sandboxSkill
- deploy-kimi-k26-on-rtx-pro-6000Skill
- deploy-kimi-k3-on-rtx-pro-6000Skill
- deploy-stealth-rustdeskSkill
- ghostty-terminfoSkill
- herdr-rename-hookSkill
- imsg-permissionsSkill
- llm-inference-benchmarkSkill
- lxd-docker-firewall-conflictSkill
- mac-mini-as-headless-serverSkill
- macos-app-multi-instancesSkill
- macos-smb-serverSkill
- mac-smb-automountSkill
- openclaw-diagnoseSkill
- paseo-profilesSkill
- private-fork-syncSkill
- refresh-claude-accountSkill
- ssh-claude-authSkill
- sync-claude-accountsSkill
- to-featuresSkill
- trusted-publishSkill
- ubuntu-lxd-gpu-serverSkill
- ubuntu-zfs-mirror-installSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 12
Run as a sudo user (over SSH is fine).
- Very specific: line 37
Add intel_iommu=on iommu=pt (AMD: amd_iommu=on iommu=pt).
- Distinctive: line 87
5b — NVIDIA repo + toolkit: Add the repo (REFERENCE §5), then EITHER:
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Ubuntu NVIDIA GPU Enablement
Bring a fresh UEFI Ubuntu server with NVIDIA GPUs to a serving-ready state. Order matters: driver → IOMMU cmdline → CUDA → persistence → container access → verify. Driver and cmdline changes each need a reboot — batch them (Steps 1 + 2 reboot together).
Run as a sudo user (over SSH is fine). REFERENCE.md holds the why, GRUB/AMD variants, and troubleshooting. Boot-affecting steps are deliberate — keep BMC/console access as a fallback.
Pre-flight
- UEFI:
[ -d /sys/firmware/efi ]. - Secure Boot:
mokutil --sb-state. If enabled, DKMS modules need MOK enrollment at the console — plan for it or disable SB. (Off is typical with a custom bootloader; verify, don't assume.) - GPUs present:
sudo apt-get install -y pciutils && lspci -nn | grep -i nvidia. Note the architecture. - Bootloader = GRUB or ZFSBootMenu? (decides where the cmdline lives — Step 2.)
- Egress to
archive.ubuntu.com(+nvidia.github.iofor the container repo).
Step 1 — Driver (OPEN kernel module)
sudo apt-get update
sudo apt-get install -y nvidia-driver-580-server-open # example version- Blackwell requires the open modules (proprietary won't support it); Hopper too. Pick the right branch
with
ubuntu-drivers devices. Datacenter / "Server Edition" cards →nvidia-driver-<ver>-server-open. - Auto-blacklists nouveau; needs a reboot (do it with Step 2).
- Fabric Manager is only for NVLink/NVSwitch systems — skip it if your GPUs have no NVLink.
Step 2 — IOMMU kernel cmdline
Add intel_iommu=on iommu=pt (AMD: amd_iommu=on iommu=pt). iommu=pt = passthrough/identity DMA:
~zero bare-metal cost and preserves GPU P2P (full translation can disable P2P → NCCL falls back to SHM).
- GRUB: append to
GRUB_CMDLINE_LINUX_DEFAULTin/etc/default/grub, thensudo update-grub. - ZFSBootMenu (no GRUB): the cmdline is the ZFS property
org.zfsbootmenu:commandlineon the boot ROOT dataset. Read-then-APPEND — never clobber existing tokens (e.g.zfs_force=1):cur=$(zfs get -H -o value org.zfsbootmenu:commandline rpool/ROOT) sudo zfs set org.zfsbootmenu:commandline="$cur intel_iommu=on iommu=pt" rpool/ROOT - Reboot now (applies Steps 1 + 2 together).
Step 3 — CUDA toolkit
sudo apt-get install -y cuda-toolkit # often already pulled by the drivernvccis not on PATH by default — add it system-wide:printf 'export CUDA_HOME=/usr/local/cuda\nexport PATH=$CUDA_HOME/bin:$PATH\n' | sudo tee /etc/profile.d/cuda.sh- ⚠️ On bleeding-edge Ubuntu (glibc ≥ 2.41), some CUDA versions can't compile C++ (rsqrt/rsqrtf header clash). Serving is unaffected — prebuilt wheels use the driver + libcudart, not nvcc. See REFERENCE §3.
Step 4 — Persistence
sudo systemctl enable --now nvidia-persistenced- ⚠️ Ubuntu ships the unit
static→enablewon't stick at boot. Force it:sudo install -d /etc/systemd/system/nvidia-persistenced.service.d printf '[Install]\nWantedBy=multi-user.target\n' | sudo tee /etc/systemd/system/nvidia-persistenced.service.d/install.conf sudo systemctl daemon-reload && sudo systemctl enable --now nvidia-persistenced - ⚠️
nvidia-smikeeps showingPersistence-M: Disabled— that is correct. The daemon (not the deprecated flag) provides persistence; do not "fix" it withnvidia-smi -pm 1.
Step 5 — Container access (optional)
5a — Container engine. The toolkit only wires an existing engine, so install one first. Docker's
official docker-ce repo lags new Ubuntu releases by months — if it has no suite for your codename
(curl -fsI download.docker.com/linux/ubuntu/dists/$(. /etc/os-release; echo $VERSION_CODENAME)/Release
→ 404/000, e.g. 26.04 resolute), install Ubuntu's own docker.io (current — 26.04 ships Docker 29.x).
Both provide docker + the nvidia runtime hook.
sudo apt-get install -y docker.io # or docker-ce once the repo carries your codename
sudo usermod -aG docker "$USER" # rootless docker CLI — re-login to take effect5b — NVIDIA repo + toolkit. Add the repo (REFERENCE §5), then EITHER:
- Full (required for Docker/containerd):
sudo apt-get install -y nvidia-container-toolkit→nvidia-ctk- an auto-generated CDI spec. Wire the engine, then restart it:
Adds only asudo nvidia-ctk runtime configure --runtime=docker # or containerd sudo systemctl restart dockerruntimes.nvidiablock to/etc/docker/daemon.json; default runtime staysrunc(opt in per container with--gpus). Podman needs no wiring (uses CDI directly). - Minimal (enumeration only):
sudo apt-get install -y libnvidia-container-tools→nvidia-container-cli.
5c — Verify the GPU reaches a container:
sudo docker run --rm --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all ubuntu:24.04 nvidia-smi -L # lists every GPUNVIDIA_DRIVER_CAPABILITIES=all is what injects nvidia-smi/CUDA libs — --gpus all selects the GPUs but
caps decide the userspace, so bare --gpus all can return "executable not found". CDI path (Podman, or
Docker ≥ 25): nvidia-ctk cdi list; run with --device nvidia.com/gpu=all.
Step 6 — BIOS (max serving perf / passthrough)
Generic GPU-serving profile: Above-4G + Resize BAR on, Performance power/EPP, disable deep
C-states / C1E-promotion / ASPM, SR-IOV on (for VM passthrough), VT-d/IOMMU available, ACS off for fast
bare-metal P2P (flip on only when isolating GPUs for VFIO), Sub-NUMA Clustering off. Full table in REFERENCE §6.
For ASUS ESC8000-E12P exact attribute tokens + apply/verify, use the asus-esc8000-gpu-bios-tuning skill.
Verify (final)
nvidia-smi # all GPUs, VRAM, ECC, driver ver
cat /proc/driver/nvidia/version # "Open Kernel Module"
cat /proc/cmdline # iommu flags present (+ preserved tokens)
sudo dmesg | grep -iE 'DMAR|IOMMU' # "IOMMU enabled"
ls /sys/kernel/iommu_groups | wc -l # > 0
nvidia-smi topo -p2p r # P2P "OK" across GPUs (multi-GPU)
systemctl is-enabled nvidia-persistenced # enabled
bash -lc 'nvcc --version' # runs (login shell)Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

