agentleFS
Aidas-devAidas-dev/k8s-agent-skills/skills/nvidia-device-plugin/SKILL.md

nvidia-device-plugin

Skill4 starsChanged 5 months agoMITView on GitHub

Use when working with NVIDIA GPU support on Kubernetes — device plugin, GPU Feature Discovery, Node Feature Discovery, CDI, MIG, time-slicing. Covers Helm chart, values, configuration. No CRDs (ConfigMap-based).

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Add annotation to any pod that needs GPU:

    line 36
  2. Very specific:

    deviceListStrategy: envvar is deprecated for CDI: When using CDI, set deviceListStrategy: cdi-annotations to use CDI annotations instead …

    line 201

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

NVIDIA Device Plugin

Overview

NVIDIA device plugin exposes GPUs to Kubernetes workloads. It runs as a DaemonSet on GPU nodes, discovers NVIDIA GPUs via NVML, and advertises them as allocatable resources (nvidia.com/gpu). Includes GPU Feature Discovery (GFD) for node labelling and optional Node Feature Discovery (NFD) for hardware detection.

No CRDs. ConfigMap-based configuration.

Latest: chart 0.19.2, app v0.19.2.

Architecture

node-feature-discovery (NFD) worker
  → detects PCI devices (class 02/03 = GPU)
    → labels node: nvidia.com/gpu.present=true

gpu-feature-discovery (GFD) daemon
  → queries NVML for GPU details
    → labels node: nvidia.com/gpu.product, nvidia.com/gpu.memory, etc.
    → labels node: nvidia.com/gpu.present=true

k8s-device-plugin daemon
  → watches for pods requesting nvidia.com/gpu
    → mounts GPU devices + drivers into containers
    → generates CDI specs (optional)

Quick Start

Add annotation to any pod that needs GPU:

spec:
  runtimeClassName: nvidia
  containers:
    - resources:
        limits:
          nvidia.com/gpu: 1

Runtime class nvidia must be configured on the node (Talos: .spec.runtime.runtimes.nvidia in machine config).

Key Features

FeatureEnabledConfig
GPU device discoveryAlwaysNVML
GPU Feature Discovery (GFD)✅ (gfd.enabled: true)Labels nodes with GPU properties
Node Feature Discovery (NFD)✅ (nfd.enabled: true)Detects PCI GPUs, labels nodes
CDI support✅cdi.nvidiaHookPath, cdi.featureFlags
Config-manager✅Dynamic config via node labels
MIG (Multi-Instance GPU)OptionalmigStrategy flag
Time-slicingOptionalConfig file config.name
Health checksAlwaysXid detection, device health

Helm Values

Core Settings

ValueDefaultDescription
image.repositorynvcr.io/nvidia/k8s-device-pluginImage repo
image.taglatestImage tag
runtimeClassNamenvidiaRuntime class for GPU pods
priorityClassName—Pod priority
failOnInitErrortrueFail if no GPU found at startup
deviceListStrategyenvvarHow device IDs are passed to container
deviceIDStrategyuuidHow devices are identified
nvidiaDriverRoot/Host driver root path

GFD (GPU Feature Discovery)

gfd:
  enabled: true
  nameOverride: gpu-feature-discovery
  noTimestamp: true
  sleepInterval: 60s
  securityContext:
    privileged: true  # Required for NVML access

GFD labels the node with GPU properties (product name, memory, CUDA version, driver version). These labels are used by NFD to set nvidia.com/gpu.present.

NFD (Node Feature Discovery)

nfd:
  enabled: true
  enableNodeFeatureApi: false  # Use legacy label-based NFD
  master:
    config:
      extraLabelNs: ["nvidia.com"]
  worker:
    config:
      sources:
        pci:
          deviceClassWhitelist: ["02", "03"]  # Display controllers + 3D controllers
          deviceLabelFields: ["vendor"]

NFD worker detects PCI devices with class 02 (display) or 03 (3D) from NVIDIA vendor and labels the node. GFD then adds detailed GPU labels.

Node Selection & Tolerations

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: nvidia.com/gpu.present
              operator: In
              values: ["true"]

tolerations:
  - key: CriticalAddonsOnly
    operator: Exists
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

CDI (Container Device Interface)

cdi:
  nvidiaHookPath: null     # Path to nvidia-cdi-hook/nvidia-ctk on host
  featureFlags: null       # Enable/disable specific CDI features

CDI generates device specifications so containers can use GPUs without privileged mode. When enabled, the device plugin adds cdi.k8s.io/* annotations to pods.

MIG (Multi-Instance GPU)

Configured via CLI flags (not in values.yaml — use config.map or CLI args).

MIG strategies:

  • none — no MIG (default)
  • single — expose all MIG devices as single resources
  • mixed — expose MIG devices individually

Time-Slicing

GPU sharing by time-slicing. Configured via ConfigMap:

config:
  name: ""     # ConfigMap name
  map: {}      # Inline config
  default: ""  # Default config name
  fallbackStrategies: ["named", "single"]

Update Strategy

updateStrategy:
  type: RollingUpdate

Deployment (Flux HelmRelease)

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: nvidia
  namespace: gpu-operator
spec:
  interval: 24h
  url: https://nvidia.github.io/k8s-device-plugin
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: nvidia-device-plugin
  namespace: gpu-operator
spec:
  chart:
    spec:
      chart: nvidia-device-plugin
      sourceRef:
        kind: HelmRepository
        name: nvidia
      version: 0.19.1

Common Mistakes

  • NFD labels the node, not the device plugin. NFD sets nvidia.com/gpu.present=true based on PCI device detection. If NFD is not deployed or fails, the device plugin won't schedule on the node. Check NFD pod logs.
  • GFD requires privileged access. Without securityContext.privileged: true, GFD can't query NVML and won't label the node. The device plugin then won't detect GPUs.
  • nvidia.com/gpu.present label not set. This label is set by NFD (not by GFD directly). GFD sets detailed labels (nvidia.com/gpu.product, etc.), but the presence label comes from NFD's PCI detection.
  • runtimeClassName: nvidia not set in workload. Without the runtime class, the container won't have GPU devices mounted even if it requests nvidia.com/gpu.
  • deviceListStrategy: envvar is deprecated for CDI. When using CDI, set deviceListStrategy: cdi-annotations to use CDI annotations instead of env vars.
  • MIG and time-slicing don't stack. Time-slicing can share MIG devices but the interaction is complex. Test thoroughly.
  • NodeFeature API vs legacy mode. With enableNodeFeatureApi: false (deployed), NFD uses labels directly. With the API enabled, NFD creates NodeFeature CRs. The device plugin doesn't care either way, but GFD must match.
  • Chart version doesn't match app version. Chart 0.19.1 deploys device plugin v0.18.2. Always check image.tag separately.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.