l8k: Configuration Files

PREREQUISITE: Read ../k8s-launch-kit-shared/SKILL.md for install paths, global flags, and output modes.

Understand, create, or edit l8k configuration files.

Desired networkOperator.secondaryNetwork.multus and .cniPlugins each accept auto/managed/external (empty/omitted/null means auto). Discovery writes discoveredSecondaryNetwork evidence without replacing desired policy. Multus does not prove reference plugin binaries exist. See ownership rules.

File Types

Usage

# Use a config file for generation
l8k generate --user-config my-config.yaml \
  --save-deployment-files ./output

# Combine user config with live discovery (hardware refreshed, settings kept)
l8k discover --user-config my-config.yaml \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./updated-config.yaml

Profile Resolution

Fresh discovery and file-backed generation resolve settings with this precedence:

1. Canonical built-in defaults from the repository-root cluster-config.yaml.
2. Hardware-derived profile defaults.
3. Existing config values.
4. Explicit CLI flags.

Generation never rewrites the source config. It stores the exact effective configuration at <output>/.l8k/resolved-config.yaml; deploy and validate use that sidecar automatically when no explicit config is passed. Explicit false, zero, and empty values override defaults. YAML null means unset.

Discovery with --user-config follows a stricter refresh contract: only clusterConfig is replaced. Every other section remains as loaded unless an explicit CLI flag overrides its corresponding field. Generation still fills missing profile fields when it consumes that config.

Config Sections Quick Reference

Discovery saves the effective RA naming in clusterConfig[].spectrumX.interfaceNames (netdevPrefix and rdmaPrefix). These are derived values: resolution refreshes them after mode or naming changes, including for older configs without this block. Change the existing spectrumX.singlePlane, hwplb, or swplb prefix settings for custom naming. PF inventory names remain observed values. Naming templates and PF selectors consume the same resolved per-group prefixes; standard profiles are unaffected. VR HWPLB uses eth_r%rail_id%_p%plane_id% and roce_r%rail_id%_n%nic_id%; traditional HWPLB retains roce_r%rail_id%.

Each clusterConfig[] entry has these key fields:

• identifier — group name (used for NicNodePolicy naming). For groups with both machineType and gpuType resolved, this is the lowercased machine/GPU identity with complete NVIDIA segments removed and common machine segments shortened (ThinkSystem → ts, PowerEdge → pe), bounded to 30 bytes with balanced component prefixes and a 6-character deterministic hash when needed; otherwise a fallback group-N. The Launch Kit machine node label uses the same value.
• machineType — server model (e.g. PowerEdge-XE9680); populated from nvidia.com/gpu.machine label or DMI fallback.
• gpuType — GPU SKU (e.g. NVIDIA-H200); populated from nvidia.com/gpu.product label or nvidia-smi fallback. Note: this field used to be called productType — the rename happened to disambiguate it from the server model. Old productType: keys in hand-authored configs must be renamed to gpuType:.
• netplanManaged — true when any worker has an NVIDIA PF whose current MAC is selected by a host netplan match.macaddress stanza with a non-empty set-name. The flag identifies a potential conflict with NCO udev naming; host-specific MACs are not persisted. If generation would emit a NicInterfaceNameTemplate for this group, clean up the affected set-name stanzas and re-run discovery instead of editing this flag by hand.
• netplanManagementUnknown — true when discovery found no match but a worker's netplan observation was unavailable or incomplete, or a MAC-based set-name rule could not be checked because an expected PF has no readable current MAC. A readable /etc without a netplan directory or a complete configuration without MAC-based set-name rules is a verified negative. Generation warns and continues with interface-name rules unless netplanManaged is true. Review host netplan configuration and re-run discovery to resolve the warning.
• capabilities.nodes.{sriov,rdma,ib} — what the underlying hardware supports.
• pfs[] — physical function list with PCI address, device ID, RDMA device, network interface, traffic class, rail, NUMA, GPU affinity, and model (the VPD model/description string read from NicDevice.Status.modelName).
  • One rail per NIC (default). Discovery advertises one rail per physical NIC: a NIC's multi-plane east-west PFs (planes of one port, e.g. Spectrum-X ConnectX-8/9) collapse to the master PF, so an 8-PF node lists 4 east-west PFs / 4 rails. A NIC whose model is genuinely dual-port (2-port/Dual-port) keeps a rail per port. Run l8k discover --collapse-nic-rails=false to emit one rail per PF (legacy/dev behaviour).
• nodeSelector — Kubernetes node selector for this group. Source groups key on the machine label written by l8k discover: nvidia.kubernetes-launch-kit.machine: <identifier>. Auto-merged groups (different machineTypes sharing a GPU type) key on nvidia.kubernetes-launch-kit.gpu: <gpuType> instead — the GPU label retains its discovered value, including NVIDIA, so the merged selector binds correctly across source machineTypes.
• workerNodes — explicit hostnames (populated by discovery).

For the full field-by-field reference with types, defaults, and descriptions, read references/config-reference.md.

Topology Presets

The presets/ directory contains pre-recorded topologies for known hardware combinations. A preset is a topology.yaml file with the following shape:

machineType: PowerEdge-XE9680   # required
gpuType: NVIDIA-H200            # required — both keys are matched as a pair
nicModel: BlueField-3 SuperNIC (ConnectX-7)
gpuInterconnect: NV18
numaNodes: 2

# Required if the preset will be used with `l8k generate --for`.
# Discovery-time overlay does not need this block.
capabilities:
  nodes:
    sriov: true
    rdma: true
    ib: false

pfs:
  - deviceID: a2dc
    pciAddress: 0000:1a:00.0
    traffic: east-west
    rail: 0
    numaNode: 0
    connectedGPU: GPU0
    gpuProximity: PIX

Lookup is exact-match on (machineType, gpuType). No any-GPU fallback — a preset that doesn't declare gpuType: is rejected at load time. Multi-variant presets for the same machine (different GPU SKUs) live in separate directories with composite names like PowerEdge-XE9680-H200 / PowerEdge-XE9680-B200. The directory name is shown by l8k preset list and is what l8k generate --for <name> accepts.

Validation deviations. When the matched preset's PF count, PCI addresses, or device IDs don't exactly match discovered hardware, the preset is NOT applied — discovery keeps the live-discovered topology (traffic/rail/NUMA), because overlaying a preset onto a different PCI layout would corrupt the classification (a coincidentally-overlapping PCI address would inherit the preset's unrelated traffic/rail/GPU fields). The discrepancies are recorded under clusterConfig[*].presetDeviation and every subsequent config load re-emits a warning listing each deviation. The preset's authoritative topology is overlaid only on an exact match (zero deviations); presetApplied: true appears only in that case. Part-number and PSID differences are expected (firmware/SKU variants) and never block application.

Common Edits

# Change VF count per PF and fabric-specific MTUs
sriov:
  numVfs: 16
  ethernetMtu: 9000
  infinibandMtu: 4000

# Set the DOCA driver version. Advanced users can also forward literal
# environment variables through the generated NicClusterPolicy or
# NicNodePolicy; custom values override any value produced by another
# docaDriver field with the same name.
docaDriver:
  version: "doca3.3.0-26.01-1.0.0.0-6"
  env:
    - name: THIRD_PARTY_RDMA_MODULES
      value: "nvidia_peermem"

# Allow four simultaneous maintenance operations. Network Operator 26.1+
# uses the global Maintenance Operator limits; older releases use the legacy
# SR-IOV/OFED limits where applicable.
maintenance:
  maxParallelOperations: 4
  maxUnavailable: 4
  maxNodeMaintenanceTimeSeconds: 3600
  maxParallelUpgrades: 4

# The selected release supplies the Helm repository URL.
networkOperator:
  # Keep generating/applying Network Operator CRs but let another system own
  # the Helm release. Generate/deploy/validate have the equivalent CLI flag
  # --skip-network-operator-helm; clean reads this persistent ownership setting
  # and retains the release while deleting Network Operator CRs.
  skipHelmChart: true

# Configure NV-IPAM subnets manually
nvIpam:
  perNodeBlockSize: 10
  subnets:
    - subnet: "10.10.0.0/16"
      gateway: "10.10.0.1"

# Namespaces for the secondary-network CRs + example test DaemonSets.
# One independent copy is rendered per namespace (shared resources like
# IPPools and NodePolicies are NOT duplicated). Defaults to ["default"].
networkNamespaces: ["my-namespace"]

Tips

• Start by running discovery (l8k discover) to generate a baseline, then edit it.
• A discovered config can be passed directly to l8k generate without repeating profile flags; use flags only for overrides.
• Use l8k schema to discover the Network Operator release keys supported by the installed l8k version.
• Treat a GA catalog entry as sticky during nightly synchronization: an upstream beta or release candidate for the next patch must not replace its public Network Operator artifact set. Update the entry again when the patch is GA.
• nvIpam subnets are auto-generated if not specified — one per rail using non-routable ranges.
• nvIpam.perNodeBlockSize defaults to 10; generation warns when it is lower than sriov.numVfs because the per-node allocation may be too small for all VFs.
• nvIpam.startingGateway is a YAML-only positive host offset in each automatic subnet (default 1); it must select a usable host below broadcast. For 192.168.2.128/25, use 126 for gateway 192.168.2.254. Manual entries retain their own gateway. nvIpam.offset separately controls subnet spacing; NV-IPAM automatically excludes the gateway from allocation.
• docaDriver.unloadThirdPartyRDMAModules: true auto-populates UNLOAD_THIRD_PARTY_RDMA_MODULES from discovered OFED-dependent modules.
• docaDriver.env is an advanced escape hatch. Values override generated MOFED environment entries by name, duplicate custom names use the last value, and bad driver options can disrupt node networking.
• For release 26.1+, SR-IOV requestor mode requires both the Network Operator drain requestor and the SR-IOV external drainer. l8k renders both; applying only CRs cannot enable their Deployment environment variables.
• Updating an existing release to the generated requestor-mode Helm values requires --overwrite-existing.

See Also

• k8s-launch-kit-shared — Global flags
• k8s-launch-kit-discover — Generate a config from live cluster
• references/config-reference.md — Complete annotated YAML, including maintenance value restrictions

OpenShift

OpenShift configuration uses top-level flavor: ocp, sriov.operatorNamespace, nfd.operatorNamespace, nfd.configurationName, and maintenance.operatorNamespace. Install operator packages externally; l8k configures existing installations.

Standard NIC tuning

nicConfigurationOperator.deployNicConfigurationTemplate defaults to true. Set it to false or pass --deploy-nic-configuration-template=false to opt out; explicit YAML and CLI false values are preserved. Root, generate and discover accept --deploy-nic-configuration-template[=false]. It enables NCO independently of interface naming and renders per-source, east-west PCI-scoped templates for standard SR-IOV, RDMA-shared and host-device profiles on k8s/OCP. VF count comes from sriov.numVfs; link type comes from profile.fabric. Ethernet gets DSCP trust and priority-3 PFC; InfiniBand omits RoCE tuning. Spectrum-X keeps its dedicated required templates independently of this standard-profile toggle. Only SR-IOV disables the Mellanox plugin (Helm values on k8s, additive disablePlugins merge on OCP). Kubernetes external Helm owners must configure that plugin themselves.

Deploy and validate consume the generated template and require all matched NicDevices to report current ConfigUpdateInProgress=False with reason UpdateSuccessful; stale/partial status waits and failures surface. Opting out does not restore applied NIC settings. On k8s, omitted templates (also for unselected groups) block deploy preflight unless --overwrite-existing is used, which deletes them. OCP preserves existing templates and disabled plugins. Generation rejects inventory with selected and excluded ports on the same PCI device. Review whole-NIC scope, including ports absent from inventory, and qualify runtime RoCE support for the installed NCO/DOCA/RHCOS versions before live use. See docs/reference/configuration.md#nic-configuration-and-naming.

Cluster address collision inputs

cluster.podCIDRs and cluster.serviceCIDRs are optional authoritative lists of all applicable IPv4/IPv6 ranges. Omitted/null means discover at deploy; explicit empty lists are invalid. Each override replaces its class of discovery. Generation checks actual IPPools against supplied ranges without API access. Deploy checks all Node InternalIPs and, absent overrides, Node pod ranges and ServiceCIDRs before any writes. Missing pod ranges warn by Node name; missing ServiceCIDR APIs warn, but denied/failed reads and empty served APIs fail. Use authoritative pod ranges for CNIs whose IPAM differs from Node podCIDRs. See configuration for fallback, permissions and the IPPool-only scope.