l8k: Cluster Discovery

PREREQUISITE: Read ../k8s-launch-kit-shared/SKILL.md for install paths, global flags, and output modes.

This workflow uses the default host target; --target host is equivalent. The CLI snapshots discovery arguments and runs the concrete Host discovery operation through the target registry; command syntax and artifacts are unchanged.

Discover cluster hardware and produce a cluster-config.yaml describing NICs, GPUs, rails, and node groups. Use docs/user/profiles.md#check-prerequisites-and-site-inputs for the site-owned decisions and docs/user/discovery.md#review-saved-inventory for the post-discovery review before generation.

Four things to know about the saved file:

• Fresh discovery persists a resolved profile. Without --user-config, discovery fills profile fields from hardware/defaults and applies explicit CLI overrides last. With --user-config, it replaces clusterConfig and discoveredSecondaryNetwork, preserves every other section, and then applies explicit CLI overrides.
• North-south-only groups are dropped. A node group whose NICs are all north-south (BlueField DPUs / OOB NICs) produces no manifests and is omitted from cluster-config.yaml (it would otherwise consume an NV-IPAM subnet slice and shift the per-rail pools). North-south NICs alongside east-west NICs in a mixed group are still listed.
• Dual-port BlueField-3 NICs are north-south. Physical port count comes from the VPD model (dual-port / 2-port), not PCI function count. This rule overrides a false status.dpu, an unlisted SKU, and GPU PIX proximity. Single-port multi-plane SuperNICs keep their existing classification.
• Documentation comments are preserved. The field-reference comments from the source config (l8k-config.yaml or --user-config) are carried over into the saved cluster-config.yaml, with a Generated by l8k discover banner at the top.

l8k discover is self-contained. It does NOT require a pre-installed Network Operator. On every run it bootstraps the NIC Configuration Daemon (and the 5 nic-configuration-operator CRDs, only if missing) into a private namespace nvidia-k8s-launch-kit, reads NicDevice CRs published by the daemon, then tears the namespace down on exit (unless --keep-namespace is set). The daemon image is pulled from <networkOperator.repository>/ nic-configuration-operator-daemon:<networkOperator.componentVersion> (both fields come from l8k-config.yaml).

The temporary discovery daemon requests 64 MiB of memory and has a 512 MiB limit to accommodate hardware probes on nodes with large PCI inventories.

Discovery refresh also replaces discoveredSecondaryNetwork, preserving component ownership policy. Known API-visible installer inspection is read-only; unavailable evidence still saves hardware inventory, but automatic generation requires usable ownership evidence. Unknown host-only installers need an explicit site decision. See platform observations.

For library consumers, read-only discovery is separate from l8k discover. DiscoverReadOnly requires a safe group selector by default; inventory-only callers can opt in to empty selectors and use WorkerNodes as the group membership source. See the read-only library discovery contract in the user guide before using that option.

Usage (from AI agent)

l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml \
  --output json >discovery.json 2>discovery.log &&
  jq . discovery.json

Check the l8k exit status before parsing; retain discovery.log on failure.

Usage (human-interactive)

l8k discover --save-cluster-config <OUTPUT> [--kubeconfig <PATH>]

Flags

Examples

# Basic discovery
l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml

# Using $KUBECONFIG env var (no --kubeconfig needed)
l8k discover --save-cluster-config ./cluster-config.yaml

# Keep the bootstrap namespace for debugging
l8k discover \
  --kubeconfig ~/.kube/config \
  --keep-namespace \
  --save-cluster-config ./cluster-config.yaml

# Refresh clusterConfig in an existing config
l8k discover --user-config my-config.yaml \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml

# Persist explicit profile overrides
l8k discover \
  --kubeconfig ~/.kube/config \
  --fabric infiniband --deployment-type rdma_shared \
  --multirail=false \
  --save-cluster-config ./cluster-config.yaml

# Agent mode (JSON output)
l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml \
  --output json >discovery.json 2>discovery.log

Output Format

The generated cluster-config.yaml contains the final profile plus a clusterConfig[] hardware-group array:

profile:
  fabric: ethernet
  deployment: sriov
  multirail: true
clusterConfig:
  - identifier: "dgx-b200-h100-nvl"
    machineType: DGX-B200
    gpuType: NVIDIA-H100-NVL
    netplanManaged: true
    capabilities:
      nodes:
        sriov: true
        rdma: true
    pfs:
      - deviceID: "101e"
        networkInterface: "eth0"
        rail: 0
    workerNodes: [node-01, node-02]
    nodeSelector:
      nvidia.kubernetes-launch-kit.machine: "dgx-b200-h100-nvl"
    thirdPartyRDMAModules: [nv_peer_mem]

Discovery patches every node in the group with two labels:

• nvidia.kubernetes-launch-kit.machine: <identifier> — per-source-group identity, identical to the persisted group identifier and used as its nodeSelector.
• nvidia.kubernetes-launch-kit.gpu: <gpuType> — used as the merged-group nodeSelector when l8k generate auto-merges source groups sharing a GPU type.

Group identifiers and machine-label values are lowercase and omit complete NVIDIA segments. The same pass maps ThinkSystem to ts and PowerEdge to pe. They are bounded to 30 bytes; long values retain balanced machine/GPU prefixes plus a 6-character deterministic hash, and unused prefix space moves to the longer component. GPU-only labels retain their discovered case and vendor segment, use Kubernetes' 63-byte limit, and keep the existing 8-character shortening hash. When machineType or gpuType couldn't be resolved (GPU operator labels absent and hardware probe failed), a sole source keeps an empty identifier and multiple sources use group-N; the machine label is not written. Exact selector proof is still required; the GPU label is still written when gpuType alone is resolved.

Discovery validates all proposed source identities before label writes and publishes selectors only after a fresh full-node read proves exact worker membership. Patch failures, duplicate identities and stale machine/GPU labels on excluded nodes stop discovery; completed patches may remain. Inspect stale labels before retrying. Unresolved identity requires another proven selector, including hostname for a singleton, rather than a broad PCI-vendor selector.

Prerequisites

• Node Feature Discovery (NFD) is not required. Before bootstrapping the daemon, discovery lists Nodes and renders node-name affinity for nodes that are Ready=True and not spec.unschedulable. The DaemonSet still has no NFD nodeSelector and tolerates taints, so eligible control-plane/tainted nodes are included. NIC-bearing nodes are detected by a sysfs probe for PCI vendor 0x15b3 rather than the NFD feature.node.kubernetes.io/pci-15b3.present label. BlueFields for which mlxprivhost reports level: restricted are excluded because the daemon will not publish NicDevice resources for them. Mixed nodes remain eligible when they have at least one non-restricted NVIDIA NIC. Normal CLI output reports each confirmed restricted PCI function by node/address and a total; the probe summary logs restrictedDevicesSkipped separately from published north-south filtering. Restricted devices remain absent from the saved config.
• Image-pull access from every worker node to <networkOperator.repository>/nic-configuration-operator-daemon:<networkOperator.componentVersion>. Use --image-pull-secrets <name> (or networkOperator.imagePullSecrets in the config file) for private registries.
• Network Operator does NOT need to be pre-installed.

Fresh discovery writes validation.gpuDirect.enabled. It is true only when every worker in every discovered group can satisfy its render bucket's topology-derived validation.gpuDirect.gpuResourceType request; otherwise it is false. With --user-config, the supplied validation section is preserved. Do not add GPU counts or resource maps under clusterConfig.

Tips

• GPU topology uses the NIC daemon's available nvidia-smi, then a verified same-node GPU Operator gpu-feature-discovery main container. The fallback needs pod read/exec and DaemonSet read permissions; it does not change runtime classes, inject GPUs into NIC pods, or start probe pods. Missing labels are not proof of CPU-only hardware. Full host GPU PCI/query/matrix coverage and all-worker/raw-PF agreement are required before publishing topology.
• Read-only GPU enrichment additionally requires WithReadOnlyGPUTopology("<gpu-operator-namespace>") and REST credentials. Default/REST-only calls acquire no GPU probes. Consume the library's context UI sink for unavailable-evidence warnings. GPU probing does not run netplan discovery. Missing/denied/partial observations keep hardware/preset fallback; caller cancellation returns an error. Qualify the actual GPU stack live.
• Fresh profile precedence is hardware/default < explicit CLI. On a rerun with --user-config, no profile fields are recalculated; explicit CLI flags still override their corresponding values.
• A fresh discovery ignores the example profile in the installed reference config. Supplying that file explicitly through --user-config makes its profile values user input and therefore preserves them.
• Fabric is published only after every selected worker/PF/port agrees, including raw sibling PFs behind collapsed rails. Probes use node-local PCI identity; saved representative RDMA names are not universal aliases. Normal discovery batches one exec per worker, at most four concurrently with ten seconds each.
• Read-only library discovery enables this work only through WithReadOnlyFabricDiscovery() plus WithReadOnlyRESTConfig; REST-only callers retain publisher verification without fabric enrichment. Permission/partial failures leave fabric unknown; caller cancellation returns an error.
• If fabric probes are mixed or unconfirmed, discovery still writes the file with an empty profile.fabric; set --fabric on discovery or generation.
• clusterConfig[].netplanManaged is true when any worker has an NVIDIA PF selected by a host netplan match.macaddress plus set-name rule. Discovery checks worker nodes until it finds a match or has a conclusive observation for each, and uses PF MACs only transiently; they are not saved in the shared group config. A readable host /etc without a netplan directory, or a complete readable configuration without MAC-based set-name rules, is a confirmed negative and does not require PF MACs. When no match is found but a worker observation is unavailable or incomplete, or a MAC-based rule cannot be checked because an expected PF MAC is missing, discovery warns and sets clusterConfig[].netplanManagementUnknown. If generation would emit a NicInterfaceNameTemplate for an unknown group, it warns and continues. A confirmed match (netplanManaged: true) still blocks generation; clean up the affected host set-name stanzas and re-run discovery before retrying. Review host netplan configuration and re-run discovery to resolve unknown status; do not clear either field manually. Refresh configs created before this unknown-state field was added.
• The bootstrap namespace is always nvidia-k8s-launch-kit — not configurable. The daemon's SA / ClusterRole / ClusterRoleBinding are renamed to k8s-launch-kit-nic-config-daemon so they don't collide with the cluster-scoped names a coexisting Network Operator install would create.
• CRDs (nicdevices, nicconfigurationtemplates, etc.) are applied only when missing. If they already exist (because Network Operator or a prior l8k discover run created them), they're left alone — discovery never overwrites a different version.
• After discovery finishes the namespace is torn down (cascade-delete handles SA / RoleBinding / DaemonSet / pods / NicDevice CRs); the CRDs intentionally persist so any external NicDevice consumers survive. Cluster-scoped ClusterRole / ClusterRoleBinding are deleted explicitly.
• Pass --keep-namespace to leave the namespace in place — useful when debugging daemon pod start-up failures (kubectl describe pod -n nvidia-k8s-launch-kit).
• Before bootstrapping, discovery pre-cleans the nvidia-k8s-launch-kit namespace (deletes any leftover DaemonSet/pods/NicDevice CRs from a crashed prior run and waits up to 2 min for it to clear) so a fresh daemon is never layered on stale state.
• Discovery waits up to 5 minutes for daemon pods to be Ready. It proceeds with the Ready pods when other daemon pods are on NotReady nodes, are stuck (e.g. ImagePullBackOff/CrashLoopBackOff), or the wait timeout expires while the caller's context remains active. If the caller context is cancelled or reaches its deadline first, discovery returns that context error. It aborts if no pod ever becomes Ready. Common causes of a total failure: image tag missing in your registry, pull-secret missing.
• If discovery reports "no nodes with an NVIDIA NIC (PCI vendor 15b3) were found", the daemon ran but no node's sysfs exposed a 0x15b3 device (no Mellanox/NVIDIA NICs, or /sys not mounted/readable in the pod).
• If discovery reports "no discoverable NVIDIA NICs were found", every detected BlueField is in zero-trust (restricted) mode. Those devices cannot be configured from the host and intentionally do not publish NicDevice resources.
• After determining each group's (machineType, gpuType), discovery looks up a topology preset under presets/ using exact-match lookup on that pair. A matching preset overrides heuristic-derived topology fields (traffic class, rail, NUMA, GPU affinity). There is no any-GPU fallback — a preset with empty gpuType: is rejected at load time. If no preset matches, discovery proceeds with heuristic classification.
• If you already know the SKU and want to skip cluster discovery entirely, use l8k generate --for <preset> (see k8s-launch-kit-generate).

Rail collapsing (one rail per NIC)

By default (--collapse-nic-rails, default true) discovery advertises one rail per NIC, not one per PF. A single NIC can expose several east-west PFs that are planes of one physical port (Spectrum-X multi-plane ConnectX-8/9). For those, only the master PF (lowest PCI function) is written to cluster-config.yaml, so the rail count reflects physical NICs — e.g. an 8-PF ConnectX-8 node yields 4 rails, not 8. The per-PF planes are reconstructed at generate time from the multiplane mode + numberOfPlanes.

The exception is a genuinely dual-port NIC: when the NIC's VPD model name (read from NicDevice.Status.modelName, surfaced as the model field on each PF) contains a port-count keyword like 2-port/Dual-port, each port is kept as its own rail. Single-port (1P) and multi-plane models (no port-count keyword) collapse. An empty/unreadable model collapses by default.

Pass --collapse-nic-rails=false to restore the legacy one-rail-per-PF behaviour (handy on dev setups). Note: a collapsed group no longer has the same PF count as a full-PF topology preset, so such presets won't apply by exact match and the live (collapsed) classification is kept (a preset-deviation warning is emitted).

See Also

• k8s-launch-kit-shared — Global flags and output modes
• k8s-launch-kit-config — Understand the config file discovery produces
• k8s-launch-kit-generate — Generate manifests from cluster config
• Read-only library discovery — Existing-daemon inventory and publisher-inspection permissions

OpenShift

On OpenShift, pass --flavor ocp. Use explicit clusterConfig[].workerNodes in --user-config to bound discovery; the temporary discovery ServiceAccount gets privileged SCC use in its own namespace. Follow the complete prerequisite and review sequence in docs/user/openshift.md.

RA2.4 and RA3.1 integration

RA2.3 is limited to Network Operator 26.7; RA2.4 and RA3.1 default to 26.10 and accept newer catalogued releases. RA2.4 and RA3.1 need a full doSPCX ConfigMap, preserving binaryData and annotations and rendering in the resolved operator namespace. With RA2.4/RA3.1, VR with ConnectX-9 defaults to hwplb and eight planes; explicit YAML and CLI values take precedence. Explicit swplb with an omitted plane count retains the CX9 four-plane fallback; this does not qualify that recipe. VR discovery requires eight east-west ASIC NICs in four reciprocal NetworkBay pairs. NetworkBays are numbered by ascending first-ASIC (asic: 1) PCI address; rails cross first/second ASICs between NetworkBays 1/2 and 3/4. Missing or inconsistent status fails discovery. See docs/user/discovery.md for the rail mapping. Discovery saves the effective RA naming in clusterConfig[].spectrumX.interfaceNames (netdevPrefix and rdmaPrefix). These are derived values: resolution refreshes them after mode or naming changes, including for older configs without this block. Change the existing spectrumX.singlePlane, hwplb, or swplb prefix settings for custom naming. PF inventory names remain observed values. Naming templates and PF selectors consume the same resolved per-group prefixes; standard profiles are unaffected. VR HWPLB uses eth_r%rail_id%_p%plane_id% and roce_r%rail_id%_n%nic_id%; traditional HWPLB retains roce_r%rail_id%.

VR generation supports IPv4 and IPv6 HWPLB with eight planes on four dual-NIC rails. It emits four per-rail naming templates (pfsPerNic: 4), eight per-NIC railRnicN topology entries and CIDRPools, and eight workload attachments. Each NIC endpoint has four PFs, one schedulable VF and its own IPv4 /31 or IPv6 /64. DRA uses one claim per logical rail with one GPU and two VF requests. NIC input order is preserved; it must agree with the physical switch topology. RDMA names are roce_rR_n0/roce_rR_n1; custom RDMA prefixes require %nic_id% and no plane placeholder. Compatible NCO must support disjoint PCI naming templates. Reference topology JSON must include all eight planes per rail and explicit zero-based host attributes.host_index; AIR uses its parsed host ordinal. VR IPv4 defaults to site-zero 240.0.0.0/8, with NIC(1), rail(2), POD(5), SU(7), host(8), peer(1) bits after the first octet. Eight-plane POD is always zero and host index is at most 251. Pools are per NIC/rail /11, allocations are /31 with even host and odd gateway, and routes cover /11 and /8. VR IPv6 uses fd02:000S:NRPP:UUHH::peer/64: site nibble S, NIC nibble N, rail nibble R, POD byte PP, SU byte UU, and host byte HH. profile.spectrumX.ipv6SiteID selects site 0..15 (default 0; YAML-only). The two NICs use distinct per-NIC/rail /40 pools; each host gets a static /64, with host ::1, gateway ::2 (excluded from allocation), and route fd02::/28. POD remains zero, SU is 0..127, and host index is 0..251. hostFirstOctet affects only IPv4; nonzero ipv6SiteID is rejected outside VR IPv6. Non-HWPLB VR, incomplete topology, and extra software-plane offsets are rejected. Generation does not qualify runtime support or configure switches. 26.10 beta.2 lacks the packaged NCO platformType CRD: verify a compatible NCO implementation and matching CRDs before deployment. Deploy checks the served NCT schema after policy bootstrap and blocks additional resource apply when platformType support is absent or cannot be inspected. External Helm, overwrite, and dry-run do not bypass this check; dry-run needs APIs installed already.

Derived clusterConfig[].spectrumX.platformType uses the longest case-insensitive substring from the internal doSPCX platform list in gpuType. A machineType containing VR (case-insensitive) takes precedence and selects vr, even with an empty gpuType. No preset or mapping setting is needed. Discovery saves empty values and warns; generation rejects unresolved selected RA2.4/RA3.1 groups. RTX/CX8 defaults to none/one plane. Mapping does not itself qualify hardware. Optional spectrumX.ovsConfig is a string map; clusterConfig[].spectrumX.swPlaneByRail assigns existing HWPLB rails, defaults to zero, and must agree across merged groups. Refresh preserves explicit plane assignments only for unchanged source/rail hardware and verified unchanged worker membership (ordering does not matter). Changed or unknown membership drops prior assignments with a review/reapply warning. Every selected RA2.4/RA3.1 source needs complete dense rail IDs, uniform NICs per rail, and a divisible planes/NIC ratio; merged sources must have the same effective layout. Merged GPU selectors must not include excluded same-GPU sources in other buckets; select one source in that case. One physical NIC cannot span rails (the current ThinkSystem-SR650-V4-RTX-PRO-6000 preset has this unsupported layout). Multiple hardware partitions inside one logical rail are outside this layout. Feature gates are not added.