l8k: Validate

PREREQUISITE: Read ../k8s-launch-kit-shared/SKILL.md for install paths, global flags, and exit codes.

This workflow uses the default host target; --target host is equivalent. The CLI snapshots validation arguments and runs the standalone Host validation and report service through the target registry; checks and report semantics are unchanged.

Verify that a previously generated and deployed NVIDIA networking deployment is correctly applied and matches the selected Network Operator release. Apply docs/user/validation.md#worked-acceptance-decisions to the report; compare intended and completed worker/rail coverage before acceptance.

External Multus/CNI prerequisite rows report API-level coverage, in-progress rollouts, ownership/coverage errors, or UNVERIFIED evidence. Non-detection and denied optional health reads do not prove missing networking or successful readiness. See external prerequisites.

InfiniBand partition diagnostics

Connectivity setup inspects selected IPoIB interfaces in test pods and warns about limited-only or empty P_Key tables. Unavailable or ambiguous evidence is reported as unverified. These advisory diagnostics appear in text, JSON and HTML; they do not change exit status or run during static validation. Follow the partition diagnostic guidance for scope, limitations and subnet manager remediation.

What it checks

1. Network Operator Helm release version. Reads the chart's appVersion from any release Secret named sh.helm.release.v1.<release>.v<N> whose release name contains "network-operator", in the operator namespace. Compares with the version expected by networkOperator.selectedRelease in cluster-config.yaml (looked up in l8k's embedded release catalog).
2. Manifest state. Every YAML manifest under --deployment-files (skipping example workloads and Helm values.yaml) is fetched from the cluster and classified by the per-Kind resource-state registry. NicConfigurationTemplate and NicFirmwareTemplate wait for the operator-populated status.nicDevices list to reflect current node, NIC type, PCI-address, serial-number, and part-number selectors and validate only the named devices. Their propagated template payload and condition observedGeneration must also be current; unrelated discovered NIC configuration state is ignored. A configuration template checks FirmwareUpdateInProgress only for a matched device with spec.firmware, so a stale firmware condition does not block a deployment with no NicFirmwareTemplate. Each manifest is reported READY, IN-PROGRESS, ERROR, or MISSING.
3. Connectivity matrix. By default, l8k validate applies the generated example DaemonSet, waits for ready pods, and runs source-bound gateway, icmp, rping, and ib_write_bw tests. When validation.gpuDirect.enabled is true and ib_write_bw is selected, a distinct DMA-BUF bandwidth family uses the connected GPU index for each source and destination rail. Profile templates declare both validation containers: the release-specific full-runtime DOCA image for the RDMA checks and netshoot for ICMP and route checks. GPUDirect-enabled templates set LD_LIBRARY_PATH on the DOCA container to prefer host-injected NVIDIA driver libraries over bundled CUDA compatibility libraries. Validate applies the generated DaemonSet without runtime container injection. The default mode is strict.

If --deployment-files contains only user-provided *example*.yaml test DaemonSets, run connectivity only. If values.yaml or any non-example YAML is also present, run the complete validation pipeline. The DaemonSet does not need to be generated by Launch Kit and no separate manifest flag is used.

Validation also reports deploy-preflight drift without remediating it.

Validate constructs one strict desired-artifact snapshot before cluster checks. Malformed or nameless documents fail with source details and exit 2; values.yml and case variants must be renamed to values.yaml. --wait, connectivity, and reports reuse that desired snapshot while reading fresh cluster state. SriovNetworkPoolConfig, SriovNetworkNodePolicy, and OVSNetwork objects labeled with spectrumx.nvidia.com/owner-name are excluded from stray results because the Spectrum-X operator owns them as children of SpectrumXRailPoolConfig.

Exit code is non-zero (4) on any missing manifest, version mismatch, or gating connectivity failure. A skipped, empty, or incomplete connectivity-only matrix also exits 4; every selected check family must produce a gating test (gateway permits explicit no-configured-gateway applicability skips), and non-gating cross-rail observations do not count. Version checks soft-skip when prerequisites are absent, but connectivity requires a user-owned cluster config.

The Helm version check selects the highest revision of the exact network-operator release before considering names that contain network-operator (case-insensitive). A single custom-named release remains a fallback; multiple fallback releases produce a skipped check with an ambiguity reason. A skipped check is not a verified version match. Values validation still uses the exact network-operator release.

If networkOperator.skipHelmChart: true is set, or the user passes --skip-network-operator-helm, both Helm release/version and values checks are reported as skipped. Component-version, manifest, stray-resource, connectivity, and report stages still run.

SR-IOV policy and IPPool checks compare live selectors against workers from the intended bundle's source-group metadata and effective config; live annotations cannot redefine intent. Scope drift or denied reads fails inspection. All-empty preset/manual inventory remains visibly target-scope-unverified alongside device readiness; do not interpret READY as inventory-scope proof. Missing expected PFs remain errors and empty SR-IOV interface status remains in progress. Preserve the resolved sidecar, or supply explicit config whose source IDs match the bundle.

Usage

l8k validate [--user-config <PATH>] [--deployment-files <DIR>] [--kubeconfig <PATH>]

Flags

Gateway checks

Gateway checks run first after test pods are ready, before all pair-based connectivity families, regardless of validation.checks order. Every mode checks each pod/secondary rail once; --validation-checks gateway also works with one pod. Validation follows the exact namespaced NetworkAttachmentDefinition from Multus status and its single explicit NV-IPAM poolName to the live IPPool in networkOperator.namespace. Its spec.gateway is probed even when the pod has only a connected route and no via route. Source IP membership in that pool's subnet must match. Standalone CNI and plugin chains are supported; multiple pools/plugins or CNI args overrides fail as ambiguous. Validation also checks the matching pod network-selection entry by NAD namespace/name and requested interface. Nonempty per-pod cni-args on NV-IPAM attachments are unsupported and fail before pool lookup; malformed or ambiguous selections also fail gateway measurement. Args on unrelated attachments do not override the selected gateway. The runner uses runtime interface routes when no IPPool gateway is set, for other IPAM providers, and for explicit poolType: cidrpool. CIDRPool per-node configured allocations are not separately resolved by this check. It never guesses a .1 address or uses the management default route. A missing, malformed, denied or ambiguous gateway lookup fails measurement. When neither configuration nor interface routes has a gateway, the row is explicitly SKIPPED, with zero contribution to passed or failed counts. This applicability assessment can exit zero when gateway is the only selected family; zero endpoints and missing coverage for other selected families still fail connectivity-only acceptance.

A configured gateway requires a positive, source-IP-bound ICMP echo after route qualification selects the intended interface. IPv6 link-local gateways are scoped to that interface. No echo fails gateway ICMP reachability; it does not prove that ARP/ND resolution failed because a gateway may filter ICMP. Other selected families continue after a gateway failure. Text reports gateway endpoints and status. JSON, HTML and JUnit retain gateway configuration, route and probe evidence; skipped rows remain skipped in JUnit. The automatic deadline includes a shared 10-second NAD/IPPool lookup limit, a 10-second runtime discovery limit when needed, a 10-second source-route limit and a 5-second echo limit per endpoint. Direct Go callers must supply connectivity.Options.IPAMNamespace for NV-IPAM IPPool attachments.

Connectivity Modes

• quick: all same-rail node pairs plus one non-gating cross-rail canary per source-rail/destination-rail mapping.
• full: every source rail × every destination rail × every ordered pod pair; cross-rail results are reported but do not gate pass/fail.
• strict: full matrix. Cross-rail gates by profile.routing: source-based must succeed, destination-based must stay isolated.

Connectivity discovery supports IPv4-only and IPv6-only secondary attachments. On dual-stack attachments it retains IPv4 preference; it does not test both families. IPv6 uses ping -6, ip -6 route get, and ib_write_bw --ipv6 --ipv6-addr on both endpoints. rping accepts IPv6 addresses directly. Link-local and malformed source endpoint addresses are excluded; IPv6 link-local gateways use their secondary interface scope.

All checks are source-bound. Before every ICMP probe, validate confirms that ip route get <dst> from <src> selects the requested source interface. A route using another interface is reported as not connected for that rail pair rather than forcing traffic onto it. When the source rail is selected, ICMP uses ping -I <src-ip> so source-based policy rules participate naturally. Routing diagnostics expect the source interface for source-based routing and the destination interface for destination-based routing. rping uses -I <src-ip>, and ib_write_bw uses --bind_source_ip <src-ip>. GPUDirect adds --use_cuda=<endpoint-index> --use_cuda_dmabuf independently on the client and server. Treat missing or ambiguous connectedGPU topology as a failure; never substitute GPU 0. Pull Secrets come from networkOperator.imagePullSecrets and must exist in every validation namespace.

With the default --connectivity-timeout=0, validate logs one total automatic budget after planning the matrix. The calculation reflects the enabled test families, per-command limits, ordered pod-pair batches, bounded workload setup, cleanup, and a safety margin. A positive flag value replaces that calculation with a user-supplied hard deadline.

The minimum connectivity config is:

profile:
  routing: source-based # or destination-based
validation:
  gpuDirect:
    enabled: false

The test workload must be an apps/v1 DaemonSet with a name, namespace, netshoot route helper, and an RDMA tools container when RDMA checks are selected. When GPUDirect is enabled with ib_write_bw, each clusterConfig group must list workers and east-west PFs with a non-negative rail and connectedGPU: GPU<N>.

Examples

# Defaults: ./cluster-config.yaml + ./deployment, $KUBECONFIG
l8k validate

# Explicit paths
l8k validate --user-config ./cluster-config.yaml \
  --deployment-files ./deployment \
  --kubeconfig ~/.kube/config

# Connectivity only from user-provided *example*.yaml DaemonSets
l8k validate --user-config ./cluster-config.yaml \
  --deployment-files ./connectivity-test \
  --kubeconfig ~/.kube/config

# Agent mode (JSON objects on stdout, logs on stderr; preserve l8k status)
l8k validate --output json >validation.jsonl 2>validation.log &&
  jq -s . validation.jsonl

# Diagnose stage or batch progress without raw command output
l8k validate --log-level debug

# Capture bounded route, ICMP, RDMA client, and RDMA server evidence
l8k validate --log-level trace --keep

Debug logs show the endpoint inventory, plan, source-route cache statistics, static checks, stages, RDMA batches, cleanup, report writes, elapsed time, and remaining timeout. Trace adds bounded commands and per-test stdout/stderr. Failed RDMA server logs are collected before the temporary files and test workload are removed. Add --keep only when follow-up pod inspection is needed.

JUnit report

Use l8k validate --output json --junit-path ./reports/validation.xml to retain both formats. The XML has a suite per catalog connectivity family, with individual directional pod/rail testcase children and suite counters. Disabled or unexecuted checks are skipped with reasons; existing gating semantics apply. Reports retain partial results and execution errors. A write failure fails an otherwise successful command; existing validation failure codes are preserved. Coverage failures remain visible alongside other failures, and preset cases include matching and skipped groups. See docs/user/validation.md#junit-xml for names, timing and interpretation. Preserve the process exit status.

Output

Text mode prints a short report:

Network Operator release
  selectedRelease: 26.4
  expected version: v26.4.0-beta.6
  deployed: network-operator (chart=26.4.0-beta.6 app=v26.4.0-beta.6 rev=3 status=deployed)
  result: MATCH

Manifests
  [READY      ] NicClusterPolicy/nic-cluster-policy in (cluster-scoped)
  [IN-PROGRESS] NicConfigurationTemplate/spectrum-x-config in network-operator — waiting for nic-configuration-operator to populate status.nicDevices with matched devices
  [MISSING    ] SriovNetwork/sriov-network-rail-0 in default — not found in cluster
  ...

Summary: 1/3 ready, 1 in-progress, 0 error, 1 missing; version: match; topology mismatches: 0 group(s)

JSON mode emits a sequence of objects. Full validation emits versionCheck, manifests, and summary; connectivity and the HTML reportPath are separate objects when those stages run. Parse with jq -s or a streaming parser and preserve the process exit status. summary.success is not the final verdict. Full Kubernetes validation can return success with skipped/incomplete connectivity; require explicit coverage evidence before accepting deployment. See docs/user/validation.md#acceptance-outcomes.

Omit --user-config to reuse the bundle's .l8k/resolved-config.yaml. An explicit config overrides that metadata. Connectivity accepts the sidecar or a user-owned config with explicit routing and GPUDirect decisions.

When this skill activates

Trigger phrases include: "validate my deployment", "is my cluster correct", "are all the manifests applied", "does the chart version match", "did the deploy succeed", or any discrepancy claim about expected vs deployed state.

See Also

• k8s-launch-kit-deploy — apply manifests
• k8s-launch-kit-troubleshoot — investigate failures uncovered by validate
• k8s-launch-kit-shared — global flags and exit codes

OpenShift

OpenShift validation checks Network Operator CSV version and creates a temporary l8k-validation-* namespace for each source workload namespace. It copies required NADs and image pull secrets and creates dedicated ServiceAccount/SCC/RBAC there. The kubeconfig needs namespace/SCC creation and source NAD/Secret read permissions. Cleanup removes the temporary resources; --keep retains them. Inspect exact retained names after interrupted cleanup. Platform-managed Multus and reference CNI plugin installer rows are UNVERIFIED; their DaemonSet count, selectors, pods, and rollout status do not gate validation. NCP ownership checks still reject managed OpenShift artifacts, implicit ownership transitions, and failed mandatory reads. NCP readiness and connectivity still apply. NCP readiness also excludes non-ready Multus/CNI appliedStates when both desired and live specs leave them external. Every configured component in either spec must report ready; missing coverage and other component failures keep the aggregate error/progress state. Unknown spec fields, malformed/duplicate status, and policy-level reasons retain the original aggregate verdict. Live YAML keeps the raw controller status. OpenShift with connectivity enabled requires at least one gating test per selected family, except explicit no-configured-gateway applicability skips; in-progress manifests and incomplete coverage fail. Other profiles beyond SR-IOV Ethernet still need live hardware qualification; see docs/user/openshift.md.

Standard NIC tuning

nicConfigurationOperator.deployNicConfigurationTemplate defaults to true. Set it to false or pass --deploy-nic-configuration-template=false to opt out; explicit YAML and CLI false values are preserved. Root, generate and discover accept --deploy-nic-configuration-template[=false]. It enables NCO independently of interface naming and renders per-source, east-west PCI-scoped templates for standard SR-IOV, RDMA-shared and host-device profiles on k8s/OCP. VF count comes from sriov.numVfs; link type comes from profile.fabric. Ethernet gets DSCP trust and priority-3 PFC; InfiniBand omits RoCE tuning. Spectrum-X keeps its dedicated required templates independently of this standard-profile toggle. Only SR-IOV disables the Mellanox plugin (Helm values on k8s, additive disablePlugins merge on OCP). Kubernetes external Helm owners must configure that plugin themselves.

Deploy and validate consume the generated template and require all matched NicDevices to report current ConfigUpdateInProgress=False with reason UpdateSuccessful; stale/partial status waits and failures surface. Opting out does not restore applied NIC settings. On k8s, omitted templates (also for unselected groups) block deploy preflight unless --overwrite-existing is used, which deletes them. OCP preserves existing templates and disabled plugins. Generation rejects inventory with selected and excluded ports on the same PCI device. Review whole-NIC scope, including ports absent from inventory, and qualify runtime RoCE support for the installed NCO/DOCA/RHCOS versions before live use. See docs/reference/configuration.md#nic-configuration-and-naming.