l8k: Troubleshooting

PREREQUISITE: Read ../k8s-launch-kit-shared/SKILL.md for install paths, global flags, and exit codes.

Debug NVIDIA Network Operator issues on Kubernetes, with or without a sosreport.

l8k Troubleshooting Commands

# Show validation endpoints, planning, route-cache statistics, stage/batch
# progress, timings, and failed RDMA evidence.
l8k validate --kubeconfig <PATH> --deployment-files <DIR> --log-level debug

# Add bounded route command output, RDMA client stdout/stderr, and server logs.
l8k validate --kubeconfig <PATH> --deployment-files <DIR> --log-level trace --keep

# Collect a diagnostic dump from the cluster
l8k sosreport [--kubeconfig <PATH>] --output-dir ./sosreport

Start with debug. Escalate to trace when a route, ICMP, rping, or ib_write_bw failure needs command output. Trace fields are bounded; failed RDMA server logs are collected before cleanup. Add --keep only when the workload must remain available for follow-up kubectl exec inspection.

l8k sosreport gathers cluster state, CRDs, operator logs, and per-node NIC info into a structured directory for offline analysis. Release installations include the Network Operator collection script under share/l8k/scripts/; the command never downloads executable code at runtime. If the script is missing, report the upstream URL and exact expected path emitted by l8k so the user can install it manually. Use the diagnostic commands and triage workflow below to interpret the dump.

Diagnostic Commands

# NicClusterPolicy status (check state: ready vs notReady)
kubectl get nicclusterpolicy -o yaml

# Network operator pods
kubectl get pods -n <operator-ns> -o wide

# SR-IOV node states (VF allocation)
kubectl get sriovnetworknodestates -A -o yaml

# OFED driver pod logs
kubectl logs -n <operator-ns> -l app=mofed-<os> --tail=100

# NIC configuration daemon logs
kubectl logs -n <operator-ns> -l app=nic-configuration-daemon --tail=100

# Check for pods stuck on network resources
kubectl get pods -A -o wide | grep -E 'ContainerCreating|Init'

Common Failure Patterns

For detailed triage workflow, read references/common-failures.md.

sosreport Analysis

If the user has a pre-collected sosreport directory (from l8k sosreport or manual collection):

sosreport/
├── metadata/          # Cluster info, node list
├── crds/              # NicClusterPolicy, SriovNetworkNodePolicy, IPPool, etc.
├── operator/          # Network operator pod logs
├── nodes/             # Per-node device info
└── network/           # Interface config, routing tables

Triage Checklist

1. Read metadata/diagnostic-summary.yaml for overview
2. Check pod health in operator/pods.yaml
3. Inspect CRDs in crds/ for status fields
4. Read operator logs in operator/logs/ for errors
5. Check per-node NIC state in nodes/<node>/

See Also

• k8s-launch-kit-shared — Exit codes and error structure
• k8s-launch-kit-discover — Re-discover to verify hardware state
• references/common-failures.md — Detailed triage workflow

OpenShift

For OpenShift, check expected operator CSV/API status, SR-IOV device prefix openshift.io, NV-IPAM pools in the Network Operator namespace, and temporary SCC admission in the workload namespace.