GKE Kueue Workload & Admission Debugging Playbook

[!WARNING] This skill is experimental and under active validation for Cluster Toolkit. Diagnostic queries are read-only, but remediation plans must be reviewed carefully by cluster operators before execution.

Use this playbook when distributed AI/ML training jobs (PyTorchJob, RayCluster, JobSet, Job, LeaderWorkerSet) submitted to GKE remain indefinitely in Pending, Inadmissible, Admitted: False, or experience unexpected evictions.

1. Diagnostic Triage (Read-Only)

Step 0: Fast-Path Workload Inspection (SRE First Response)

If investigating a parent Job, JobSet, or RayCluster, locate its generated Workload name first:

# Method A: Direct lookup via owner UID (safely queries installed controllers without failing on missing CRDs)
JOB_UID=$(kubectl get job <PARENT_JOB_NAME> -n <NAMESPACE> -o jsonpath='{.metadata.uid}' 2>/dev/null)
if [ -z "$JOB_UID" ]; then
  for kind in jobset raycluster rayjob pytorchjob mpijob leaderworkerset; do
    JOB_UID=$(kubectl get "$kind" <PARENT_JOB_NAME> -n <NAMESPACE> -o jsonpath='{.metadata.uid}' 2>/dev/null)
    [ -n "$JOB_UID" ] && break
  done
fi
[ -n "$JOB_UID" ] && kubectl get workloads -n <NAMESPACE> -l "kueue.x-k8s.io/job-uid=$JOB_UID"

# Method B: Universal filter by parent job name prefix
kubectl get workloads -n <NAMESPACE> | grep <PARENT_JOB_NAME>

Always inspect high-level workload events and conditions:

kubectl describe workload <WORKLOAD_NAME> -n <NAMESPACE>

Look directly at the bottom Events: section for QuotaReserved, AdmissionCheckFailed, Inadmissible, Preempted, or WorkloadEvicted.

Step 1: Query Workload Conditions, Evictions & LocalQueue Assignment

Inspect workloads in the target namespace. Do NOT query Pods directly -- Kueue suspends Jobs before Pod creation (.spec.suspend: true).

kubectl get workloads.kueue.x-k8s.io -n <NAMESPACE> -o custom-columns='NAME:.metadata.name,QUEUE:.spec.queueName,RESERVED:.status.conditions[?(@.type=="QuotaReserved")].status,ADMITTED:.status.conditions[?(@.type=="Admitted")].status,EVICTED:.status.conditions[?(@.type=="Evicted")].status,RESERVED_REASON:.status.conditions[?(@.type=="QuotaReserved")].reason,EVICTED_REASON:.status.conditions[?(@.type=="Evicted")].reason,MSG:.status.conditions[?(@.type=="QuotaReserved")].message'

Condition Interpretation:

• QuotaReserved: False, Reason: Pending or Inadmissible: Workload is either waiting for quota (Step 3) OR failed flavor matching (Step 4). Always inspect MSG:
  • If MSG indicates insufficient quota ... > maximum capacity, proceed to Step 3.
  • If MSG indicates doesn't match node affinity or untolerated taint, proceed to Step 4 and Step 6.
• QuotaReserved: True, Admitted: False: Quota is reserved, but the workload is gated by AdmissionChecks (e.g., GKE DWS Flex-start, TPU Dynamic Slicing). Inspect checks:

  kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{range .status.admissionChecks[*]}{"Check: "}{.name}{" | State: "}{.state}{" | RequeueAfter: "}{.requeueAfterSeconds}{"s | Message: "}{.message}{"\n"}{end}'

• Evicted: True: Workload was interrupted after admission. Check EVICTED_REASON:
  • Preempted: Evicted to reclaim quota or satisfy higher-priority job. Proceed to Step 5.
  • PodsReadyTimeout: Worker pods failed to become ready within waitForPodsReady.timeout (check pod image pulls / driver daemonsets).
  • AdmissionCheck: An admission check transitioned to Retry/Rejected while running.
  • Deactivation: Workload deactivated (spec.active: false) due to a rejected admission check.
  • ClusterQueueStopped / LocalQueueStopped: Queue was paused with stopPolicy: HoldAndDrain.

Step 2: Verify LocalQueue & Upstream ClusterQueue Binding

Verify that the LocalQueue exists, routes to an active ClusterQueue, and is not administratively stopped:

kubectl get localqueue <QUEUE_NAME> -n <NAMESPACE> -o custom-columns='NAME:.metadata.name,CLUSTER_QUEUE:.spec.clusterQueue,STOP_POLICY:.spec.stopPolicy,ACTIVE:.status.conditions[?(@.type=="Active")].status,REASON:.status.conditions[?(@.type=="Active")].reason,PENDING:.status.pendingWorkloads,RESERVING:.status.reservingWorkloads,ADMITTED:.status.admittedWorkloads'

Root Cause Check:

• If STOP_POLICY is Hold or HoldAndDrain, the queue is inactive and will not admit new workloads.
• If ACTIVE is False or the resource does not exist, the workload will remain pending indefinitely.
• RESERVING indicates workloads that have passed quota checks but are awaiting AdmissionChecks.

Step 3: Audit ClusterQueue Quotas, Cohorts, and Reservation vs. Usage

Query the upstream ClusterQueue without dumping full YAML. Extract queue policy, nominal quotas, borrowing limits, reservations, and active usage:

# 1. Inspect ClusterQueue policy, cohort, and strategy
kubectl get clusterqueue <CLUSTER_QUEUE_NAME> -o jsonpath='{"ClusterQueue: "}{.metadata.name}{"\nCohort: "}{.spec.cohortName}{.spec.cohort}{"\nQueueingStrategy: "}{.spec.queueingStrategy}{"\nStopPolicy: "}{.spec.stopPolicy}{"\nPendingWorkloads: "}{.status.pendingWorkloads}{" | ReservingWorkloads: "}{.status.reservingWorkloads}{" | AdmittedWorkloads: "}{.status.admittedWorkloads}{"\n---\n"}{range .spec.resourceGroups[*].flavors[*]}{"Flavor: "}{.name}{"\n"}{range .resources[*]}{"  Resource: "}{.name}{" | Nominal: "}{.nominalQuota}{" | BorrowingLimit: "}{.borrowingLimit}{"\n"}{end}{end}'

# 2. Check live reservations vs. actual running pod usage
kubectl get clusterqueue <CLUSTER_QUEUE_NAME> -o jsonpath='{"=== FLAVORS RESERVATION (Admitted + Reserving) ===\n"}{range .status.flavorsReservation[*]}{"Flavor: "}{.name}{"\n"}{range .resources[*]}{"  Resource: "}{.name}{" | Reserved: "}{.total}{" | Borrowed: "}{.borrowed}{"\n"}{end}{end}{"=== FLAVORS USAGE (Actively Admitted Pods) ===\n"}{range .status.flavorsUsage[*]}{"Flavor: "}{.name}{"\n"}{range .resources[*]}{"  Resource: "}{.name}{" | Used: "}{.total}{" | Borrowed: "}{.borrowed}{"\n"}{end}{end}'

Root Cause Check:

• Head-of-Line (HOL) Blocking: If QueueingStrategy is StrictFIFO, an inadmissible older job at the head of the queue blocks all newer jobs across the cluster even if quota is free! Inspect oldest pending workloads for this queue:

  # In target namespace (sorted by creation timestamp to spot head-of-line blockers)
  kubectl get workloads -n <NAMESPACE> --sort-by=.metadata.creationTimestamp -o custom-columns='NAME:.metadata.name,QUEUE:.spec.queueName,RESERVED:.status.conditions[?(@.type=="QuotaReserved")].status,CREATED:.metadata.creationTimestamp' | head -n 10
  
  # Or across all namespaces sharing this ClusterQueue
  kubectl get workloads -A --sort-by=.metadata.creationTimestamp -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,QUEUE:.spec.queueName,RESERVED:.status.conditions[?(@.type=="QuotaReserved")].status,CREATED:.metadata.creationTimestamp' | head -n 10

• Cohort Saturation: If borrowingLimit is empty/null, borrowing is unlimited up to the cohort's total available quota. If total >= nominalQuota and cohort borrowing is exhausted, the queue must wait for active jobs to finish. For multi-tenant quota sharing architectures, refer to the GKE Kueue Cohort Tutorial.
• Reservation vs. Usage Delta: If Reserved is high but Used is 0, quota is tied up by workloads waiting on AdmissionChecks (e.g. DWS capacity acquisition).

[!NOTE] Tenant RBAC Fallback: If querying ClusterQueue returns Error from server (Forbidden), inspect the namespaced admission message directly:

kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{.status.conditions[*].message}'

Step 4: Verify ResourceFlavor Node Selectors, Taints, and Nodepools

When QuotaReserved: False with Reason: Pending or Inadmissible (and MSG indicates "couldn't assign flavors" or "doesn't match node affinity"):

# 1. Inspect ResourceFlavors in cluster (nodeLabels, nodeTaints, tolerations, topology)
kubectl get resourceflavor -o jsonpath='{range .items[*]}{"Flavor: "}{.metadata.name}{"\nNodeLabels: "}{.spec.nodeLabels}{"\nNodeTaints: "}{.spec.nodeTaints}{"\nTolerations: "}{.spec.tolerations}{"\nTopologyName: "}{.spec.topologyName}{"\n\n"}{end}'

# Or inspect a single flavor directly:
# kubectl get resourceflavor <FLAVOR_NAME> -o jsonpath='{"Flavor: "}{.metadata.name}{"\nNodeLabels: "}{.spec.nodeLabels}{"\nNodeTaints: "}{.spec.nodeTaints}{"\nTolerations: "}{.spec.tolerations}{"\nTopologyName: "}{.spec.topologyName}{"\n"}'

# 2. GKE accelerator node inspection (custom columns for GPU/TPU labels, taints, and nodepools)
kubectl get nodes -o custom-columns='NAME:.metadata.name,NODEPOOL:.metadata.labels.cloud\.google\.com/gke-nodepool,GPU:.metadata.labels.cloud\.google\.com/gke-accelerator,TPU:.metadata.labels.cloud\.google\.com/gke-tpu-accelerator,TPU_TOPOLOGY:.metadata.labels.cloud\.google\.com/gke-tpu-topology,TAINTS:.spec.taints' | head -n 25

# For large clusters, target accelerator pools directly with label selectors:
# kubectl get nodes -l 'cloud.google.com/gke-accelerator' -o custom-columns=... | head -n 25
# kubectl get nodes -l 'cloud.google.com/gke-tpu-accelerator' -o custom-columns=... | head -n 25

# 3. Inspect requested accelerator resource types in the workload's PodSets
kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{range .spec.podSets[*]}{"PodSet: "}{.name}{" | Count: "}{.count}{" | Requests: "}{.template.spec.containers[*].resources.requests}{"\n"}{end}'

Verify that the workload's Pod template (or individual spec.podSets[*]):

• Heterogeneous / Gang Scheduling: For multi-podSet workloads (e.g., RayCluster with CPU head and GPU workers, or LeaderWorkerSet), check the pod set <name> prefix in MSG to identify which specific podSet failed flavor assignment.

1. NVIDIA GPU Jobs (nvidia.com/gpu): Has toleration for GPU node taints using operator: Exists (e.g., key: nvidia.com/gpu, operator: Exists, effect: NoSchedule) and nodeSelector matching cloud.google.com/gke-accelerator (e.g., nvidia-h100-80gb, nvidia-h100-mega-80gb, nvidia-l4). If running on GKE Dynamic Workload Scheduler (DWS Flex) node pools, also verify toleration for cloud.google.com/gke-queued=true:NoSchedule.
2. Google TPU Jobs (google.com/tpu): Has toleration for TPU node taints using operator: Exists (e.g., key: google.com/tpu, operator: Exists, effect: NoSchedule). Verify nodeSelector matches cloud.google.com/gke-tpu-accelerator (e.g. tpu-v5p-slice, tpu-v6e-slice) and topology matches cloud.google.com/gke-tpu-topology (e.g. 2x2x1 for 3D torus, 2x4 for 2D mesh). If running on GKE DWS Flex node pools, also verify toleration for cloud.google.com/gke-queued=true:NoSchedule.
3. Has nodeSelector / nodeAffinity compatible with the flavor's nodeLabels.

Step 5: Audit Workload Priority, Flavor Fungibility & Preemption

If a high-priority workload is blocked behind lower-priority workloads, inspect priority classes, fungibility, and preemption policies:

# 1. Check Workload Priority and WorkloadPriorityClass
kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{"PriorityClass: "}{.spec.priorityClassName}{" | Priority: "}{.spec.priority}{"\n"}'

# 2. Check ClusterQueue Preemption and Flavor Fungibility
kubectl get clusterqueue <CLUSTER_QUEUE_NAME> -o jsonpath='{"Preemption: "}{.spec.preemption}{"\nFlavorFungibility: "}{.spec.flavorFungibility}{"\n"}'

Preemption Check:

• If withinClusterQueue is Never, high-priority workloads cannot preempt lower-priority workloads in the same queue.
• If reclaimWithinCohort is Never, workloads cannot reclaim borrowed quota from peer ClusterQueues.
• If FlavorFungibility.whenCanPreempt is TryNextFlavor, Kueue attempts to find available quota in fallback flavors before preempting running workloads.

Step 6: Topology-Aware Scheduling (TAS) & AdmissionCheck Deep-Dive

A. Diagnosing Topology-Aware Scheduling (TAS) Failures

When multi-node distributed training jobs specify compact placement:

# 1. Check if the workload requested a specific topology (TopologyRequest or PodSet annotations)
kubectl get workload <WORKLOAD_NAME> -n <NAMESPACE> -o jsonpath='{range .spec.podSets[*]}{"PodSet: "}{.name}{" | TopologyRequest: "}{.topologyRequest}{" | Annotations: "}{.template.metadata.annotations}{"\n"}{end}'

# 2. Inspect Topology hierarchy and levels
kubectl get topology -o custom-columns='NAME:.metadata.name,LEVELS:.spec.levels'

If nodes are fragmented across racks/blocks, TAS cannot satisfy contiguous block requirements, leaving the job Inadmissible.

B. Diagnosing GKE Dynamic Workload Scheduler (DWS Flex) Provisioning

When using DWS Flex-start (ProvisioningRequest):

kubectl get provisioningrequests.autoscaling.x-k8s.io -n <NAMESPACE> -o custom-columns='NAME:.metadata.name,PROVISIONED:.status.conditions[?(@.type=="Provisioned")].status,FAILED:.status.conditions[?(@.type=="Failed")].status,BOOKED:.status.conditions[?(@.type=="BookingExpired")].status'

If FAILED is True, DWS rejected the reservation due to regional stockout. The workload will enter AdmissionCheck: Retry or Rejected.

2. Safety Guidelines & Blast Radius Protocol

• Safe Commands (Zero Turn Delay): Read-only commands (kubectl get, kubectl describe, kubectl logs) execute freely during diagnosis.
• Mutating Commands (Confirmation Strictly Required): Deleting workloads, evicting pods, restarting controllers, altering ClusterQueue quota, or changing stop policies must NEVER execute without presenting a structured remediation plan and obtaining user confirmation.

Required Remediation Plan Format:

[PROPOSED REMEDIATION PLAN]
- Target Resource: <ClusterQueue|LocalQueue|Workload>/<RESOURCE_NAME>
- Root Cause Identified: <Precise diagnostic finding>
- Proposed Action: <Exact command to be executed>
- Blast Radius: <Affected workloads, namespaces, and cohort tenants>
- Confirmation Required: Reply 'yes' to proceed.

Canonical Non-Destructive Patch Recipes:

• Unpausing a LocalQueue (stopPolicy: None):

  kubectl patch localqueue <NAME> -n <NAMESPACE> --type=merge -p '{"spec":{"stopPolicy":"None"}}'

• Switching StrictFIFO to BestEffortFIFO (resolving Head-of-Line deadlock):

  kubectl patch clusterqueue <NAME> --type=merge -p '{"spec":{"queueingStrategy":"BestEffortFIFO"}}'

• Reactivating an Inactive Workload (spec.active: true):

  kubectl patch workload <NAME> -n <NAMESPACE> --type=merge -p '{"spec":{"active":true}}'

3. References & Canonical Documentation

• Kueue Concepts & Architecture: references/kueue-docs.md | kueue.sigs.k8s.io/docs/concepts
• GKE Cohorts & DWS Hardware Profiles: references/gke-kueue-docs.md | docs.cloud.google.com/kubernetes-engine/docs/tutorials/kueue-cohort