hyperpod-cluster-debugger

---
name: hyperpod-cluster-debugger
description: Diagnose and remediate cluster-wide HyperPod (EKS or Slurm) problems — creation / deployment failures (CloudFormation, EFA health check, lifecycle scripts, capacity), EKS access, node replacement, CloudFormation nested-stack errors, post-maintenance rollback state, dangling nodes, autoscaler conflicts. Includes `--validate` pre-flight. Read-only.
metadata:
  version: "0.0.1"
---

# HyperPod Cluster Debugger

**Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer to run it. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes).

**Before any state-changing CLI: ask if it's IaC-managed.** HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).

`scripts/diagnose-cluster.sh` is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as `[FAIL] ... → references/<file>.md § <section>`.

| Reference                                                                 | Open when                                                           |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| [cluster-diagnostics-detail.md](references/cluster-diagnostics-detail.md) | Per-finding remediation runbook (§ A–L)                             |
| [cluster-operations.md](references/cluster-operations.md)                 | Operational deep-dives (EFA SG, EKS access, SSM, Slurm, filesystem) |
| [cloudformation-errors.md](references/cloudformation-errors.md)           | § H needs the full per-resource CFN error catalog                   |
| [capacity-planning.md](references/capacity-planning.md)                   | § B or `--validate` flags capacity / subnet sizing                  |
| [lifecycle-scripts.md](references/lifecycle-scripts.md)                   | § C points at a specific lifecycle failure                          |
| [iam-permissions.md](references/iam-permissions.md)                       | Full IAM policy for the diagnostic                                  |

---

## Workflow

1. Collect HyperPod cluster name (not EKS name), region, exact error string.
2. Run `scripts/diagnose-cluster.sh` (or `--validate` for pre-create).
3. For every `[FAIL]` line, `Read` the referenced section.
4. Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval.
5. Re-run the diagnostic to confirm.

---

## Step 1: Run diagnostics

```bash
# Diagnose an existing cluster:
bash scripts/diagnose-cluster.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>

# Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints,
# optionally S3 lifecycle scripts and per-AZ capacity:
bash scripts/diagnose-cluster.sh --validate --region <REGION> \
  --sg-ids <sg-1,sg-2> --subnet-ids <sub-1,sub-2> [--iam-role <role-arn>] \
  [--s3-uri s3://<BUCKET>/path/] [--instance-type ml.p5.48xlarge]
```

Pass `--instance-type` when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time).

Tags: `[PASS]` · `[FAIL]` (counted, has `→ references/...` pointer) · `[WARN]` · `[INFO]`. Priorities: **P0** blocks operation · **P1** degraded · **P2** informational.

---

## Step 2: Match signal → section

**Error messages / events:**

| Signal                                                                       | Section                                                        |
| ---------------------------------------------------------------------------- | -------------------------------------------------------------- |
| `"EFA health checks did not run successfully"` (public-doc verbatim signal)  | **[A: EFA Health Checks](#a-efa-health-checks)**               |
| Insufficient-capacity or AZ-mismatch failure at creation                     | **[B: Capacity & AZ](#b-capacity--az)**                        |
| Lifecycle-script failure or timeout during provisioning                      | **[C: Lifecycle Scripts](#c-lifecycle-scripts)**               |
| kubectl auth error (server asks for credentials / no API group list)         | **[D: EKS Access](#d-eks-access--kubectl)**                    |
| `InService` but not all instances visible                                    | **[E: Cluster Provisioning](#e-cluster-provisioning)**         |
| `"Target is not connected"` / SSM errors                                     | **[F: SSM Connectivity](#f-ssm-connectivity)**                 |
| Node replacement not happening / `batch-replace` not working                 | **[G: Node Replacement](#g-node-replacement)**                 |
| `"Embedded stack failed"` / any CloudFormation error                         | **[H: CloudFormation Errors](#h-cloudformation-errors)**       |
| `UpdateClusterSoftware` failed or cluster in post-maintenance rollback state | **[J: AMI & Cluster Updates](#j-ami--cluster-updates)**        |
| Dangling / orphaned nodes in EKS vs `list-cluster-nodes`                     | **[K: Dangling Nodes & Cleanup](#k-dangling-nodes--cleanup)**  |
| Cluster Autoscaler breaks after HyperPod attached                            | **[L: Autoscaler Compatibility](#l-autoscaler-compatibility)** |
| Slow I/O, FSx throughput saturated                                           | [cluster-operations.md § 9](references/cluster-operations.md)  |
| Slurm node name → instance ID lookup                                         | **[I: Utilities](#i-utilities)**                               |

---

## A: EFA Health Checks

SG missing self-reference. Add inbound + outbound self-ref to every SG on the cluster, plus least-privilege egress for the AWS APIs the node needs (HTTPS 443 to S3 / ECR / SageMaker / SSM / STS / CloudWatch Logs — via VPC-endpoint prefix-lists when possible). Full procedure: [cluster-diagnostics-detail.md § A](references/cluster-diagnostics-detail.md#a-efa-health-checks).

## B: Capacity & AZ

Instance type unavailable in the requested AZ. Verify with `describe-instance-type-offerings`, then change AZ, use Flexible Training Plans, or request ODCR. Full: [§ B](references/cluster-diagnostics-detail.md#b-capacity--az) · strategy: [capacity-planning.md](references/capacity-planning.md).

## C: Lifecycle Scripts

Script failed or timed out during provisioning. Read CloudWatch under `/aws/sagemaker/Clusters/<name>/<id>` — common causes: missing S3 VPC endpoint, IAM gap, CRLF line endings, instance-group name mismatch. Full: [§ C](references/cluster-diagnostics-detail.md#c-lifecycle-scripts) · layout: [lifecycle-scripts.md](references/lifecycle-scripts.md).

## D: EKS Access / kubectl

IAM identity not in EKS access entries. Verify with `sts get-caller-identity`, create an access entry with admin policy, update kubeconfig. Full: [§ D](references/cluster-diagnostics-detail.md#d-eks-access--kubectl).

## E: Cluster Provisioning

`InService` without all instances is expected under Continuous Provisioning — failures surface as events, not cluster errors. For stuck `Creating`/`Updating`/`Deleting`: check CFN nested stacks (§ H), IAM, capacity, events; if stuck `Deleting` check VPC ENI dependencies. Full: [§ E](references/cluster-diagnostics-detail.md#e-cluster-provisioning).

## F: SSM Connectivity

`Target is not connected`: use `sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID>` format (not raw EC2 ID), install session-manager-plugin, confirm node `Running`. Check IAM + VPC endpoints on timeouts. Full: [§ F](references/cluster-diagnostics-detail.md#f-ssm-connectivity).

## G: Node Replacement

Auto-repair: confirm `NodeRecovery=Automatic`, check Health Monitoring Agent (HMA) logs + node labels / Slurm reason, confirm capacity. Manual: reboot first, replace only if reboot fails. Replace requires the cluster to have been patched via `UpdateClusterSoftware` at least once and cannot target a Slurm controller node. Full: [§ G](references/cluster-diagnostics-detail.md#g-node-replacement).

## H: CloudFormation Errors

`Embedded stack failed` hides the real error. Drill into nested stacks via Events tab (filter Failed) until you reach a non-stack resource. CLI: `describe-stack-events --query 'StackEvents[?ResourceStatus==\`CREATE_FAILED\`]'`. Also covers SLR creation failures and permission-boundary denials. Full: [§ H](references/cluster-diagnostics-detail.md#h-cloudformation-errors) · catalog: [cloudformation-errors.md](references/cloudformation-errors.md).

## I: Utilities

Map Slurm node names (`ip-10-x-y-z`) to HyperPod instance IDs via `list-cluster-nodes` or on-node `/opt/ml/config/resource_config.json`. Full: [§ I](references/cluster-diagnostics-detail.md#i-utilities).

## J: AMI & Cluster Updates

`UpdateClusterSoftware` fails and rolls back, or the cluster stays in a post-maintenance rollback state. Common causes: lifecycle script incompatible with new AMI, HMA version too old, insufficient rolling-update capacity. If the cluster has active nodes, collect diagnostics and escalate rather than delete-and-recreate. Full: [§ J](references/cluster-diagnostics-detail.md#j-ami--cluster-updates).

## K: Dangling Nodes & Cleanup

Nodes in `kubectl get nodes` but not in `list-cluster-nodes` (ghost EKS nodes), or the inverse (HyperPod nodes that never registered kubelet). Script flags both. Full: [§ K](references/cluster-diagnostics-detail.md#k-dangling-nodes--cleanup).

## L: Autoscaler Compatibility

Cluster Autoscaler errors on HyperPod provider IDs and breaks autoscaling for all node groups. No officially endorsed workaround — escalate to AWS Support. Karpenter does not conflict with HyperPod nodes by default. Full: [§ L](references/cluster-diagnostics-detail.md#l-autoscaler-compatibility).

---

## Prerequisites

- `aws` CLI v2.13+ authenticated to the cluster's account
- `jq`, `python3`, `bash` 4.2+
- `kubectl` authenticated to the EKS cluster (EKS checks skipped if absent)
- `session-manager-plugin` (Slurm controller health checks only)

IAM policy: [references/iam-permissions.md](references/iam-permissions.md).

## Defaults

- **Region** — required: pass `--region` or set `$AWS_DEFAULT_REGION`.
- **Mode** — `--cluster <NAME>` (diagnose) or `--validate` (pre-create).
- **Event window** — up to 500 most recent events (5 × 100, paginated).
- **Colors** — auto-disabled on non-TTY; `--no-color` to force off.

## Error handling

| Failure                                             | Script                                                     | Tell the customer                                     |
| --------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------- |
| `aws sts get-caller-identity` fails                 | Exit 1                                                     | "Fix AWS credentials and rerun."                      |
| Cluster not found                                   | Exit 1 after listing region's clusters                     | "Confirm HyperPod cluster name (not EKS) and region." |
| `sagemaker:*` / `ec2:*` / `eks:*` / `logs:*` denied | Warn, add `Missing IAM permission for <API>`, continue     | "Grant the listed IAM action and rerun."              |
| `kubectl` absent or unauthenticated                 | Skip EKS checks (access entries, add-ons, aws-auth, nodes) | "Install/authenticate kubectl."                       |
| `session-manager-plugin` absent (Slurm)             | Skip Slurm controller probe                                | "Install session-manager-plugin."                     |
| SSM throttled / times out (180s)                    | Retry with backoff; warn and continue if still failing     | "Rerun later — script is idempotent."                 |
| CloudWatch log group not found                      | Skip CloudWatch check                                      | "CloudWatch not configured on this cluster."          |

Exit codes: `0` no critical failures · `1` one or more critical failures (cluster not found, fatal prerequisite missing, or any `[FAIL]` in diagnose or `--validate` mode). `[WARN]` lines do not affect the exit code.

## Skill delegation

| Need                            | Use                        |
| ------------------------------- | -------------------------- |
| Shell on nodes                  | `hyperpod-ssm`             |
| Version comparison across nodes | `hyperpod-version-checker` |

## Escalate to AWS Support

Escalate when:

1. EFA health checks fail despite correct SG rules.
2. Capacity errors persist despite a valid Flexible Training Plan / ODCR.
3. Node replacement fails repeatedly without clear events / log signal.
4. Cluster stuck in a non-terminal state (`Creating`, `Updating`, or a post-maintenance rollback state) for an extended period.
5. CloudFormation root-cause is an internal service error.

### Before opening the case

Run these commands and attach the output. Goal: AWS Support has everything at case open.

```bash
# 1. Cluster identity + status (confirms region, ARN, orchestrator, instance groups)
aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION>

# 2. Full cluster-level diagnostic bundle
bash scripts/diagnose-cluster.sh --cluster <CLUSTER> --region <REGION> > diag.txt

# 3. Per-node log/config bundle to S3 (delegates to hyperpod-issue-report skill)
#    See skills/hyperpod-issue-report/SKILL.md for the exact invocation.
```

### Include in the case

- Cluster name + ARN (or `ClusterId` suffix) and AWS region
- `ClusterStatus` + `FailureMessage` from `describe-cluster`
- Timestamp window (UTC start / end) of the failure
- Exact error strings observed (copy verbatim from events / logs / console)
- Affected instance IDs / `NodeLogicalId`s / instance group names
- `diag.txt` from step 2 above
- S3 URI of the `hyperpod-issue-report` bundle from step 3

More agent context in awslabs/agent-plugins

36 other files this repository gives its agents.