hyperpod-slurm-debugger

---
name: hyperpod-slurm-debugger
description: Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide. Invoke when the user reports a Slurm node stuck in down/drain, "Node unexpectedly rebooted" after auto-repair, slurmd not running, jobs stuck PENDING with REASON=Resources while sinfo shows idle nodes, jobs stuck COMPLETING after node replacement, GRES/GPU counts wrong, scontrol ping failing, slurmctld unresponsive, an Action:Reboot/Replace request that did not trigger HyperPod auto-recovery, or auto-resume not restarting a job. Also triggers on "drain before reboot", "diagnose a Slurm node", "investigate stuck jobs."
metadata:
  version: "0.0.1"
---

# HyperPod Slurm Debugger

Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on
HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command.
For remediation, link to the official AWS or Slurm documentation.

## When to invoke

Invoke when the user reports any of the symptoms in the [decision table](#decision-table).

## When NOT to invoke

- Cluster has `Orchestrator.Eks` — invoke `hyperpod-node-debugger` or `hyperpod-nccl`.
- Single-node hardware fault with healthy Slurm scheduler — invoke `hyperpod-node-debugger`.
- NCCL training-hang investigation — invoke `hyperpod-nccl`.
- Node unreachable via SSM — invoke `hyperpod-ssm`.

## Constraints

- Read-only. Do not run, recommend, or print state-mutating commands.
- For any remediation, link to AWS or Slurm docs. The user authorizes and executes.
- IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation
  drifts the live state from the IaC plan.

Canonical recovery URLs:
[references/slurm-details.md → Authoritative recovery documentation](references/slurm-details.md).

## Prerequisites

- AWS CLI v2, authenticated for the target account and region with permissions:
  - `sagemaker:DescribeCluster`, `sagemaker:ListClusterNodes`
  - `ssm:StartSession` on the HyperPod-created SSM document
- [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)
  installed locally.
- `jq` ≥ 1.6.
- `unbuffer` (from the `expect` package). Required — without it `aws ssm start-session`
  returns empty stdout intermittently with `Cannot perform start session: EOF` and every
  check silently misreports. Install: `expect` package on Amazon Linux / RHEL / Debian /
  Ubuntu / macOS. Script exits at prerequisite check if missing.

## Procedure

### Step 1 — Collect inputs

Ask the user for:

1. HyperPod cluster name (not Slurm partition name).
2. AWS region.
3. Optional: a specific Slurm node name.

### Step 2 — Confirm orchestrator

```bash
aws sagemaker describe-cluster --cluster-name <NAME/ARN> --region <REGION> \
  --query 'Orchestrator' --output json
```

If `Orchestrator.Eks` is present, stop. Route per [When NOT to invoke](#when-not-to-invoke).

### Step 3 — Run the diagnostic script

```bash
bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION>
# Scope to a node:
bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION> --node <SLURM_NODE>
```

Relay the script output to the user verbatim.

### Step 4 — Map findings → docs

For each finding, look up the section in the [decision table](#decision-table) and link
the user to the corresponding AWS / Slurm doc. Do not type out remediation commands.

## Decision table

| Symptom (`sinfo -o "%N %T %30E"` or script finding)         | Section                                                |
| ----------------------------------------------------------- | ------------------------------------------------------ |
| Node state = `down` or `down*`, reason other than below     | [A: Node Down](#a-node-down)                           |
| Node state = `down*`, Reason = `Node unexpectedly rebooted` | [B: Unexpected Reboot](#b-unexpected-reboot)           |
| Jobs `PENDING` with `REASON=Resources` while nodes are idle | [C: Controller State](#c-controller-state)             |
| Jobs stuck `COMPLETING` after node replacement              | [C: Controller State](#c-controller-state)             |
| `scontrol ping` returns `DOWN` for the controller           | [C: Controller State](#c-controller-state)             |
| GRES (GPU) counts incorrect or not released                 | [C: Controller State](#c-controller-state)             |
| `state=fail` issued but no recovery occurred                | [D: Action Reason Mismatch](#d-action-reason-mismatch) |
| Accounting errors or RPC errors mentioning `dbd`            | [C: Controller State](#c-controller-state) (slurmdbd)  |
| `slurm.conf` edited; new partitions or nodes not visible    | [C: Controller State](#c-controller-state) (config)    |
| Job exited on a hardware failure but did not restart        | [E: Auto-resume](#e-auto-resume)                       |

## Defaults

| Behavior             | Default                                                                                            | Override                   |
| -------------------- | -------------------------------------------------------------------------------------------------- | -------------------------- |
| Mode                 | read-only — always; no remediation flag exists                                                     | n/a                        |
| Region               | `$AWS_DEFAULT_REGION`, falling back to `us-east-1`                                                 | `--region <R>`             |
| Scope                | all nodes in `down` / `drain` / `fail` / "unexpectedly rebooted"                                   | `--node <SLURM_NODE_NAME>` |
| Output               | colorized terminal                                                                                 | `--no-color`               |
| SSM target format    | `sagemaker-cluster:<clusterId>_<instanceGroupName>-<instanceId>` (derived)                         | n/a                        |
| Controller discovery | `--controller-group` (if set) → `SlurmConfig.NodeType=Controller` → `provisioning_parameters.json` | `--controller-group <N>`   |

## Error handling

| Failure                                            | Skill behavior                         | Required user action                            |
| -------------------------------------------------- | -------------------------------------- | ----------------------------------------------- |
| `describe-cluster` fails                           | Print AWS error; exit 1                | Fix credentials/region; verify cluster name     |
| Cluster has `Orchestrator.Eks`                     | Exit 1 with pointer to EKS-side skills | Use `hyperpod-node-debugger` or `hyperpod-nccl` |
| `session-manager-plugin` missing / SSM unreachable | `sinfo` returns empty; exit 1          | Install plugin; verify node `InService`         |
| Disk ≥ 95 % full on a `down` node                  | Report finding `disk-full-<node>`      | Refer to AWS troubleshooting docs               |
| Missing `jq` or `aws`                              | Exit 1 at prerequisite check           | Install per [Prerequisites](#prerequisites)     |

---

## A: Node Down

Node is `down` because `slurmd` stopped responding. Causes: `slurmd` crash, disk full,
OOM, network partition, hardware fault.

Script checks: `systemctl is-active slurmd`, `srun -w <NODE> hostname` (RPC layer), disk,
memory.

Link: <https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md>

If node returns to `down` after a manual resume → escalate to `hyperpod-node-debugger`.

Context: [references/slurm-details.md § A](references/slurm-details.md#-a-node-down--diagnostic-context).

---

## B: Unexpected Reboot

Node is `down*` with Reason `"Node unexpectedly rebooted"` because `slurmd`
re-registered after an out-of-band reboot. Upstream Slurm behavior, not HyperPod.
Node is typically healthy.

Links:

- <https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md>
- <https://slurm.schedmd.com/scontrol.html> (`state=resume` semantics)

If node reboots again within minutes → escalate to `hyperpod-node-debugger`.

Context: [references/slurm-details.md § B](references/slurm-details.md#-b-unexpected-reboot--diagnostic-context).

---

## C: Controller State

`slurmctld` in-memory state can desync from the on-disk state. A controller restart reloads from `StateSaveLocation` and clears bad caches. User decides and executes.

Restart may help:

| Symptom                                            | Why                                         |
| -------------------------------------------------- | ------------------------------------------- |
| `PENDING` with `REASON=Resources`, idle nodes      | Re-evaluates the queue                      |
| Jobs stuck `COMPLETING` after node replacement     | Controller held a reference to the old node |
| GRES (GPU, EFA) not released after a job ends      | Resource accounting de-synced               |
| Nodes stuck `Unknown` after reboot, `slurmd` is up | Re-registration was not processed           |
| `scontrol ping` times out                          | Controller event loop is hung               |
| Lost connection to `slurmdbd` / RPC errors         | DBD connection wedged                       |

Do NOT restart when:

- HyperPod replacement (`Action:Replace`) in progress on any node — concurrent changes
  fail the replacement.
- Only one compute node is bad — restart `slurmd` on that node.
- `sinfo` and `squeue` are responsive — problem is elsewhere.
- `journalctl -u slurmctld` not reviewed yet — panic / OOM will reproduce.
- `slurm.conf` was just edited — try `scontrol reconfigure` first.

### Folded triggers

- **slurmdbd disconnected** — `sacct` fails, accounting fields show `Unknown`,
  controller log spams `Unable to contact slurmdbd`. Restore `slurmdbd` before
  considering controller restart.
  <https://slurm.schedmd.com/accounting.html> ·
  [details](references/slurm-details.md#slurmdbd-connectivity).
- **Stale config** — `slurm.conf` / `topology.conf` mtime > slurmctld start.
  `scontrol reconfigure` first; restart is fallback.
  <https://slurm.schedmd.com/scontrol.html> ·
  [details](references/slurm-details.md#scontrol-reconfigure-vs-restart).

Restart procedure / what's preserved:

- <https://slurm.schedmd.com/slurmctld.html>
- <https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md>

Context: [references/slurm-details.md § C](references/slurm-details.md#-c-controller-state--diagnostic-context).

---

## D: Action Reason Mismatch

`scontrol update state=fail reason=...` was issued with a `reason` that does not match
`Action:Reboot` or `Action:Replace` exactly. HyperPod silently ignores anything else.
Script detects near-misses on nodes in `fail` state.

Required strings (case-sensitive, no whitespace, no punctuation):

- `Action:Reboot`
- `Action:Replace`

Link: <https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-replace-faulty-instance.html>

Context: [references/slurm-details.md § Action reason-string validation](references/slurm-details.md#action-reason-string-validation).

---

## E: Auto-resume

`--auto-resume=1` is an `srun` step option. It re-runs the step after HMA (the Health
Monitoring Agent) flags a node and Automatic node recovery replaces it.

Why it didn't restart the job:

- Flag on `sbatch` not `srun` — per-step; `sbatch` directives are silently ignored.
- HMA did not flag the node — failure was application/transient, not hardware. Step
  exits as a normal Slurm failure.
- Cluster `NodeRecovery` is `None` — faulty nodes are labeled but not replaced.
- No checkpointing — step restarts from process zero each iteration.
- AMI predates HMA support (released 2025-09-11) — needs AMI / cluster-software update.

Link: <https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-auto-resume.html>

Context: [references/slurm-details.md § HyperPod auto-resume](references/slurm-details.md#hyperpod-auto-resume).

---

## Escalation

| Condition                                                       | Next skill                            |
| --------------------------------------------------------------- | ------------------------------------- |
| Node returns to `down` shortly after a manual resume            | `hyperpod-node-debugger` (hardware)   |
| `slurmd` logs contain CUDA / NVIDIA / XID errors                | `hyperpod-node-debugger` § G          |
| Disk full or `/dev/shm` exhausted                               | `hyperpod-node-debugger` § I          |
| Node unreachable via SSM                                        | `hyperpod-ssm`                        |
| Controller restart does not clear `COMPLETING` after 2 attempts | `hyperpod-issue-report` + AWS Support |

More agent context in awslabs/agent-plugins

36 other files this repository gives its agents.