NVIDIAKnown publisherNVIDIA/deepops/skills/deploy-runai-kai/SKILL.md

deploy-runai-kai

Skill1.5k starsChanged 10 days agoBSD-3-ClauseView on GitHub

Use when adding or removing the optional KAI Scheduler or Run:ai platform on a DeepOps Kubernetes GPU cluster, enabling fractional-GPU scheduling, or diagnosing KAI/Run:ai install or scheduling failures.

Oct 8, 2020Add support to display different GPU types, specifically NVIDIA Multi-Instance GPUs — kubeflow/kubeflow6 reactions · 13 comments
Jun 28, 2020Really need a Kubeflow Ecosystem Section to the Website, (Note list is attached I found who are in the Ecosystem) — kubeflow/website4 reactions · 11 comments
Jun 6, 2026[Add] NVIDIA DeepOps (NVIDIA/deepops) — PyTorchKR/landscape0 reactions
Jul 19, 2024Allow dynamic loading of slurm_conf on Anvil. — glotzerlab/hoomd-blue0 reactions
Mar 24, 2024Error with provider libvirt — hashicorp/vagrant0 reactions · 1 comment

19 more on GitHub

Aug 13, 2023Set HPA(Horizontal Pod Autoscaler) of Triton Inference Server with Prometheus metric — MrSyee/SAM-remove-background0 reactions
Apr 4, 2023Configuration issues with containerd failing Kubespray install with "unknown service runtime.v1alpha2.ImageService" error — kubernetes-sigs/kubespray0 reactions · 6 comments
Nov 10, 2022Prometheus metric gpu_capacity is not found. — NTHU-LSALAB/KubeShare0 reactions · 4 comments
Feb 26, 2022SLURM gpus-per-task issue — NVIDIA/pyxis0 reactions · 5 comments
Sep 5, 2021feature(GPU): kubespray install NVIDIA GPU — k8sli/kubeplay0 reactions · 1 comment
Aug 12, 2021Ensure systemd unit reloaded when restarting — NVIDIA/ansible-role-nvidia-docker0 reactions
May 18, 2021Super huge log messages from journald - misconfigured filter_concat? — kokuwaio/helm-charts0 reactions · 2 comments
Mar 22, 2021Add var to allow skipping reload of Docker daemon — NVIDIA/ansible-role-nvidia-docker0 reactions
Dec 18, 2020Multiple Pods stucks in CrashLoopBackOff — kubeflow/kubeflow0 reactions · 9 comments
Oct 19, 2020Update PPA URL — DeepOps/ansible-maas0 reactions
Oct 7, 2020enroot-nsenter: failed to create user namespace: Invalid argument — NVIDIA/enroot0 reactions · 2 comments
Aug 20, 2020Istio Dex Manifest v1.0.2 is "Unable to fetch Nodes", centraldashboard throwing 504 — kubeflow/community-distribution0 reactions · 2 comments
Jul 16, 2020Multi-GPU workloads hang in Jupyter Notebooks launched through Kubeflow — kubeflow/kubeflow0 reactions · 3 comments
Jun 16, 2020update_ood_portal may not work on Ubuntu — OSC/ood-ansible0 reactions · 1 comment
May 18, 2020details = "failed to connect to all addresses" in "helm" module — ansible/ansible0 reactions · 8 comments
Jan 22, 2020Failed to fetch https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/index.yaml : 403 Forbidden — prometheus-operator/kube-prometheus0 reactions · 4 comments
Sep 14, 2019ambassador always CrashLoopBackOff — kubeflow/kubeflow0 reactions · 8 comments
Aug 9, 2019"Copy kubectl binary to ansible host" sometimes fails with memory error due to known issue with 'fetch' module. — kubernetes-sigs/kubespray0 reactions · 8 comments
Nov 13, 2018Tiller init command failing on primary master — kubernetes-sigs/kubespray0 reactions

In this repository11 other files it gives its agents.
- AGENTS.mdAGENTS.md
- deploy-airgappedSkill
- deploy-k8s-gpu-clusterSkill
- deploy-slurm-clusterSkill
- diagnose-driver-installSkill
- finetune-modelSkill
- manage-model-cacheSkill
- model-workflowsSkill
- provision-with-maasSkill
- serve-modelSkill
- validate-gpu-clusterSkill

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 2 are distinctive, and 3 are very specific to this project.

≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.

Common
Distinctive
Very specific

Very specific:
Run commands from the DeepOps repository root.
line 8
Distinctive:
Verify with the validator, not with the playbook result:
line 33
Very specific:
Remove: add -e runai_state=absent, then python3 scripts/validation/validate_runai.py --json --verify-uninstalled must report "ok": true.
line 44
Distinctive:
Put the admin login value in a command line, an inventory file in Git, or a log.
line 65
Very specific:
Treat a finished playbook as success.
line 66

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.

Files

SKILL.md

Deploy KAI Scheduler or Run:ai (optional)

Run commands from the DeepOps repository root. This adds GPU-aware scheduling on top of a working Kubernetes GPU cluster. It is off by default and never changes the GPU Operator or the NVIDIA driver. Full guide: docs/k8s-cluster/runai.md.

Preconditions

python3 scripts/validation/validate_k8s.py --json --cuda-smoke reports "ok": true (the cluster already runs GPU pods). If not, use skills/deploy-k8s-gpu-cluster/ first.
Confirm the inventory is the intended cluster and nobody else's workloads run there; the install adds cluster-wide admission webhooks.
Pick one mode. kai and platform exclude each other (the platform ships its own KAI). platform is experimental and also needs a default StorageClass, kube-prometheus-stack 88.6.5+, a DNS name and TLS (see the guide).
Fractioning with hard memory limits (runai_kai_gpu_fractioning: true) needs GPU Operator v26.7.1+ and an r615+ driver already in place. Do not change the default driver to get it without the cluster owner's agreement.

Procedure

Set runai_mode (and, for platform, the domain, admin user and a vaulted login value) in config/group_vars/k8s_cluster.yml.
Preview: ansible-playbook -l kube_control_plane playbooks/k8s-cluster/runai.yml --check --diff
Apply: ansible-playbook -l kube_control_plane playbooks/k8s-cluster/runai.yml. Rerun once; the second run must report changed=0.
Verify with the validator, not with the playbook result:
```
python3 scripts/validation/validate_runai.py --json                      # kai
python3 scripts/validation/validate_runai.py --json --memory-enforcement # + fractioning
```
Require exit 0 and "ok": true with gpu_job_ok and fraction_job_ok true. Platform mode: add --mode platform --platform-domain <fqdn>, set RUNAI_USERNAME/RUNAI_PASSWORD in the environment (never as arguments), and use a Run:ai project's namespace and queue.
Remove: add -e runai_state=absent, then python3 scripts/validation/validate_runai.py --json --verify-uninstalled must report "ok": true.

Failure branches

Signal	Do
Role stops with "No node advertises nvidia.com/gpu"	GPU stack not ready; wait, or run the GPU Operator playbook. Do not bypass with `runai_require_gpu: false` on a real install.
Role stops with "ships its own KAI"	Remove the other mode first (`runai_state: absent`).
Role stops on GPU Operator version or `r615`	Missing fractioning prerequisites. Set `runai_kai_gpu_fractioning: false`, or ask the cluster owner to approve the GPU stack change.
Validator `scheduler deployments missing/not available`	KAI operator still starting; wait and rerun. `kubectl -n kai-scheduler get pods`.
`gpu_job_ok` false, pod `Pending`	Wrong queue or none: `kubectl get queues.scheduling.run.ai`; pod must set `schedulerName: kai-scheduler` and the queue label.
`fraction_job_ok` false: pods on different GPUs	The node has more than one GPU and KAI chose another; with one GPU this means sharing is off (`global.gpuSharingMode`).
`memory_enforcement_ok` false	Limits are not enforced: check the GPU Operator CDI/NRI settings and `kubectl get gpufractioningconfig default -o yaml`. Do not claim enforcement.
Platform: control plane unreachable	The domain must resolve to the ingress address; pass `--ca-file`/`--resolve-ip`.
Uninstall check lists leftovers	Remove the named namespaces/releases; add `-e runai_uninstall_remove_crds=true` only if queues/projects may be deleted.

Do not

Install standalone KAI beside the Run:ai cluster chart.
Put the admin login value in a command line, an inventory file in Git, or a log.
Treat a finished playbook as success; only the validator's "ok": true counts.
Claim a live result you did not run. Offline tests prove the logic, not a GPU job.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

It worked
Partly
It didn't

What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free