rook-ceph-operator
Skill4 starsChanged 5 months agoMITView on GitHub
Use when deploying or managing Rook-Ceph CRDs — CephCluster, CephBlockPool, CephFilesystem, CephObjectStore, CephNFS, CephRBDMirror, and supporting resources. Covers operator chart config, cluster spec, CSI settings, storage classes.
- Reads credentials
In this repository74 other files it gives its agents.
- AGENTS.mdAGENTS.md
- atlasSkill
- cert-managerSkill
- cilium-gatewaySkill
- cilium-networkSkill
- cnpgSkill
- dragonflySkill
- external-dnsSkill
- external-secretsSkill
- flaggerSkill
- fluxSkill
- gitea-apiSkill
- gitea-registrySkill
- gitea-runnerSkill
- giteaSkill
- gitea-teaSkill
- gitea-webhooksSkill
- harbor-apiSkill
- harbor-helmSkill
- harborSkill
- harbor-terraformSkill
- helm-chartSkill
- helm-opsSkill
- helmSkill
- higress-helmSkill
- higress-operatorSkill
- higressSkill
- kserve-helmSkill
- kserve-operatorSkill
- kserveSkill
- kubeflow-pipelinesSkill
- kubeflowSkill
- kubeflow-trainerSkill
- kubeflow-training-operatorSkill
- mariadb-helmSkill
- mariadb-operatorSkill
- mariadbSkill
- netbird-cliSkill
- netbird-operatorSkill
- netbird-serverSkill
- netbirdSkill
- netbird-terraformSkill
- nextcloud-apiSkill
- nextcloud-helmSkill
- nextcloudSkill
- nvidia-device-pluginSkill
- rook-cephSkill
- rook-ceph-toolboxSkill
- sealed-secretsSkill
- stakater-reloaderSkill
- tailscaleSkill
- tailscale-talosSkill
- talosconfigSkill
- talosctlSkill
- talosSkill
- talos-terraformSkill
- tekton-operatorSkill
- tekton-pacSkill
- tekton-pipelinesSkill
- tektonSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 166
If upgrade fails with apply conflicts, use helm upgrade --force-conflicts
- Very specific: line 167
Verify: kubectl get deploy -n rook-ceph -l app.kubernetes.io/name=rbd shows ctrlplugin + nodeplugin running
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Rook-Ceph Operator CRDs
Charts: rook-ceph + rook-ceph-cluster (from https://charts.rook.io/release) + ceph-csi-drivers (v1.20+, from https://charts.ceph.io/ceph-csi-operator)
Latest: v1.20.x (charts) | Ceph: v20.2.2 (Tentacle) | API: ceph.rook.io/v1
Install/upgrade order: rook-ceph → ceph-csi-drivers → rook-ceph-cluster
⚠️ v1.20 breaking change: CSI drivers are now managed by the separate
ceph-csi-driversHelm chart. Rook no longer creates CSI ServiceAccounts, RBAC, or Driver CRs. Without this chart, all PVC provisioning breaks. See Upgrade Notes below.
CRD Reference
| CRD | API | Purpose | Key Fields |
| CephCluster | v1 | Core cluster CRD — defines monitors, OSDs, manager, network, storage topology. | spec.mon.count, spec.mgr.count, spec.storage (nodes[], useAllNodes, deviceFilter, config.deviceClass), spec.cephVersion.image, spec.dataDirHostPath, spec.dashboard, spec.network, spec.placement |
| CephBlockPool | v1 | RBD block storage pool — creates Ceph pool + optional StorageClass. | spec.failureDomain, spec.replicated.size, spec.deviceClass, spec.erasureCoded (dataChunks, codingChunks), spec.application |
| CephFilesystem | v1 | CephFS shared filesystem — metadata + data pools, MDS daemons. | spec.metadataPool, spec.dataPools[], spec.metadataServer.activeCount, spec.metadataServer.activeStandby |
| CephObjectStore | v1 | S3-compatible object store — RGW gateway, metadata + data pools. | spec.metadataPool, spec.dataPool, spec.gateway.port, spec.gateway.instances, spec.gateway.placement |
| CephObjectStoreUser | v1 | S3 access key/secret for object store users. | spec.store, spec.displayName |
| CephNFS | v1 | NFS export of CephFS filesystem. | spec.rados.namespace, spec.server.active |
| CephRBDMirror | v1 | RBD mirroring daemon for cross-cluster replication. | spec.count |
| CephFilesystemMirror | v1 | CephFS snapshot mirroring across clusters. | spec.placement (placement only) |
| CephFilesystemSubVolumeGroup | v1 | Subvolume group for CephFS quota/UID/GID isolation. | spec.filesystemName, spec.pinning, spec.quota |
| CephBucketTopic | v1 | S3 bucket notification topic (Kafka, AMQP, webhook). | spec.endpoint, spec.persistent, spec.opaqueData |
| CephBucketNotification | v1 | Binds topic to object store events. | spec.topic, spec.events[], spec.filter (key/metadata/tagFilters) |
| CephClient | v1 | Ceph auth key management for external clients. | spec.caps |
| CephBlockPoolRadosNamespace | v1 | Rados namespace within a block pool for tenant isolation. | spec.blockPoolName, spec.mirroring |
| CephCOSIDriver | v1 | COSI (Container Object Storage Interface) driver. | Standard COSI config |
Deployed Pattern
# CephCluster — via rook-ceph-cluster chart values.cephClusterSpec
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
mon:
count: 3
allowMultiplePerNode: false
mgr:
count: 1
dashboard:
enabled: true
ssl: false
storage:
useAllNodes: false
useAllDevices: false
nodes:
- name: worker-01
deviceFilter: ^vdb$
- name: worker-02
deviceFilter: ^vdb$
config:
deviceClass: nvme
osdsPerDevice: "1"
- name: worker-03
deviceFilter: ^vdb$
config:
deviceClass: nvme
osdsPerDevice: "1"
- name: worker-04
deviceFilter: ^nvme
config:
deviceClass: nvme
osdsPerDevice: "1"
skipUpgradeChecks: true
dataDirHostPath: /var/lib/rook
---
# CephBlockPool — ceph-blockpool from chart
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: ceph-blockpool
namespace: rook-ceph
spec:
failureDomain: host
replicated:
size: 2
deviceClass: nvme
---
# StorageClass created automatically from CephBlockPool
# provisioner: rook-ceph.rbd.csi.ceph.com
# parameters:
# clusterID: rook-ceph
# imageFormat: "2"
# imageFeatures: layeringQuick Reference
| Need | CRD | Notes |
| Block storage (PVC) | CephBlockPool + StorageClass | ceph.rook.io/v1 |
| Shared filesystem (RWX) | CephFilesystem + StorageClass | Requires MDS, CephFS CSI |
| S3-compatible object store | CephObjectStore + CephObjectStoreUser | RGW gateway, port 80 |
| NFS export of CephFS | CephNFS | NFS-Ganesha server |
| Cross-cluster RBD mirror | CephRBDMirror | For DR |
| CephFS snapshot mirror | CephFilesystemMirror | For DR |
| S3 bucket notifications | CephBucketTopic + CephBucketNotification | Kafka/AMQP/webhook |
| Tenant isolation in pool | CephBlockPoolRadosNamespace | Per-namespace RADOS namespace |
ceph-csi-drivers Chart (v1.20+)
CRITICAL: Rook v1.20 delegates CSI driver management to the ceph-csi-operator sub-chart (bundled with rook-ceph) + the separate ceph-csi-drivers chart. The drivers chart creates the ServiceAccounts, RBAC, and Driver CRs that the operator needs.
Always websearch rook/docs/v1.20/Helm-Charts/csi-drivers-chart/ for the latest recommended values.yaml before installing.
Installation
helm repo add ceph-csi-operator https://charts.ceph.io/ceph-csi-operator
helm install ceph-csi-drivers ceph-csi-operator/ceph-csi-drivers \
--namespace rook-ceph \
-f https://raw.githubusercontent.com/rook/rook/v1.20.2/deploy/charts/ceph-csi-drivers/values.yamlCritical: Driver Names Must Match StorageClasses
The ceph-csi-drivers chart defaults to driver names like rbd.csi.ceph.com, but existing StorageClasses reference rook-ceph.rbd.csi.ceph.com (prefixed with the operator namespace). Set the correct names:
drivers:
rbd:
enabled: true
name: rook-ceph.rbd.csi.ceph.com # Must match StorageClass provisioner
cephfs:
enabled: true
name: rook-ceph.cephfs.csi.ceph.com # Must match StorageClass provisioner
nfs:
enabled: false
nvmeof:
enabled: falseWrong names → StorageClass provisioner mismatch, volumes fail to provision.
Upgrade Notes (v1.19 → v1.20)
Breaking: CSI drivers moved to separate chart
In v1.19, Rook managed CSI drivers internally. In v1.20, a new ceph-csi-operator manages them via Driver CRs, and the ceph-csi-drivers Helm chart is required to create the backing ServiceAccounts + RBAC.
Installation order: rook-ceph → ceph-csi-drivers → rook-ceph-cluster
Known Issues (websearch before deploying)
- v1.20.0: Missing SA/RBAC — Without
ceph-csi-driverschart, ctrlplugin Deployments stuck at 0/2 withserviceaccount not found. Fixed by installing the drivers chart with correct driver names. (rook#17644) - v1.20.1: Leader election broken — RoleBindings only included prefixed SA names, but pods use unprefixed names (
rbd-ctrlplugin-sa). Provisioner can't acquire lease → PVCs hang. Fixed in v1.20.2 or patch RoleBindings to add unprefixed SAs. (rook#17787) - Helm upgrade conflicts — Server-side apply may conflict with existing
Driver/OperatorConfigCRs managed by Rook. Add--force-conflictsor delete old CRs before upgrade. - Ceph v20.2.0 read-affinity bug — Use v20.2.1+ or v20.2.2.
Migration Steps
- Upgrade
rook-cephchart to v1.20 (this deploys ceph-csi-operator sub-chart) - Retrieve existing CSI settings from your
rook-ceph-operator-configConfigMap - Install
ceph-csi-driverschart with driver names matching your StorageClasses - If upgrade fails with apply conflicts, use
helm upgrade --force-conflicts - Verify:
kubectl get deploy -n rook-ceph -l app.kubernetes.io/name=rbdshows ctrlplugin + nodeplugin running - Upgrade
rook-ceph-clusterchart
Config Changes
- CSI settings removed from
rook-ceph-operator-configConfigMap — now managed viaOperatorConfig+DriverCRs - CSI custom images remain in
rook-cephchart values (csi.csiOperator.controllerManager.manager.env) rookUseCsiOperator: trueis the new default in v1.20
Common Mistakes
- deviceFilter regex —
^vdb$matches exactlyvdb,^nvmematches any nvme device. Wrong regex = no OSDs created. - failureDomain: osd vs host —
osdallows OSDs on same node,hostspreads across nodes. Usehostfor production. - CephFS
dataPools[].name— Required when multiple data pools. First pool is default. - CephFS preservePoolsOnDelete: false — Deleting the CRD deletes ALL data. Set
truefor safety. - ObjectStore gateway placement — RGW must be scheduled on a node with OSDs or have network access to them.
- CephBlockPool size: 1 — No redundancy. Data lost if OSD fails. Only for test/dev or slow tier with HDD.
- OSD per device filter —
osdsPerDevice: "1"creates one OSD per device. Default may create too many. - dashboard ssl: false — Only for internal clusters. Enable SSL (
ssl: true) for external access. - Insufficient mon memory — MONS need minimum 512Mi RAM. Starving mons causes cluster instability.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

