agentleFS
Aidas-devAidas-dev/k8s-agent-skills/skills/rook-ceph-operator/SKILL.md

rook-ceph-operator

Skill4 starsChanged 5 months agoMITView on GitHub

Use when deploying or managing Rook-Ceph CRDs — CephCluster, CephBlockPool, CephFilesystem, CephObjectStore, CephNFS, CephRBDMirror, and supporting resources. Covers operator chart config, cluster spec, CSI settings, storage classes.

  • Reads credentials

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If upgrade fails with apply conflicts, use helm upgrade --force-conflicts

    line 166
  2. Very specific:

    Verify: kubectl get deploy -n rook-ceph -l app.kubernetes.io/name=rbd shows ctrlplugin + nodeplugin running

    line 167

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Rook-Ceph Operator CRDs

Charts: rook-ceph + rook-ceph-cluster (from https://charts.rook.io/release) + ceph-csi-drivers (v1.20+, from https://charts.ceph.io/ceph-csi-operator)
Latest: v1.20.x (charts) | Ceph: v20.2.2 (Tentacle) | API: ceph.rook.io/v1
Install/upgrade order: rook-ceph → ceph-csi-drivers → rook-ceph-cluster

⚠️ v1.20 breaking change: CSI drivers are now managed by the separate ceph-csi-drivers Helm chart. Rook no longer creates CSI ServiceAccounts, RBAC, or Driver CRs. Without this chart, all PVC provisioning breaks. See Upgrade Notes below.

CRD Reference

CRDAPIPurposeKey Fields
CephClusterv1Core cluster CRD — defines monitors, OSDs, manager, network, storage topology.spec.mon.count, spec.mgr.count, spec.storage (nodes[], useAllNodes, deviceFilter, config.deviceClass), spec.cephVersion.image, spec.dataDirHostPath, spec.dashboard, spec.network, spec.placement
CephBlockPoolv1RBD block storage pool — creates Ceph pool + optional StorageClass.spec.failureDomain, spec.replicated.size, spec.deviceClass, spec.erasureCoded (dataChunks, codingChunks), spec.application
CephFilesystemv1CephFS shared filesystem — metadata + data pools, MDS daemons.spec.metadataPool, spec.dataPools[], spec.metadataServer.activeCount, spec.metadataServer.activeStandby
CephObjectStorev1S3-compatible object store — RGW gateway, metadata + data pools.spec.metadataPool, spec.dataPool, spec.gateway.port, spec.gateway.instances, spec.gateway.placement
CephObjectStoreUserv1S3 access key/secret for object store users.spec.store, spec.displayName
CephNFSv1NFS export of CephFS filesystem.spec.rados.namespace, spec.server.active
CephRBDMirrorv1RBD mirroring daemon for cross-cluster replication.spec.count
CephFilesystemMirrorv1CephFS snapshot mirroring across clusters.spec.placement (placement only)
CephFilesystemSubVolumeGroupv1Subvolume group for CephFS quota/UID/GID isolation.spec.filesystemName, spec.pinning, spec.quota
CephBucketTopicv1S3 bucket notification topic (Kafka, AMQP, webhook).spec.endpoint, spec.persistent, spec.opaqueData
CephBucketNotificationv1Binds topic to object store events.spec.topic, spec.events[], spec.filter (key/metadata/tagFilters)
CephClientv1Ceph auth key management for external clients.spec.caps
CephBlockPoolRadosNamespacev1Rados namespace within a block pool for tenant isolation.spec.blockPoolName, spec.mirroring
CephCOSIDriverv1COSI (Container Object Storage Interface) driver.Standard COSI config

Deployed Pattern

# CephCluster — via rook-ceph-cluster chart values.cephClusterSpec
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 1
  dashboard:
    enabled: true
    ssl: false
  storage:
    useAllNodes: false
    useAllDevices: false
    nodes:
      - name: worker-01
        deviceFilter: ^vdb$
      - name: worker-02
        deviceFilter: ^vdb$
        config:
          deviceClass: nvme
          osdsPerDevice: "1"
      - name: worker-03
        deviceFilter: ^vdb$
        config:
          deviceClass: nvme
          osdsPerDevice: "1"
      - name: worker-04
        deviceFilter: ^nvme
        config:
          deviceClass: nvme
          osdsPerDevice: "1"
  skipUpgradeChecks: true
  dataDirHostPath: /var/lib/rook
---
# CephBlockPool — ceph-blockpool from chart
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: ceph-blockpool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 2
  deviceClass: nvme
---
# StorageClass created automatically from CephBlockPool
# provisioner: rook-ceph.rbd.csi.ceph.com
# parameters:
#   clusterID: rook-ceph
#   imageFormat: "2"
#   imageFeatures: layering

Quick Reference

NeedCRDNotes
Block storage (PVC)CephBlockPool + StorageClassceph.rook.io/v1
Shared filesystem (RWX)CephFilesystem + StorageClassRequires MDS, CephFS CSI
S3-compatible object storeCephObjectStore + CephObjectStoreUserRGW gateway, port 80
NFS export of CephFSCephNFSNFS-Ganesha server
Cross-cluster RBD mirrorCephRBDMirrorFor DR
CephFS snapshot mirrorCephFilesystemMirrorFor DR
S3 bucket notificationsCephBucketTopic + CephBucketNotificationKafka/AMQP/webhook
Tenant isolation in poolCephBlockPoolRadosNamespacePer-namespace RADOS namespace

ceph-csi-drivers Chart (v1.20+)

CRITICAL: Rook v1.20 delegates CSI driver management to the ceph-csi-operator sub-chart (bundled with rook-ceph) + the separate ceph-csi-drivers chart. The drivers chart creates the ServiceAccounts, RBAC, and Driver CRs that the operator needs.

Always websearch rook/docs/v1.20/Helm-Charts/csi-drivers-chart/ for the latest recommended values.yaml before installing.

Installation

helm repo add ceph-csi-operator https://charts.ceph.io/ceph-csi-operator
helm install ceph-csi-drivers ceph-csi-operator/ceph-csi-drivers \
  --namespace rook-ceph \
  -f https://raw.githubusercontent.com/rook/rook/v1.20.2/deploy/charts/ceph-csi-drivers/values.yaml

Critical: Driver Names Must Match StorageClasses

The ceph-csi-drivers chart defaults to driver names like rbd.csi.ceph.com, but existing StorageClasses reference rook-ceph.rbd.csi.ceph.com (prefixed with the operator namespace). Set the correct names:

drivers:
  rbd:
    enabled: true
    name: rook-ceph.rbd.csi.ceph.com     # Must match StorageClass provisioner
  cephfs:
    enabled: true
    name: rook-ceph.cephfs.csi.ceph.com   # Must match StorageClass provisioner
  nfs:
    enabled: false
  nvmeof:
    enabled: false

Wrong names → StorageClass provisioner mismatch, volumes fail to provision.

Upgrade Notes (v1.19 → v1.20)

Breaking: CSI drivers moved to separate chart

In v1.19, Rook managed CSI drivers internally. In v1.20, a new ceph-csi-operator manages them via Driver CRs, and the ceph-csi-drivers Helm chart is required to create the backing ServiceAccounts + RBAC.

Installation order: rook-ceph → ceph-csi-drivers → rook-ceph-cluster

Known Issues (websearch before deploying)

  1. v1.20.0: Missing SA/RBAC — Without ceph-csi-drivers chart, ctrlplugin Deployments stuck at 0/2 with serviceaccount not found. Fixed by installing the drivers chart with correct driver names. (rook#17644)
  2. v1.20.1: Leader election broken — RoleBindings only included prefixed SA names, but pods use unprefixed names (rbd-ctrlplugin-sa). Provisioner can't acquire lease → PVCs hang. Fixed in v1.20.2 or patch RoleBindings to add unprefixed SAs. (rook#17787)
  3. Helm upgrade conflicts — Server-side apply may conflict with existing Driver/OperatorConfig CRs managed by Rook. Add --force-conflicts or delete old CRs before upgrade.
  4. Ceph v20.2.0 read-affinity bug — Use v20.2.1+ or v20.2.2.

Migration Steps

  1. Upgrade rook-ceph chart to v1.20 (this deploys ceph-csi-operator sub-chart)
  2. Retrieve existing CSI settings from your rook-ceph-operator-config ConfigMap
  3. Install ceph-csi-drivers chart with driver names matching your StorageClasses
  4. If upgrade fails with apply conflicts, use helm upgrade --force-conflicts
  5. Verify: kubectl get deploy -n rook-ceph -l app.kubernetes.io/name=rbd shows ctrlplugin + nodeplugin running
  6. Upgrade rook-ceph-cluster chart

Config Changes

  • CSI settings removed from rook-ceph-operator-config ConfigMap — now managed via OperatorConfig + Driver CRs
  • CSI custom images remain in rook-ceph chart values (csi.csiOperator.controllerManager.manager.env)
  • rookUseCsiOperator: true is the new default in v1.20

Common Mistakes

  • deviceFilter regex — ^vdb$ matches exactly vdb, ^nvme matches any nvme device. Wrong regex = no OSDs created.
  • failureDomain: osd vs host — osd allows OSDs on same node, host spreads across nodes. Use host for production.
  • CephFS dataPools[].name — Required when multiple data pools. First pool is default.
  • CephFS preservePoolsOnDelete: false — Deleting the CRD deletes ALL data. Set true for safety.
  • ObjectStore gateway placement — RGW must be scheduled on a node with OSDs or have network access to them.
  • CephBlockPool size: 1 — No redundancy. Data lost if OSD fails. Only for test/dev or slow tier with HDD.
  • OSD per device filter — osdsPerDevice: "1" creates one OSD per device. Default may create too many.
  • dashboard ssl: false — Only for internal clusters. Enable SSL (ssl: true) for external access.
  • Insufficient mon memory — MONS need minimum 512Mi RAM. Starving mons causes cluster instability.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.