k8s-storage

Patterns and decisions for ConfigMaps, Secrets, and persistent storage in self-hosted Kubernetes clusters.

ConfigMap vs Secret

Rule: When in doubt, use a Secret. Secrets can be encrypted at rest; ConfigMaps cannot. The only cost is base64 encoding, which is trivial.

ConfigMap YAML

apiVersion: v1
kind: ConfigMap
metadata:
  name: chlabs-io-config
data:
  PORT: "12345"                  # numbers must be quoted — data field is strings only
  MULTILINE_DATA: |
    Hello World!
    This is a multiline data.
binaryData:
  HELLO_WORLD: SGVsbG8sIFdvcmxkIQ==   # base64-encoded binary; decoded automatically when volume-mounted

Limits: Total size < 1 MB (keys + values). Name ≤ 253 chars, lowercase alphanumeric + - + ..

Secret YAML (Opaque)

# Create from .env file (most common)
kubectl create secret generic app.secret.env \
  --from-env-file=.env \
  --namespace=production
secret/app.secret.env created

# Inspect without revealing values
kubectl describe secret app.secret.env -n production

# Decode a specific key
kubectl get secret app.secret.env -n production -o json \
  | jq '.data | map_values(@base64d)'

Secret types: Opaque (generic), kubernetes.io/tls (TLS certs), kubernetes.io/dockerconfigjson (registry auth), kubernetes.io/ssh-auth, kubernetes.io/basic-auth.

Warning: Secrets are stored unencrypted in etcd by default. Enable encryption at rest for production clusters. Any user who can create a Pod in a namespace can read all Secrets in that namespace.

Env Vars vs Volume Mounts

Recommendation: Mount Secrets as volumes (readOnly: true). Use envFrom for ConfigMaps only when the app is designed to read env vars.

Pod consuming both (canonical pattern)

apiVersion: v1
kind: Pod
metadata:
  name: cm-secret-tester
spec:
  volumes:
    - name: secrets
      secret:
        secretName: app.secret.env
  containers:
    - name: app
      image: registry.k8s.io/busybox
      volumeMounts:
        - name: secrets
          readOnly: true
          mountPath: "/etc/secrets"   # each Secret key → /etc/secrets/<KEY>
      envFrom:
        - configMapRef:
            name: app.configmap       # all ConfigMap keys → env vars
      command: ["/bin/sh"]
      args: ["-c", "ls /etc/secrets && printenv"]

Selective keys: To mount only specific keys (or rename them), use items: under the secret: or configMap: volume definition.

Handling ConfigMap and Secret Updates

When a ConfigMap or Secret changes, running Pods do not automatically pick up the new values:

• envFrom — never updates; env vars are set at container start
• Volume mounts — files do update eventually (kubelet sync period, ~1 min), but most apps don't re-read files at runtime

Update strategies (pick one):

Hash-based naming:

# Creates configmap/app-config-<hash> — name changes when data changes
kubectl create configmap app-config --from-env-file=app.env --append-hash
# Update the Deployment to reference the new name → Pods restart automatically

Immutable resources (prevent accidental changes):

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-v3
immutable: true   # prevents updates; must delete and recreate to change
data:
  KEY: value

StorageClass

A StorageClass is cluster-scoped and defines the provisioner, reclaim policy, and binding mode. Installed by the CSI driver.

# k3s default (local-path) — inspect with:
kubectl get storageclass local-path -o yaml

Key fields:

Default StorageClass: Only one should have storageclass.kubernetes.io/is-default-class: "true". When a CSI driver installs its own StorageClass, verify it doesn't accidentally become the default.

Storage option comparison (self-hosted)

Access Modes

Most cloud block storage (EBS, Hetzner Cloud Volumes) only supports RWO. If you need multiple Pods on different nodes to share a volume, use Longhorn (RWX) or co-locate Pods with podAffinity.

PVC: Standalone (Deployment pattern)

Use when multiple Pods share the same volume. Requires podAffinity if the StorageClass only supports RWO.

# 1. Create the PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc.hcloud-volumes
  namespace: misc
spec:
  storageClassName: hcloud-volumes
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

# 2. Deployment referencing the PVC — podAffinity co-locates Pods on same node (required for RWO)
# Use preferredDuringScheduling (soft) not requiredDuringScheduling (hard) to avoid first-deploy deadlock:
# hard self-referential affinity deadlocks — no node has an app=cache Pod yet, so all Pods stay Pending
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache
  namespace: misc
spec:
  replicas: 5
  selector: { matchLabels: { app: cache } }
  template:
    metadata: { labels: { app: cache } }
    spec:
      affinity:
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values: [cache]
                topologyKey: kubernetes.io/hostname   # prefer same node
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: pvc.hcloud-volumes
      containers:
        - name: app
          image: ubuntu:24.04
          volumeMounts:
            - name: data
              mountPath: /etc/data

Tip: If your StorageClass supports RWX (e.g., Longhorn), drop the podAffinity rule and Pods can spread across nodes for better fault tolerance.

PVC: StatefulSet with volumeClaimTemplates

Use when each Pod needs its own independent volume (databases, per-instance state). Each Pod gets a dedicated PVC named <template-name>-<statefulset-name>-<ordinal> (e.g., data-cache-0, data-cache-1).

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cache
  namespace: misc
spec:
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: hcloud-volumes
        resources: { requests: { storage: 10Gi } }
  selector: { matchLabels: { app: cache } }
  template:
    metadata: { labels: { app: cache } }
    spec:
      containers:
        - name: app
          image: ubuntu:24.04
          volumeMounts:
            - name: data
              mountPath: /etc/data

No podAffinity needed — each Pod has its own PVC, so they can schedule independently on different nodes.

Deployment+PVC vs StatefulSet+volumeClaimTemplates

PVC cleanup: kubectl delete statefulset cache does not delete the PVCs. Delete them explicitly after verifying:

kubectl get pvc -n misc | grep cache   # verify which PVCs will be deleted
kubectl delete pvc data-cache-0 data-cache-1 data-cache-2 -n misc   # delete by name, not label

Avoid kubectl delete pvc -l app=cache — label-based bulk delete affects all PVCs with that label across all StatefulSets in the namespace.

kubectl Storage Commands

# PVCs (namespaced)
kubectl get pvc -n <namespace>
kubectl describe pvc <name> -n <namespace>
kubectl delete pvc <name> -n <namespace>   # WARNING: irreversible if reclaimPolicy=Delete — verify with kubectl get pv first

# PVs (cluster-scoped)
kubectl get pv
kubectl describe pv <name>

# StorageClasses (cluster-scoped)
kubectl get storageclass
kubectl get storageclass <name> -o yaml

# Expand a PVC (StorageClass must have allowVolumeExpansion: true)
kubectl patch pvc <name> -n <namespace> \
  -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

Backup Strategy

Two categories require backup: cluster datastore and PersistentVolume data.

Cluster datastore (k3s)

# On-demand etcd snapshot
k3s etcd-snapshot save --name manual-$(date +%Y%m%d)

# Upload to S3 automatically (add to k3s server flags)
--etcd-s3 --etcd-s3-bucket=my-backup-bucket --etcd-s3-region=us-east-1

Critical: Save /var/lib/rancher/k3s/server/token — etcd data is encrypted with this token. Without it, snapshots cannot be restored.

PersistentVolume data

General recommendation: Use Velero writing to S3-compatible storage. Store backups on a different system than the source data (different provider or region). Periodically test restores — untested backups are not backups.

# Velero backup (after installation)
velero backup create my-backup --include-namespaces production
velero restore create --from-backup my-backup