Kubernetes GitOps (homelab)

~/homelab-gitops (Flux, pushes to main) is the source of truth and its CLAUDE.md is the detailed reference. Change the cluster through Git; a manual kubectl apply gets reverted by Flux.

Topology

Platform: Flux, Cilium (CNI + kube-proxy replacement), MetalLB (pool 192.168.10.202-204), Traefik v3 on .203 (*.sammasak.dev), cert-manager (letsencrypt-prod, Cloudflare DNS-01), Authentik, KEDA + HTTP add-on, Kyverno, zot. Gone (do not reintroduce without an ADR): ingress-nginx, Harbor, Falco, Tempo, Pushgateway, Cloudflare tunnel, VM workloads, incident-responder, sandboxes, forge, collab-vault.

Conventions

• New app: cp -r apps/_template apps/<app>, then add <app>/ to apps/kustomization.yaml (forgetting this deploys nothing, silently). Full walkthrough: docs/adding-an-app.md.
• Validate before pushing: devenv tasks run gitops:validate (kustomize build + kubeconform of the working tree; ~15 s). The repo's hooks re-run it on the staged tree and the pushed sha (gitops-validate <rev>); there is no Justfile.
• chat and llm are separate Flux Kustomizations so slow model pulls cannot block apps.
• Ingress: ingressClassName: traefik, traefik.ingress.kubernetes.io/* annotations, router.tls: "true", cert-manager.io/cluster-issuer: letsencrypt-prod. Authentik-gated apps add traefik.ingress.kubernetes.io/router.middlewares: "authentik-authentik-forward-auth@kubernetescrd". ntfy is deliberately un-gated.
• Resources: CPU/memory requests and a memory limit; CPU limits are optional and usually wrong. Namespace PSS: enforce: baseline, warn/audit: restricted.
• Images by @sha256: digest from registry.sammasak.dev/lab/. Static sites are amd64+arm64 indexes: after pushing the amd64 image, scripts/add-arm64.sh adds the arm64 variant without a rebuild.

Scale-to-zero traps

• HTTP apps sit at zero replicas behind the KEDA HTTP interceptor. Zero pods when idle is expected; check the HTTPScaledObject before calling it an incident.
• A single failing check after idle is usually a cold start; retry once before declaring an outage.
• If the KEDA interceptor is down, every scaled app 502s at once.

Debug entry points

Run cluster commands on Lenovo (k3s server: flux and the kubeconfig live there). Elsewhere kubectl has no kubeconfig and fails with localhost:8080 ... refused: a missing client config, not a cluster outage.

devenv tasks run gitops:status       # Kustomizations + HelmReleases
devenv tasks run gitops:unhealthy    # non-Running pods, not-Ready Flux objects
devenv tasks run gitops:sync --input name=apps
flux logs --all-namespaces --level=error

For deeper diagnosis dispatch the k8s-debugger agent.

Telemetry privacy (native OTLP)

Boundary: docs/runbooks/native-otlp-ingestion.md. Only allowlisted metrics and sanitized Claude tool_result events may leave the collector; raw output, arguments, session/user IDs and unbounded label values must not reach Loki or Prometheus, and Codex logs stay dropped. On main, log scope/schema/trace/span metadata still reaches Loki; PR #138 (open) scrubs it. A collector change (clusters/homelab/infra/observability.yaml) needs:

• ALLOY_BIN=<alloy> bash scripts/test-native-otlp-sanitizer.sh (on main): auth, size, redaction and restart at a debug exporter; it never touches Loki.
• TMPDIR=<scratch> python3 scripts/test-native-otlp-log-loki.py <alloy> <loki> (added by PR #138): real isolated Alloy and Loki from the repo config, asserting the stored log schema. Synthetic local-backend proof only.

Merging to main is the Flux rollout, so it needs explicit approval. Live acceptance additionally needs the applied revision and an authorized synthetic check at the real receiver and Loki; Ready=True or green fixtures are not acceptance (system gate: homelab-gitops#120). A privacy rollback disables native logs; never restore a known-leaking config.

Destructive deletes (kyverno-enforced)

Deleting a PVC, PV, or Namespace — or a Flux Kustomization/HelmRelease whose prune would cascade to them — is admission-denied until the object carries the break-glass annotation. That annotation is the HUMAN's confirmation step, not the agent's: the validate-bash hook blocks an agent from setting sammasak.dev/allow-delete. So do not run the annotate — instead surface the denial and ask the human to perform the two-step themselves:

# Human runs these after re-reading the denial they are overriding:
kubectl -n <ns> annotate pvc <name> sammasak.dev/allow-delete=true
kubectl -n <ns> delete pvc <name>

Removing an app's manifests from Git is the other path: Flux's own service account is exempt, so the prune deletes its namespace and PVCs (reclaim Delete) with no annotation. The Git change is the gate, so get the owner's explicit OK and archive the volumes first (/var/lib/rancher/k3s/storage/pvc-* on the node, as root).