Secrets Command

Structured guidance for secrets strategy, External Secrets Operator, Sealed Secrets, rotation, and Kubernetes-side secrets audit.

Activation

/platform-skills:secrets design   # ESO vs Sealed Secrets decision; choose a strategy for the cluster
/platform-skills:secrets eso      # ExternalSecret/SecretStore scaffold; debug sync errors
/platform-skills:secrets sealed   # seal, rotate, backup master key; troubleshoot decryption failures
/platform-skills:secrets rotate   # rotation runbook — update provider value, force sync, verify pods reload
/platform-skills:secrets audit    # find bad SA tokens, unhealthy ExternalSecrets, missing secrets

Interactive Wizard (fires when no mode is provided)

When invoked with no arguments, ask before proceeding:

Q1 — Mode?

What do you need?
  1. design   — choose ESO vs Sealed Secrets for a cluster or workload
  2. eso      — scaffold ExternalSecret/SecretStore, or debug a sync error
  3. sealed   — seal a new secret, rotate an existing one, or back up the master key
  4. rotate   — end-to-end rotation runbook for a live secret
  5. audit    — find service account token secrets, unhealthy ExternalSecrets, plain secrets in Git

Enter 1–5 or mode name:

Q2 — Context (after mode selected):

• design: What is the backend — AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, or no cloud provider?
• eso: Paste the ExternalSecret or SecretStore YAML, or describe the sync error.
• sealed: New secret or rotating an existing one? Which namespace?
• rotate: Which secret is rotating — database password, API key, TLS cert? Which backend?
• audit: Cluster name and namespace scope. Any specific concern — Git scan, SA tokens, ESO health?

Mode: design

Triggers: which pattern, ESO vs Sealed Secrets, choose secrets strategy, what should I use

Read references/secrets.md before responding.

Decision matrix

Pick ESO when:

• Cluster has workload identity (IRSA on EKS, Azure Workload Identity on AKS, Vault Kubernetes auth)
• Rotation should propagate automatically without a Git commit
• Secrets are managed centrally in a provider (team already uses AWS SM / Azure KV / Vault)

Pick Sealed Secrets when:

• No cloud provider or Vault
• Air-gapped environment
• Team prefers all cluster state — including encrypted secrets — reviewable in Git without a runtime backend

Handoffs:

• For filesystem or Git history scanning for leaked secrets → /platform-skills:trivy (--scanners secret)
• For Azure Key Vault identity setup → /platform-skills:azure identity
• For AWS IRSA setup → /platform-skills:aws

Mode: eso

Triggers: ExternalSecret, SecretStore, ClusterSecretStore, sync error, ESO, external secrets operator, SecretSyncError

Read references/secrets.md → External Secrets Operator section before responding.

Before generating commands, confirm:

• ESO namespace — common values: external-secrets, platform-system, kube-system. Discover with: kubectl get pods -A | grep external-secrets
• ESO deploy name — common values: external-secrets, external-secrets-controller. Discover with: kubectl get deploy -n <eso-namespace>
• ESO service account name — the SA annotated with IRSA or Workload Identity. Common: external-secrets-sa, but teams name it differently.

Substitute these into all namespace flags, kubectl logs, and serviceAccountRef fields below.

ExternalSecret scaffold (provider-agnostic)

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: app-team
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: my-secret-store
    kind: SecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: prod/app-team/db
        property: password

SecretStore — provider examples

AWS Secrets Manager (IRSA):

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-secret-store
  namespace: app-team
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-north-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

Required IAM (least privilege):

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:<region>:<account>:secret:prod/app-team/*"
}

Azure Key Vault (Workload Identity):

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-secret-store
  namespace: app-team
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://my-keyvault.vault.azure.net"
      serviceAccountRef:
        name: external-secrets-sa

The managed identity needs Key Vault Secrets User on the Key Vault.

HashiCorp Vault (Kubernetes auth):

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-secret-store
  namespace: app-team
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "app-team-role"
          serviceAccountRef:
            name: external-secrets-sa

Debug ESO sync errors

# Sync status and last sync time
kubectl get externalsecret -n app-team database-credentials

# Full status including error message
kubectl describe externalsecret -n app-team database-credentials

# Operator logs for this secret
kubectl logs -n external-secrets deploy/external-secrets \
  | grep -i "database-credentials"

# SecretStore health
kubectl get secretstore -n app-team my-secret-store
kubectl describe secretstore -n app-team my-secret-store

Mode: sealed

Triggers: SealedSecret, kubeseal, seal, sealed secrets, no key could decrypt, master key, backup

Read references/secrets.md → Sealed Secrets section before responding.

Interview — ask before generating any commands

Before writing any kubeseal or kubectl commands, collect the following. Different organisations deploy the controller with different names and namespaces.

Q1 — Controller namespace:
  Where is the sealed-secrets controller running?
  (common: sealed-secrets, kube-system, platform-system — check with: kubectl get pods -A | grep sealed)

Q2 — Controller Service name:
  What is the controller Service name? (used by kubeseal --controller-name — this is the Service, not the Deployment)
  (common: sealed-secrets, sealed-secrets-controller — check with: kubectl get svc -n <namespace> | grep sealed)

Q3 — Secret namespace:
  Which namespace will the unsealed Secret live in?

Q4 — Secret name:
  What is the name of the secret?

Q5 — Action:
  1. Seal a new secret
  2. Rotate an existing sealed secret
  3. Back up the master key
  4. Restore the master key
  5. Troubleshoot a decryption failure

Use the answers to substitute <controller-namespace>, <controller-name>, and <secret-namespace> in all commands below. Never output hardcoded namespace or deployment names — always use the values the user provided.

Note: <controller-name> means the Service name when used with kubeseal --controller-name, and the Deployment name when used with kubectl rollout restart deploy/<controller-name>. If they differ in your cluster, substitute accordingly.

Seal a new secret

# Fetch the cluster public key — run once per cluster or after key rotation
kubeseal --fetch-cert \
  --controller-name=<controller-name> \
  --controller-namespace=<controller-namespace> \
  > pub-cert.pem

# Create a plain Secret and immediately seal it — never commit the plain Secret
kubectl create secret generic <secret-name> \
  --namespace=<secret-namespace> \
  --from-literal=KEY=value \
  --dry-run=client -o yaml \
  | kubeseal --cert pub-cert.pem --format yaml \
  > <secret-name>-sealed.yaml

git add <secret-name>-sealed.yaml
git commit -m "chore: seal <secret-name> for <secret-namespace>"

Rotate an existing sealed secret

kubectl create secret generic <secret-name> \
  --namespace=<secret-namespace> \
  --from-literal=KEY=newvalue \
  --dry-run=client -o yaml \
  | kubeseal --cert pub-cert.pem --format yaml \
  > <secret-name>-sealed.yaml

git add <secret-name>-sealed.yaml
git commit -m "chore: rotate <secret-name>"

After GitOps applies the updated SealedSecret:

• Pods using a mounted volume pick up the new value automatically via kubelet refresh
• Pods using env vars require a rollout restart: kubectl rollout restart deployment/<name> -n <secret-namespace>

Back up the master key

kubectl get secret -n <controller-namespace> \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-master-key-backup.yaml

Store this file outside the cluster (password manager, vault, encrypted object store). Without it, sealed secrets cannot be decrypted after cluster recreation.

Restore the master key

kubectl apply -f sealed-secrets-master-key-backup.yaml
kubectl rollout restart deploy/<controller-name> -n <controller-namespace>

Troubleshoot

# Confirm the controller is running
kubectl get pods -n <controller-namespace>

# Inspect SealedSecret status
kubectl describe sealedsecret <secret-name> -n <secret-namespace>

# Controller logs for decryption errors
kubectl logs -n <controller-namespace> deploy/<controller-name> | tail -50

Mode: rotate

Triggers: rotate, rotation runbook, change password, new token, expired credential, update secret

Apply this runbook for any secret rotation:

Step 1 — Update the value in the provider

AWS Secrets Manager:

aws secretsmanager put-secret-value \
  --secret-id prod/app-team/db \
  --secret-string '{"password":"new-password"}' \
  --profile <aws-profile>

Azure Key Vault:

az keyvault secret set \
  --vault-name my-keyvault \
  --name db-password \
  --value "new-password"

Sealed Secrets: Re-seal with the new value (see sealed mode).

Step 2 — Force sync (ESO)

kubectl annotate externalsecret database-credentials \
  force-sync=$(date +%s) \
  -n app-team

Step 3 — Verify the Kubernetes Secret was updated

# Check the last sync time
kubectl get externalsecret database-credentials -n app-team

# Confirm the Secret data changed (check hash, not the value)
kubectl get secret database-credentials -n app-team \
  -o jsonpath='{.data.DB_PASSWORD}' | base64 -d | sha256sum

Step 4 — Reload pods if needed

# Pods using env vars require a restart to pick up the new Secret value
kubectl rollout restart deployment/<name> -n app-team
kubectl rollout status deployment/<name> -n app-team

Pods using mounted volumes reload automatically within the kubelet sync period (default 60s) — no restart needed unless the app reads the value at startup only.

Step 5 — Validate the application is using the new credential

# Application-specific validation — confirm connectivity to the backend
# Do NOT print secret values; test the connection functionally instead
kubectl exec -n app-team deploy/<name> -- \
  sh -c 'wget -qO- http://localhost:8080/healthz || curl -sf http://localhost:8080/healthz'
# For a database: test the connection, not the credential value
kubectl exec -n app-team deploy/<name> -- \
  sh -c 'pg_isready -h $DB_HOST -p $DB_PORT' 2>/dev/null || echo "adjust command for your backend"

Mode: audit

Triggers: audit, find plain secrets, SA token, kubernetes.io/service-account-token, secrets health, hygiene

This mode covers Kubernetes-side audit only. For filesystem and Git history scanning for leaked secrets → /platform-skills:trivy (fast, offline pattern check) or /platform-skills:kingfisher (adds live validation against the provider, blast-radius mapping, and revocation — reach for this when the question is "is it still live" rather than "does it match a pattern").

Find workloads using legacy SA token secrets

# Find all kubernetes.io/service-account-token type secrets
kubectl get secrets -A \
  --field-selector type=kubernetes.io/service-account-token \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,SA:.metadata.annotations.kubernetes\.io/service-account\.name'

Legacy manually-created SA token secrets do not auto-rotate. Migrate workloads to projected tokens (automatic since Kubernetes 1.24).

Check ExternalSecret sync health across namespaces

# All ExternalSecrets and their status
kubectl get externalsecrets -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.conditions[0].reason,LAST:.status.refreshTime'

# Find any not in Ready state
kubectl get externalsecrets -A \
  -o json \
  | jq -r '.items[] | select(.status.conditions[]?.type=="Ready" and .status.conditions[]?.status!="True") | "\(.metadata.namespace)/\(.metadata.name): \(.status.conditions[]?.message)"'

Check for plain secrets committed to Git

# Trivy handles a fast, offline first pass — hand off explicitly
# trivy repo --scanners secret <repo-url>
# or for a local clone:
# trivy fs --scanners secret .

# Kingfisher adds live validation, blast-radius mapping, and revocation
# kingfisher scan <repo-url-or-path>

For a Git history and filesystem scan → /platform-skills:trivy. For live validation of what's found, or revocation of a confirmed-live credential → /platform-skills:kingfisher

Check SecretStore health

kubectl get secretstores -A
kubectl get clustersecretstores

# Describe any not-ready stores
kubectl get secretstores -A -o json \
  | jq -r '.items[] | select(.status.conditions[]?.status!="True") | "\(.metadata.namespace)/\(.metadata.name): \(.status.conditions[]?.message)"'

Operational rules to validate

• No plain Secret manifests committed to Git — verify with git log --all -- '*.yaml' | xargs grep -l 'kind: Secret'
• All workloads use dedicated service accounts, not default
• No kubernetes.io/service-account-token type secrets for application API auth
• ClusterSecretStore used only for platform-wide secrets; namespace-scoped SecretStore for everything else
• Provider permissions scoped to minimum required paths — not Resource: "*" or all secrets in the vault

Common mistakes

• Committing plain Secret manifests to Git — use ESO, Sealed Secrets, or Vault. Rotate any secret that was ever in Git history.
• secretsmanager:GetSecretValue with Resource: "*" — scope to the specific path or secret name. Wildcard grants access to all secrets in the account.
• Using default service account for ESO — creates implicit access. Create a dedicated external-secrets-sa per namespace.
• Sealed Secrets rotation without --fetch-cert — if the controller key rotated since the last seal, the new SealedSecret cannot be decrypted. Always fetch the current cert before sealing.
• Skipping the master key backup before cluster destruction — without the backup, all sealed secrets in Git are permanently unreadable on the new cluster.
• Relying on env var pod restart for mounted volume secrets — mounted volume secrets refresh automatically; env vars do not. Know which pattern each workload uses before planning rotation.

Reference

Full guidance: references/secrets.md

For filesystem and Git history secret scanning: /platform-skills:trivy

For Azure Key Vault identity setup: /platform-skills:azure identity

For AWS IRSA setup: /platform-skills:aws