Secrets Management Review

Find where credentials are stored, who can read them, and how quickly they can be replaced.

Purpose

Use this when:

• Reviewing how a service receives its credentials.
• A secret has been committed, logged, or otherwise exposed.
• Onboarding a service to production.
• Deciding between a secret store, sealed secrets, and a cloud provider's native solution.

Do not use this when:

• The problem is TLS certificate validation — use tls-debugging.
• The problem is IAM permissions rather than credential handling — use aws-production-debugging.

This skill is defensive. It covers protecting credentials in systems you operate. It does not cover extracting credentials from systems you do not control.

What this skill assumes: access to the repository, the deployment manifests, and the CI configuration. All three hold secrets in different ways, and reviewing only one misses most of the exposure.

Operating Procedure

1. Enumerate the secrets. What credentials does this service need, and what does each grant?
2. Trace each one's path from where it is stored to where it is used. Every stop is an exposure point.
3. Check each storage location against the table in step 2 below.
4. Check access: who and what can read it, and is that audited?
5. Check rotation: can it be replaced without downtime, and has that ever been done?
6. Check blast radius: what does this credential grant, and can it be narrowed?
7. Rank findings by what an attacker with the credential could actually do.

What you found by scanning is Observed. Absence of a finding is not proof of absence — say "no secrets found by these checks", not "there are no secrets in the repository".

Investigation

Step 1 - find secrets in code and history

grep -rnE '(password|passwd|secret|token|api[_-]?key|private[_-]?key|credential)\s*[:=]\s*["'"'"'][^"'"'"']{8,}' \
  --include='*.{tf,tfvars,yaml,yml,json,env,py,js,ts,go,rb,java,sh}' . | head -30

grep -rnE 'AKIA[0-9A-Z]{16}|ghp_[A-Za-z0-9]{36}|-----BEGIN [A-Z ]*PRIVATE KEY-----' . | head -20

Then check history, because removing a secret from the current tree does nothing:

git log --all --full-history -p -S 'AKIA' -- . | head -40
git rev-list --all | head -500 | xargs -I{} git grep -l 'BEGIN RSA PRIVATE KEY' {} 2>/dev/null | head

For anything serious, use a purpose-built scanner (gitleaks, trufflehog) — the greps above find the obvious cases and miss encoded or unusually named ones.

A secret in Git history is compromised. Rewriting history does not undo it: the repository was cloned, cached by the forge, indexed by tooling, and possibly forked. Rotation is the only real remediation, and it comes first. Rewriting history is optional cleanup afterwards.

Step 2 - where secrets end up

Trace each credential from its source to the process that uses it. Every location in this table is a place someone can read it.

kubectl get secrets -A --no-headers | wc -l
kubectl get secret <name> -n <ns> -o jsonpath='{.data}' | jq 'keys'
kubectl auth can-i get secrets --namespace <ns> --as system:serviceaccount:<ns>:<sa>

Kubernetes Secrets being base64 rather than encrypted is the most widely misunderstood point here. Base64 is an encoding, not a protection. The controls that matter are RBAC (who can get them), encryption at rest in etcd, and not mounting secrets into pods that do not need them.

grep -rn 'sensitive\s*=\s*true' --include='*.tf' . | head
terraform state pull | jq -r '.resources[].instances[].attributes | keys[]' 2>/dev/null | sort -u | grep -iE 'password|secret|key' | head

sensitive = true in Terraform hides a value from plan output only. It is stored in state in plaintext regardless. Anyone who can read the state bucket can read every secret in it — which is why state buckets need the same access controls as a secret store.

Step 3 - access and least privilege

For each secret, ask what the credential actually grants:

kubectl get rolebindings,clusterrolebindings -A -o json \
  | jq -r '.items[] | select(.roleRef.name | test("admin|edit|cluster-admin"))
           | "\(.kind) \(.metadata.namespace // "-")/\(.metadata.name) -> \(.roleRef.name)"'

The strongest control available is not storing a credential at all: workload identity (IRSA on EKS, Workload Identity on GKE, managed identities on Azure, OIDC from CI) exchanges a platform-issued token for short-lived credentials. Nothing static exists to leak.

Step 4 - rotation

For each secret, answer:

1. Can it be rotated without downtime? This requires supporting two valid credentials at once during the changeover. If not, rotation means an outage, so it will not happen.
2. How long does it take? If the answer is "a day", it will not happen during an incident when it matters most.
3. When was it last rotated? "Never" is a common and honest answer, and it is the finding.
4. Is it automated? Manual rotation decays — people leave, runbooks go stale.
5. Is there an inventory? During an exposure you need to know every consumer of a credential within minutes.

A rotation procedure that has never been executed is a plan, not a capability. The useful test is whether the team has rotated this specific credential in the last year.

Step 5 - responding to an exposure

Order matters, and the instinct to clean up first is wrong.

1. Rotate first. Issue a new credential and cut consumers over. The exposed one is compromised from the moment it was exposed, not from the moment it was noticed.
2. Revoke the old one. Rotation without revocation leaves it valid.
3. Check for use. CloudTrail, audit logs, access logs — was it used, from where, for what? This determines whether this is a cleanup or an incident.
4. Assess the blast radius. What did it grant? Assume everything reachable with it was reached.
5. Then clean up. Remove it from the repository, purge history if worthwhile, rotate anything it could have been used to obtain.
6. Fix the path. How did it get there? A pre-commit hook and a CI secret scan prevent the next one; a postmortem prevents the class.

Do not skip step 3. "It was only in a private repository" is not a risk assessment — check the access logs.

Decision Tree

Reviewing secret handling
├── Secret in source or Git history
│   └── COMPROMISED → rotate, revoke, check usage, then clean up
├── Secret in an image layer or CI log
│   └── COMPROMISED → same, plus rebuild the image
├── Secret in Terraform state
│   ├── State encrypted and access restricted → acceptable, document it
│   └── Otherwise → treat as exposed to everyone with state access
├── Kubernetes Secret
│   ├── etcd encryption on + RBAC scoped + not over-mounted → acceptable
│   └── Otherwise → tighten before anything else
├── Long-lived static cloud credential
│   ├── Workload identity available → migrate; nothing to leak
│   └── Not available → shortest practical lifetime + rotation automation
└── Rotation
    ├── Never performed → the finding; test it
    └── Automated and tested → good

Common Causes

Safety

Read-only

Scanning code and history, listing secret names, reading RBAC, checking Terraform for sensitive flags, and reviewing rotation documentation.

Note what is not read-only in effect: kubectl get secret -o yaml prints the base64 value to your terminal, into your shell history, and into any session recording. terraform state pull writes every secret in the state to a local file. Prefer reading key names (jq 'keys') over values, and delete any local copies when finished.

Never paste a secret value into a ticket, a chat message, a paste service, or an AI tool — even to ask whether it is valid. That is another exposure, and it is one people create while investigating the first one.

Potentially disruptive

• Rotating a credential — every consumer using the old value fails at the moment of revocation. Enumerate consumers first, support both values during the overlap, then revoke.
• Tightening RBAC — removes access that something may depend on in a way nobody documented. Check audit logs for actual usage before removing a binding.
• Enabling etcd encryption at rest — requires a control plane change and rewriting existing Secrets to pick up encryption. Existing Secrets stay in plaintext until they are re-written.
• Migrating to a secret manager — a deployment change for every consumer. Stage it.
• Adding a CI secret scanner in blocking mode — will block merges on existing findings, some of them false positives. Run in reporting mode first.

Destructive

• Rewriting Git history (git filter-repo, BFG) to purge a secret — rewrites every commit SHA, breaks every open branch, PR and clone, and invalidates signatures and any external references to commits. Required first: rotate the credential (this is the actual remediation; history rewriting is cosmetic), a full backup of the repository, coordination with everyone who has a clone, and agreement that the disruption is worth it. Note that the forge retains unreferenced objects and caches; a rewrite does not guarantee removal.
• Revoking a credential before rotating — takes down every consumer immediately. This is occasionally the right call during an active compromise, where stopping the attacker justifies the outage. Required first: an explicit decision, stated, with someone accountable for it.
• Deleting a KMS key or a secret-store entry — anything encrypted with it becomes permanently unrecoverable. Required first: confirm nothing uses it (audit logs over a meaningful window), and prefer disabling, which is reversible.
• Deleting a Kubernetes Secret in use — pods mounting it fail to start; running pods keep the mounted copy until they restart, which makes the failure appear later and look unrelated. Required first: identify every pod and workload referencing it.

Verification

Before: 3 credentials in Git history; Terraform state unencrypted; no rotation ever performed
After:  all 3 rotated and revoked, no usage found in audit logs; state bucket encrypted with
        access limited to 2 roles; rotation automated and executed successfully once

gitleaks detect --no-banner --redact 2>/dev/null || echo "install gitleaks"
aws s3api get-bucket-encryption --bucket <state-bucket>
kubectl auth can-i get secrets --namespace <ns> --as system:serviceaccount:<ns>:default

Verify four things, in this order:

1. The old credential no longer works. Attempt to use it. Rotation without revocation is the most common incomplete remediation there is.
2. The new one works everywhere. Every consumer, not just the one you tested.
3. The exposure path is closed. The scanner runs in CI; the pre-commit hook is installed; the .gitignore covers it.
4. Rotation is repeatable. Someone other than the author has done it, following the runbook.

For an exposure, also record the audit-log finding — used or not used, from where — because that is what determines whether this was a near miss or a breach.

Anti-Patterns

Bad: Delete the commit containing the secret and move on. Better: The secret is compromised from the moment it was pushed. Rotate first; history cleanup is optional and does not undo the exposure.

Bad: Base64-encode a secret for storage. Better: Base64 is an encoding, not encryption. Anyone can decode it, and it creates a false sense of protection that stops people asking about the real controls.

Bad: sensitive = true in Terraform, so it is protected. Better: It is hidden from plan output only. State stores it in plaintext. Protect the state bucket as you would a secret store.

Bad: One service account with broad permissions, shared across services. Better: Rotating it becomes impossible because the blast radius is everything. Per-service credentials make rotation routine.

Bad: Print the secret's value to confirm it is set. Better: Check its length. Log masking is defeated by any transformation, and the log outlives the debugging session.

Bad: Document a rotation procedure and consider rotation solved. Better: An unexecuted procedure is a plan. Run it once against production before you need it at three in the morning.

Bad: Store long-lived cloud access keys in CI. Better: Use OIDC. There is no static credential to leak, the tokens are short-lived, and the trust relationship is auditable.

Composition

• container-image-review — secrets baked into image layers.
• kubernetes-production-readiness — how a workload receives and mounts credentials.
• terraform-review — credentials in infrastructure code and state.
• github-actions-debugging — OIDC, environment secrets, and fork boundaries in CI.

References

• OWASP: Secrets Management Cheat Sheet
• Kubernetes: Encrypting Secret Data at Rest
• Kubernetes: Good practices for Secrets