rotation-guard

Everyone agrees rotation is good practice, so teams build the mechanism to change the credential and stop there, treating "the secret store now holds a new value" as the finish line. It isn't — a credential has consumers, and each one keeps its own copy of the old value somewhere (a pooled connection that authenticated once and stays open, an env var a container read at boot, a client SDK that cached the value in a singleton), and nothing updates that copy just because the source of truth changed. That gap produces two opposite failures, and a rotation mechanism has to avoid both at once: revoke the old credential before every consumer has actually moved to the new one, and you get an outage (a burst of auth failures hitting every long-lived connection at the same instant); never actually verify the old one got cut off, and you get the quieter bug — "rotation" that only ever updated the secrets manager's record while every real consumer, and anyone who grabbed a copy of the old value before this run, goes right on authenticating with it.

Step 1: find every rotation mechanism and every consumer it has to reach

Locate how each credential actually gets rotated — a scheduled Lambda/cron hitting a secrets-manager rotation API, a Vault dynamic-secret lease, a manual runbook, a Kubernetes Secret object updated by a GitOps pipeline or External Secrets Operator sync. Then, separately, find every place that credential is consumed: a DB connection pool's config, an HTTP client built once at process start and reused, a Kubernetes container's env/envFrom vs. a mounted volume, a sidecar (Vault Agent, Envoy, a cache layer) that fetched it independently of the main process. These two lists rarely match in size — a single rotation event usually has to reach several independent consumers, and the rotation mechanism only controls the first list. Everything in step 2 and 3 is about whether it actually reaches the second.

Step 2: does the mechanism give consumers a window to catch up, or does it revoke on the spot?

A single-value rotation — overwrite the one live credential and revoke the old one immediately — has no room for a consumer that hasn't reconnected yet; every connection still holding the old value starts failing the instant revocation lands, whether or not it's done anything wrong. That's the shape of a self-inflicted outage, not a hypothetical: it's exactly why AWS Secrets Manager and RDS ship a documented alternating-users rotation strategy for databases — two credentials (e.g. two DB users) that take turns being "current," so the previous one stays valid through the whole rotation cycle instead of being revoked the moment a new one is minted. Check which shape this codebase's rotation actually uses:

• No overlap at all — the old secret is invalidated in the same operation that issues the new one. This only works if every consumer reconnects atomically with the rotation, which nothing in a distributed system can guarantee; flag this as a built-in outage risk regardless of how reliable the rotation job itself is.
• A real overlap window — two valid credentials exist simultaneously for a deliberate period (alternating users, a previous/pending secret-manager stage, a Vault lease's grace period) long enough for the slowest realistic consumer (a connection pool's max idle lifetime, a sidecar's slowest resync interval) to pick up the new value before the old one is revoked. Check the window is sized against an actual number from this system, not a guess — a pool configured with a 30-minute max connection lifetime needs a longer overlap than a 30-second one.
• Vault/dynamic-secret leases specifically put the renewal burden on the client, not the server: a lease that isn't proactively renewed before its TTL (or that hits its non-renewable max TTL) is simply revoked, with no grace period at all. If this codebase uses Vault dynamic secrets, confirm something actually renews the lease on a schedule tied to its TTL — a client that only fetches a secret once at startup and never renews will have it revoked out from under it with no warning.

Step 3: the specific consumers that silently keep the old value

For each consumer found in step 1, check the one thing that actually determines whether it notices a rotation:

• Connection pools. A pool authenticates each physical connection once, at open time, and reuses it until the pool's own idle/max-lifetime setting closes it — a password rotation doesn't touch connections already open. If the pool has no max lifetime (or one longer than the rotation's overlap window from step 2), a connection opened before rotation can keep authenticating with the old credential for as long as the pool lets it live, invisibly, long past when the team believes the cutover happened.
• In-memory singletons and cached clients — an SDK client, a config object, anything built once at process start from an env var or a secrets-manager Get call and held for the process's lifetime. Nothing re-reads the source unless the code explicitly does, on a timer or a signal; absent that, the only thing that picks up a rotation is a full process restart.
• Kubernetes credential injection. env/envFrom sourced from a Secret is resolved once, at container start, copied into the process environment, and never touched again — updating the underlying Secret object does nothing to a running container's env vars. A Secret mounted as a volume, by contrast, does get its file updated on the node after the object changes (on a kubelet sync interval, typically well under a couple of minutes, with a documented exception for subPath mounts, which never get refreshed at all) — but only if the application actually re-reads that file on a timer or a file-watch; many apps read it once at startup too, which defeats the volume mount's whole advantage. If this is a Kubernetes workload, identify which of the two injection methods each consumer uses, and if it's env-based, confirm something (a rollout restart triggered by the rotation pipeline, a controller like Reloader) actually restarts the workload as part of rotation — otherwise "rotated" and "the running pods noticed" are two different, disconnected events.
• Sidecars and out-of-band caches. A Vault Agent, an Envoy/service-mesh sidecar, or any process that independently cached a fetched secret has its own refresh cadence, separate from the main application's — confirm it's actually configured to re-fetch before the main app would otherwise be starved of a valid credential, not assumed to "just handle it" because it's a dedicated secrets sidecar.

Step 4: prove the cutover happened, both directions

"We rotated it" is a claim about the secrets manager's record unless something checks the other side. Two separate things need evidence, not an assumption:

• The old credential is actually dead. Has anything attempted to use the old value after the declared cutover and confirmed it's rejected — a scheduled check, a post-rotation smoke test, even a one-off manual call? Without that, revocation that silently no-ops (a DB user that was supposed to be dropped but the drop step failed quietly, a Vault policy that still grants access through a different path) looks identical to a successful rotation until someone goes looking, which is the dangerous direction: the credential nobody thinks is live anymore keeps working.
• No consumer is still using the old one. Is there any signal — an auth-method/credential-ID tag in access logs, a metric on which secret version served a request, a DB's own connection-audit log — that would catch a consumer still authenticating with the pre-rotation value after the window from step 2 has closed? If the only evidence rotation "worked" is the absence of an outage, that's consistent with success and with the quieter failure (nothing actually got cut over, so of course nothing broke) equally well — they need to be told apart.

Report

Lead with which direction failed, since the fix is opposite for each: BLOCK for a rotation with no overlap window reaching a consumer whose pool/cache lifetime exceeds it (the outage case) — name the specific consumer and the lifetime mismatch; BLOCK equally for a revocation step that was never actually verified against a live attempt to use the old credential, or a credential still reachable through env-var injection with no restart trigger wired to rotation (the silent-non-propagation case). CAUTION when the mechanism's shape is right but unverified — an overlap window exists but was sized by guess rather than against a real pool/lease lifetime, a Kubernetes volume mount is used but it's unconfirmed whether the app re-reads the file or only reads it at startup. SAFE only with evidence for both step 4 checks: a demonstrated failed auth attempt against the old credential after cutover, and a positive signal (not just an absence of alerts) that every consumer class identified in step 1 is authenticating on the new one.

Boundaries

• Not secret-spill's territory — that skill is incident response for a credential that already leaked into git history or a log and needs rotating under time pressure. This skill audits the routine rotation mechanism itself, independent of any specific leak, and applies just as much to a rotation nothing has gone wrong with yet.
• Not cipher-guard's territory — that skill audits application-level data encryption key rotation (re-wrap vs. re-encrypt, KMS key policy scope) for data at rest. This skill is about operational credentials that authenticate a connection or a request (DB passwords, API keys, service tokens, Vault leases), not the keys that encrypt stored data.
• Doesn't design or implement a rotation pipeline from scratch (which Lambda, which secrets-manager API calls) — it audits whether a rotation mechanism that exists, or is being designed, actually reaches its consumers and gets verified, not which managed-rotation product to adopt.
• Can't observe a live fleet's actual connection ages or in-flight auth traffic — it can only point at the pool/client config that determines how long a stale connection can live and say what evidence would prove the cutover, not run that observation itself. Say plainly when a claim ("no consumer is still on the old credential") has no supporting evidence in the codebase rather than assuming the absence of a reported outage proves it.
• Doesn't cover human-held credentials shared across a team (a shared admin password passed around informally) — that's a credential-hygiene problem distinct from the automated/system-to-system rotation this skill audits.