cert-manager Debug

TL;DR checklist

• Check controller, webhook, and cainjector health.
• Follow Certificate → CertificateRequest to the deepest error; for ACME issuers, continue through Order → Challenge.
• Diagnose the configured HTTP-01 or DNS-01 path before proposing a retry.

Key read-only checks

• Use kubectl describe on the issuer-appropriate resources and cmctl status certificate when available.

Common pitfalls

• Do not delete certificate resources merely to force another ACME attempt.

Agent procedure

Follow the Workflow, Safety rules, and Quality checklist below.

Quick references

• references/playbooks.md

Last verified: unverified

When to use

Use when a cert-manager-managed certificate misbehaves — a Certificate stuck not Ready, an ACME challenge that never validates, TLS that expired even though cert-manager is installed, a renewed cert the app still doesn't serve, or kubectl apply failing on cert-manager webhook errors — and the cause needs to be found.

Goal

Pinpoint where issuance fails: Issuer → Certificate → CertificateRequest → served TLS; for ACME issuers, also inspect Order and Challenge. Back the diagnosis with status/events and propose a fix without mutating live secrets, forcing renewals, or burning ACME rate limits.

Workflow

1. Baseline health first — kubectl get pods -n cert-manager: controller, webhook, and cainjector all Running. A crashlooping webhook explains "nothing cert-manager related can even be applied"; a down controller explains "nothing progresses" with zero errors anywhere.
2. Walk the chain top-down — kubectl describe the Certificate, then its CertificateRequest; for an ACME issuer, continue through Order and Challenge(s). The deepest resource with an error in status/events names the culprit; quote that exact message. cmctl status certificate <name> -n <ns> reports related resources in one read-only command if cmctl is available.
3. Issuer readiness — kubectl describe clusterissuer/<name> (or issuer): is it Ready? ACME account registration failures (missing/invalid account key secret, unreachable ACME server) stall every certificate under that issuer at once.
4. Route by challenge type once a Challenge holds the error:
   • HTTP-01: does the cm-acme-http-solver pod and configured Ingress or Gateway HTTPRoute exist, and is http://<domain>/.well-known/acme-challenge/<token> reachable from the internet? The self-check runs from inside the cluster — hairpin-NAT homelab setups fail it even when the outside world would succeed.
   • DNS-01: is the _acme-challenge.<domain> TXT record visible at the authoritative nameservers? Provider credential errors and split-horizon DNS are the two classic causes.
5. Check rate limits before any retry — if the error mentions rateLimited, stop: deleting and recreating certificates makes it worse. Point experiments at the Let's Encrypt staging endpoint instead.
6. Renewal and consumption issues — cert renewed (secret updated, check kubectl get certificate -o wide and the secret's cert dates via openssl) but clients still see the old cert: the consuming workload loads TLS at startup only and needs a rollout; ingress controllers generally pick up secret changes, pods mounting the secret do not.
7. Correlate and conclude — tie the quoted status/event message to the specific chain link and cause; propose the minimal fix and how to verify issuance end-to-end afterwards.

References

references/playbooks.md — chain-status and error-text → cause → fix tables (Certificate/ Order/Challenge states, HTTP-01 and DNS-01 walkthroughs, Let's Encrypt rate limits, webhook apply failures, renewal/consumption issues). Route there as soon as the chain walk surfaces an error message.

For reviewing TLS posture and secrets handling more broadly, see kubernetes-security and secrets-management. For the ingress resource itself misrouting traffic (as opposed to the certificate on it), start with kubernetes-debug.

Safety rules

• Read-only first, always: kubectl get/describe certificate,certificaterequest,order,challenge, issuer,clusterissuer, cmctl status certificate, dig/openssl s_client. None of these mutate state.
• Never delete a Certificate's TLS secret or run cmctl renew as a diagnostic — deleting the secret takes the endpoint's TLS down until reissuance succeeds and counts against ACME rate limits. Propose it explicitly; the user confirms.
• Never suggest delete/recreate retry loops against Let's Encrypt production — the duplicate certificate limit (5/week per identical name set) can lock issuance for days. Experiments go to the staging endpoint via a separate parallel issuer.
• Do not switch an existing production Issuer/ClusterIssuer to the staging URL in place — every certificate under it would renew as untrusted.

Output format

## Observations
<controller/webhook health, issuer readiness, from which command>

## Evidence
<chain walk: the deepest failing resource and its quoted status/event message>

## Root cause
## Fix
<manifest/config diff or action, marked as requiring user confirmation to apply>

## Verification
<certificate Ready check + end-to-end proof, e.g. openssl s_client -connect ... -servername ...>

Quality checklist

• The issuer-appropriate chain (Certificate → CertificateRequest, plus Order → Challenge for ACME) was walked and the deepest error message quoted, not paraphrased
• For ACME, the challenge type was identified and its specific playbook followed (HTTP-01 external reachability / DNS-01 TXT at authoritative NS)
• For ACME, rate-limit implications were considered before proposing any delete/retry
• Only read-only commands were run without confirmation; secret deletion and forced renewal were marked as requiring it
• The fix includes an end-to-end verification command, not just "the Certificate is Ready"