TLS Debugging

Read the actual certificate the server presents, and compare it against what the client requires.

Purpose

Use this when:

• x509: certificate has expired, certificate signed by unknown authority, SSL certificate problem: unable to get local issuer certificate.
• x509: certificate is valid for X, not Y.
• handshake failure, no shared cipher, protocol version.
• HTTPS works in a browser but fails from an application, or works from one client and not another.
• A mutual-TLS connection is rejected.

Do not use this when:

• The name does not resolve — use dns-troubleshooting.
• The handshake succeeds and the HTTP response is wrong — use http-debugging.

What this skill assumes: openssl s_client, and network reachability to the server. If the TCP connection itself fails, this is not a TLS problem.

Operating Procedure

1. Get the exact error. TLS errors are precise and each points at a distinct cause.
2. Fetch the certificate the server actually presents, with the correct SNI. What is deployed is frequently not what someone believes is deployed.
3. Check the four things in order: validity dates, hostname/SAN, chain completeness, trust.
4. Compare client requirements: protocol version, ciphers, trust store.
5. Check the clock on both ends. Skew produces "expired" and "not yet valid" errors on perfectly good certificates.
6. Fix and verify from the client that failed, not from your laptop.

The certificate you retrieved with openssl is Observed. What a browser shows is not the same evidence — browsers fill in missing intermediates from cache and from AIA fetching, which is precisely the behaviour that hides the most common production TLS bug.

Investigation

Step 1 - retrieve the certificate with SNI

openssl s_client -connect <host>:443 -servername <host> </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

-servername sets SNI and is required. Without it, a server hosting multiple sites returns its default certificate, and you diagnose the wrong one. Omitting -servername is the single most common mistake in TLS debugging.

Output to check, in order:

subject=CN = api.example.com
issuer=C = US, O = Let's Encrypt, CN = R11
notBefore=Jul 15 00:00:00 2026 GMT
notAfter=Oct 13 23:59:59 2026 GMT
X509v3 Subject Alternative Name:
    DNS:api.example.com, DNS:www.api.example.com

1. Dates — is now between them?
2. SAN — does it contain the exact name being requested? The CN is ignored by every modern client; only the SAN list counts. A certificate with the right CN and a missing SAN entry fails, and the error message mentioning CN misleads people into thinking it should work.
3. Wildcards match one label only. *.example.com covers api.example.com but not a.b.example.com and not bare example.com.

Step 2 - the chain

openssl s_client -connect <host>:443 -servername <host> -showcerts </dev/null 2>/dev/null \
  | grep -E 'depth|verify|^ *[0-9] s:|^ *[0-9] i:'

A complete chain looks like this, with each certificate's issuer matching the next one's subject:

 0 s:CN = api.example.com
   i:C = US, O = Let's Encrypt, CN = R11
 1 s:C = US, O = Let's Encrypt, CN = R11
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

The incomplete chain is the most important failure in this skill. If only certificate 0 is returned, the server is not sending its intermediate. Browsers usually paper over this — they cache intermediates from other sites and can fetch them via the AIA extension — so it works in a browser and fails from curl, Go, Java, Python and every other client.

The symptom is characteristic: "it works on my machine, it works in the browser, it fails in production code". The fix is server-side, in the certificate bundle, not in the client.

openssl s_client -connect <host>:443 -servername <host> </dev/null 2>&1 | grep -E 'Verify return code'

Step 3 - protocol and cipher

openssl s_client -connect <host>:443 -servername <host> -tls1_2 </dev/null 2>&1 | grep -E 'Protocol|Cipher'
openssl s_client -connect <host>:443 -servername <host> -tls1_3 </dev/null 2>&1 | grep -E 'Protocol|Cipher'

A handshake failure with no certificate error is a version or cipher mismatch. Common in practice: a server that has disabled TLS 1.0/1.1 (correctly) and a client stuck on an old runtime or an old OpenSSL build. Test each version explicitly to find where the negotiation fails.

Check your own openssl version too — TLS 1.3 support requires OpenSSL 1.1.1 or newer, and an old client tool will report a failure that has nothing to do with the server.

Step 4 - the client's trust store

curl -vsS https://<host>/ 2>&1 | grep -E 'CAfile|SSL certificate|subject|issuer'
openssl version -d          # OPENSSLDIR: where openssl looks

Clients disagree about trust because they use different stores:

This is why a private CA works everywhere except in one language runtime, and why a container built FROM scratch fails on every HTTPS call: it has no ca-certificates package at all. That one accounts for a large share of "TLS is broken in the container" reports.

Step 5 - clock and expiry management

timedatectl                  # or: date -u
openssl s_client -connect <host>:443 -servername <host> </dev/null 2>/dev/null \
  | openssl x509 -noout -checkend 604800 || echo "EXPIRES WITHIN 7 DAYS"

Clock skew produces certificate has expired and certificate is not yet valid on entirely valid certificates. Check the clock on both ends before replacing anything — issuing a new certificate to fix a clock problem does not help, and the new one will fail the same way.

In Kubernetes with cert-manager:

kubectl get certificate -A
kubectl describe certificate <name> -n <namespace>
kubectl get certificaterequest,order,challenge -n <namespace>

Work down that chain: Certificate → CertificateRequest → Order → Challenge. The Challenge is where ACME validation failures show up, and the message is usually specific — a DNS-01 record that was not created, or an HTTP-01 path blocked by an ingress rule.

Step 6 - mutual TLS

openssl s_client -connect <host>:443 -servername <host> \
  -cert client.crt -key client.key -CAfile ca.crt </dev/null 2>&1 | head -30

openssl x509 -in client.crt -noout -subject -dates -ext extendedKeyUsage

For mTLS, check both directions: the server must trust the client's CA, and the client certificate needs extendedKeyUsage including clientAuth. A certificate issued for server use will be rejected for client authentication, and the error rarely says so clearly.

Without -cert, a server requiring client certificates typically shows a Acceptable client certificate CA names list in the output — that list tells you exactly which CA the server expects, which is often the fastest way to find the mismatch.

Decision Tree

TLS connection fails
├── "certificate has expired"
│   ├── Clock is correct → genuinely expired; renew
│   └── Clock is wrong   → fix NTP; the certificate is fine
├── "certificate is valid for X, not Y"
│   ├── Name missing from SAN → reissue with the right SAN
│   ├── Wildcard depth wrong  → *.a.com does not cover b.c.a.com
│   └── SNI not sent          → client is getting the default vhost
├── "unable to get local issuer" / "unknown authority"
│   ├── Browser works, code fails → incomplete chain on the server
│   ├── Private CA               → the client's trust store lacks it
│   └── Container from scratch   → no ca-certificates in the image
├── "handshake failure" with no certificate error
│   ├── Protocol version mismatch → test -tls1_2 / -tls1_3
│   └── No shared cipher          → client or server cipher policy
├── Works from one client, not another → compare trust stores
└── mTLS rejected
    ├── Server does not trust the client CA
    ├── Client certificate lacks clientAuth EKU
    └── Client is not sending a certificate at all

Common Causes

Safety

Read-only

openssl s_client, openssl x509, curl -v, reading cert-manager resources, and timedatectl. These open a connection and read what is presented.

One caution: openssl s_client output includes the full certificate chain, which is public information. It does not include private keys — but the commands people reach for next often do. Never paste the contents of a .key file, a Kubernetes TLS Secret, or a PKCS#12 bundle into a ticket, a chat channel, or a log.

Potentially disruptive

• Replacing a certificate on a live listener — most servers need a reload. A graceful reload (nginx -s reload) drops nothing; a restart drops every connection. Validate the new certificate and key match before reloading:

  openssl x509 -noout -modulus -in cert.pem | openssl md5
  openssl rsa  -noout -modulus -in key.pem  | openssl md5

  Identical hashes mean the key matches the certificate. A mismatch means the server will fail to start — which, after a reload, is an outage.

• Changing cipher suites or the minimum TLS version — disabling TLS 1.0/1.1 is correct and will break genuinely old clients. Check your access logs for negotiated versions before doing it.
• Enabling HSTS — browsers will refuse plain HTTP for the max-age duration and you cannot retract it early. Start with a short max-age.
• Rotating a private CA — every client that trusts it must be updated. Distribute the new CA and let both be trusted before removing the old one.
• Enabling mTLS enforcement — every client without a valid certificate is rejected immediately. Run in a permissive mode first if the implementation supports it.

Destructive

• Deleting a TLS Secret or a private key — if the key is not backed up, the certificate is unusable and must be reissued. For a private CA, losing the key means every certificate it issued must be replaced. Required first: confirm a backup exists and has been tested, or confirm the certificate can be reissued immediately.
• Revoking a certificate — cannot be undone. Revoking the wrong certificate takes a service offline for every client that checks revocation. Required first: confirm the serial number against the certificate actually in use.

A note on the tempting wrong answer: disabling verification (curl -k, InsecureSkipVerify: true, verify=False) is not a fix. It is acceptable as a diagnostic to confirm that the problem is certificate validation and not something else — and then it must be removed. Shipping it removes the protection TLS exists to provide, silently, in a way that no monitoring will catch.

Verification

Before: Go client fails with x509: certificate signed by unknown authority; browser fine
After:  openssl verify return code 0; Go, curl and Java clients all connect; chain depth 2

openssl s_client -connect <host>:443 -servername <host> </dev/null 2>&1 \
  | grep -E 'Verify return code|Protocol|Cipher'

curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://<host>/

openssl s_client -connect <host>:443 -servername <host> </dev/null 2>/dev/null \
  | openssl x509 -noout -checkend 2592000 && echo "valid for at least 30 more days"

Verify from the client that failed. A certificate that satisfies openssl on your laptop can still fail in a Java service with its own keystore — that is the whole class of bug this skill exists for. Test every client type that connects.

Then verify renewal, not just the current certificate: a certificate fixed by hand today expires on the same schedule as the last one. Confirm automation is in place and check that expiry monitoring exists and fires with enough notice.

Anti-Patterns

Bad: It works in the browser, so the certificate is fine. Better: Browsers complete incomplete chains from cache and via AIA fetching. openssl s_client and curl do not. The browser is the least reliable TLS test available.

Bad: Add -k / InsecureSkipVerify to get past the error. Better: That disables the protection entirely and silently. Use it once to confirm the diagnosis and then fix the chain or the trust store.

Bad: Run openssl s_client without -servername. Better: You get the default virtual host's certificate and diagnose a certificate nobody is actually using.

Bad: Trust the CN field. Better: Modern clients ignore CN entirely. Only the subjectAltName extension matters.

Bad: Reissue the certificate when it reports as expired. Better: Check the clock first. Skew produces identical errors, and a new certificate fails the same way, which is a confusing half-hour.

Bad: Renew certificates manually when the alert fires. Better: Manual renewal fails eventually — during a holiday, after someone leaves. Automate it, and alert on approaching expiry as a backstop for the automation.

Composition

• http-debugging — once the handshake succeeds and the status code is wrong.
• dns-troubleshooting — when the name resolves to a host with a different certificate.
• kubernetes-networking — ingress TLS termination and Secret wiring.
• secrets-management-review — how private keys are stored, distributed and rotated.

References

• OpenSSL: s_client — every flag used here.
• RFC 6125 — why SAN supersedes CN for hostname verification.
• cert-manager: troubleshooting — the Certificate → Request → Order → Challenge chain.