DNS Troubleshooting

Resolve the name yourself, at each layer, and find the first one that gives a different answer.

Purpose

Use this when:

• no such host, NXDOMAIN, SERVFAIL, or Temporary failure in name resolution.
• A name resolves to an old address after a DNS change.
• Resolution is intermittent or slow.
• A Kubernetes pod cannot resolve a Service name, or resolution takes hundreds of milliseconds.

Do not use this when:

• The name resolves and the connection fails — that is TCP; use linux-network-debugging or kubernetes-networking.
• The failure is a TLS certificate name mismatch — use tls-debugging.

What this skill assumes: dig or drill. nslookup and ping are poor DNS tools — they hide the response code, which is the most diagnostic field in the answer.

Operating Procedure

1. Get the exact error. NXDOMAIN, SERVFAIL and a timeout mean three different things.
2. Resolve from the failing context, not from your laptop. A container, a host and a laptop have different resolvers, different search domains, and different caches.
3. Walk the chain: local cache → configured resolver → authoritative server. Find the first layer that disagrees.
4. Check the record at the authority. That is ground truth; everything else is a cache.
5. Account for TTL. Most "DNS is broken after a change" reports are a cache that has not expired.
6. Fix at the layer that is wrong, and verify from the original failing context.

Which resolver answered is Observed only if you saw it in the dig output. A record you believe is correct because you set it is an Assumption until you query the authority.

Investigation

Step 1 - resolve, with the response code visible

dig +noall +comments +answer <name>
dig <name> A
dig <name> AAAA

The response code in the ;; ->>HEADER<<- line is the whole diagnosis:

SERVFAIL versus NXDOMAIN is the key split: NXDOMAIN is a definitive answer that the name does not exist, SERVFAIL means the resolver could not determine anything. Treating a SERVFAIL as a missing record sends you to create a record that already exists.

Step 2 - which resolver, and what is configured

cat /etc/resolv.conf
dig @<resolver-ip> <name>
resolvectl status 2>/dev/null | head -30

Query each configured resolver directly. If one answers correctly and another does not, you have found the fault without going further — and an intermittent failure with two resolvers configured is almost always one of them being wrong.

/etc/resolv.conf also holds the two settings that cause the most confusion:

search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

ndots:5 means any name with fewer than 5 dots is first tried with each search domain appended. So api.example.com (2 dots) is queried as api.example.com.default.svc.cluster.local, then ...svc.cluster.local, then ...cluster.local, and only then as itself. That is four queries for every external lookup — the standard explanation for slow DNS in Kubernetes and for inflated CoreDNS load.

A trailing dot makes a name fully qualified and skips the search list entirely: api.example.com. resolves in one query.

Step 3 - the chain from the authority down

dig +trace <name>
dig SOA <domain>
dig @<authoritative-ns> <name>
dig @8.8.8.8 <name>

+trace walks from the root servers down, showing which server delegates to which. It bypasses your resolver's cache and shows where the chain breaks.

Then compare the authority's answer against your resolver's:

Step 4 - TTL and negative caching

dig <name> | grep -A2 'ANSWER SECTION'
dig +nocmd +noall +answer <name>
watch -n5 'dig +noall +answer <name>'

The TTL is the second-to-last field in the answer and counts down as the cache ages. A TTL of 300 means up to five more minutes of the old value.

Negative caching is the part that surprises people. When a name does not exist, the NXDOMAIN is cached too — for the duration in the SOA minimum field, not the record's TTL:

dig SOA <domain> | grep -A1 'ANSWER SECTION'

The last number in the SOA record is the negative TTL. If it is 3600 and someone queried the name before it was created, that NXDOMAIN is cached for an hour. Querying a record before you create it is a genuinely common and very confusing self-inflicted outage.

Lower TTLs before a planned change, not during it — lowering a TTL is itself subject to the old TTL.

Step 5 - Kubernetes specifics

kubectl exec -it <pod> -- cat /etc/resolv.conf
kubectl exec -it <pod> -- nslookup kubernetes.default
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
kubectl top pods -n kube-system -l k8s-app=kube-dns

Common causes specific to clusters:

CoreDNS being throttled or OOMKilled is a cluster-wide outage that presents as every application failing simultaneously with unrelated-looking errors. Check its resources early when many services fail at once.

Decision Tree

Name does not resolve
├── Timeout (no response at all)
│   ├── Resolver unreachable → network, not DNS → linux-network-debugging
│   └── In Kubernetes → NetworkPolicy blocking port 53, or CoreDNS down
├── SERVFAIL
│   ├── dig +trace fails at a delegation → broken NS records
│   ├── DNSSEC validation failure       → check with +cd (checking disabled)
│   └── Upstream authority timing out   → the authoritative server
├── NXDOMAIN
│   ├── Authority also says NXDOMAIN → the record does not exist
│   ├── Authority has it, resolver does not → negative cache; check SOA minimum
│   └── Wrong name assembled → search domains and ndots
├── NOERROR but no answer section
│   └── Wrong record type (A vs AAAA vs CNAME)
├── Resolves to the wrong address
│   ├── Authority is correct → cache; wait out the TTL
│   └── Authority is wrong   → fix the record
└── Slow or intermittent
    ├── ndots amplification → use FQDNs with a trailing dot
    ├── One of several resolvers is bad → query each directly
    └── 5s spikes in Kubernetes → the UDP conntrack race

Common Causes

Safety

Read-only

dig, drill, host, resolvectl status, reading /etc/resolv.conf, and reading CoreDNS configuration and logs. DNS queries are read operations; dig +trace generates a few more of them than usual and is still harmless.

Potentially disruptive

• Changing a DNS record — takes effect for new lookups and then propagates for the length of the old TTL. This is one of the few changes that is not instantly reversible: reverting is itself subject to the TTL you just served. Lower the TTL well in advance of any planned change.
• Flushing a resolver cache (resolvectl flush-caches, restarting a caching resolver) — causes a burst of upstream queries. Safe on one host, potentially a thundering herd on a fleet.
• Restarting CoreDNS — every pod in the cluster loses resolution for the restart window. Kubernetes runs multiple replicas for this reason; verify more than one is ready before restarting any of them.
• Editing the CoreDNS ConfigMap — cluster-wide. A syntax error breaks DNS for everything. Keep the previous ConfigMap to hand, and change one directive at a time.
• Changing ndots or dnsConfig on a workload — requires a pod restart and can change which names resolve at all if code relies on search-domain behaviour.

Destructive

• Deleting a DNS record or a zone — removes name resolution for everything that depends on it, and the negative caching means the outage outlives the mistake by the SOA minimum. Required first: confirm nothing resolves the name (query logs, not assumption), export the zone, and get explicit confirmation. Deleting a zone is effectively irreversible within any useful timeframe.
• Deleting the CoreDNS Deployment or its ConfigMap — a full cluster outage: every pod loses resolution and most applications fail within seconds. Required first: a saved copy of the ConfigMap and a tested restore path.

DNS failures look total and tempt a "delete and recreate" response. That instinct is wrong here specifically because of caching: recreating a record does not undo the negative cache that the deletion populated.

Verification

Before: api.internal resolves to 10.0.4.12 (decommissioned); 100% connection timeouts
After:  api.internal resolves to 10.0.9.30 from all 3 resolvers and from inside the pod;
        0 timeouts over 200 requests

dig +short <name>
for r in $(awk '/^nameserver/{print $2}' /etc/resolv.conf); do
  echo "$r -> $(dig +short @"$r" <name>)"
done
kubectl exec -it <pod> -- getent hosts <name>

Verify from three positions, because each has its own cache:

1. The authority — the record is right at the source.
2. Each configured resolver — no stale cache, and all of them agree.
3. The original failing context — the pod, host or container that reported the problem.

Verifying only from your laptop proves nothing about the pod, and is the most common false confirmation in DNS work.

Anti-Patterns

Bad: ping <name> to test DNS. Better: ping hides the response code and may use a different resolution path (nsswitch, /etc/hosts, mDNS). dig shows you NXDOMAIN versus SERVFAIL, which is the entire diagnosis.

Bad: Change the record and wait for it to propagate. Better: DNS does not propagate; caches expire. The wait is exactly the old TTL, and it started when each resolver last queried. Lower the TTL before the change.

Bad: Create the record, then test it repeatedly until it works. Better: Querying it before it exists populates the negative cache for the SOA minimum. Check the authority directly instead of hammering the resolver.

Bad: Add an /etc/hosts entry to work around it. Better: That fixes one host and creates a permanent, invisible discrepancy that outlives everyone's memory of why it is there. It is a legitimate emergency mitigation — with a ticket to remove it.

Bad: Restart CoreDNS when DNS misbehaves. Better: That is a cluster-wide outage window. Check whether CoreDNS is actually at fault first: its own logs, CPU, and whether the failure is limited to one namespace.

Bad: Set ndots:1 everywhere to make DNS faster. Better: It does reduce query amplification, but it breaks any code relying on short-name resolution of in-cluster Services. Use fully qualified names with a trailing dot instead.

Composition

• kubernetes-networking — Services, NetworkPolicy, and the rest of the cluster network path.
• linux-network-debugging — when the resolver is unreachable rather than wrong.
• http-debugging — once names resolve and the problem is at the protocol layer.
• tls-debugging — certificate name mismatches, which look like DNS problems and are not.

References

• RFC 2308: Negative Caching of DNS Queries — the source of SOA-minimum negative caching behaviour.
• Kubernetes: DNS for Services and Pods — search domains, ndots, and dnsConfig.