ExternalDNS Troubleshooting

Diagnose failures in ExternalDNS — the controller that synchronises Kubernetes Service/Ingress annotations to DNS providers (Route53, Azure DNS, CloudFlare, Google Cloud DNS, etc.).

Keywords

external-dns, externaldns, dns, records, route53, azure-dns, cloudflare, google-cloud-dns, dns-sync, dns-records, a-record, cname, txt-record, annotation, hostname, domain, zone, provider, registry, ownership

When to Use This Skill

• DNS records are not being created for new Services or Ingresses
• Existing DNS records are stale, pointing to old IPs
• ExternalDNS logs show authentication, permission, or zone errors
• TXT ownership records are missing or conflicting
• Records appear in the log as "planned" but never created
• DNS propagation seems broken after a provider migration
• ExternalDNS is running but doing nothing (no changes detected)

When NOT to Use

• DNS records exist but TLS certs are failing → use cert-manager-troubleshooting
• DNS is working but the Service has no endpoints → use k8s-namespace-troubleshooting
• ExternalDNS config is delivered by Flux and not appearing → use flux-troubleshooting

Related Skills

• cert-manager-troubleshooting - TLS certificates depend on DNS
• kyverno-troubleshooting - Policies may block annotations
• external-secrets-troubleshooting - Provider credentials from external stores
• k8s-namespace-troubleshooting - General namespace diagnosis
• k8s-platform-operations - Cluster-wide operations
• flux-troubleshooting - GitOps delivery of ExternalDNS config

Quick Reference

Diagnostic Workflow

DNS record not appearing?
├─ ExternalDNS pod running?
│   ├─ No → Check deployment, image, RBAC (Section 1)
│   └─ Yes → Check logs for errors
│       ├─ Authentication/permission error → Fix provider credentials (Section 2)
│       ├─ Zone not found / filtered out → Fix domain filter config (Section 3)
│       ├─ Source yielded 0 endpoints → Fix annotations/sources (Section 4)
│       ├─ Record planned but not created → Provider API issue (Section 5)
│       └─ No log activity at all → Check interval and RBAC (Section 6)
├─ Record exists but wrong value?
│   ├─ TXT ownership record missing → Ownership conflict (Section 7)
│   └─ Multiple ExternalDNS instances → Registry conflict (Section 7)
└─ Record was deleted unexpectedly?
    └─ Orphaned record cleanup or ownership loss (Section 7)

Section 1: Controller Health

# Pod status
kubectl get pods -n external-dns -o wide
kubectl describe deploy/external-dns -n external-dns

# RBAC — ExternalDNS needs read access to Services, Ingresses, Nodes
kubectl auth can-i list services --as=system:serviceaccount:external-dns:external-dns
kubectl auth can-i list ingresses --as=system:serviceaccount:external-dns:external-dns

# Check arguments and configuration
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n'

Key Arguments to Verify

Section 2: Provider Authentication

Full provider auth reference: See provider-authentication.md for generic diagnostic steps, auth method tables, and provider-specific issue matrices for AWS, Azure, GCP, Vault, and CloudFlare.

The commands below are ExternalDNS-specific. For general auth debugging patterns, use the shared reference.

# Quick auth check — grep ExternalDNS logs for auth failures
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -iE 'credential|auth|forbidden|access denied|assume role|sts|azure|authorization|cloudflare|google|permission'

Required Provider Permissions (ExternalDNS-Specific)

Section 3: Domain Filter and Zone Issues

# Check which domains ExternalDNS is configured to manage
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -i domain

# Check if the target zone exists at the provider
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -iE 'zone|domain|filter'

# List all zones ExternalDNS can see
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -i "all hosted zones"

Section 4: Source and Annotation Issues

# Check which sources ExternalDNS watches
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep source

# Find Services with external-dns annotations
kubectl get svc -A -o json | jq -r '.items[] | select(.metadata.annotations["external-dns.alpha.kubernetes.io/hostname"] != null) | "\(.metadata.namespace)/\(.metadata.name)\t\(.metadata.annotations["external-dns.alpha.kubernetes.io/hostname"])"'

# Find Ingresses with hostnames
kubectl get ingress -A -o json | jq -r '.items[] | select(.spec.rules) | .metadata.namespace + "/" + .metadata.name + "\t" + (.spec.rules[].host // "no-host")'

# Check if source yields endpoints
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -i "endpoint"

Required Annotations (Services)

Service Type Requirements

Common issue: LoadBalancer Service has no .status.loadBalancer.ingress yet — ExternalDNS cannot determine the target IP. Check if the cloud load balancer provisioned successfully.

Section 5: Record Creation Failures

# Check for planned vs applied changes
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'planned|create|update|delete|change|apply'

# Check for rate limiting
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'throttl|rate limit|429|too many'

# Check for provider API errors
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'error|fail|invalid|conflict'

Section 6: No Activity

# Check sync interval
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep interval

# Check if watching the right namespace
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep namespace

# Check RBAC can list resources
kubectl auth can-i list services -A --as=system:serviceaccount:external-dns:external-dns
kubectl auth can-i list ingresses -A --as=system:serviceaccount:external-dns:external-dns

# Recent log entries (any activity at all?)
kubectl logs -n external-dns deploy/external-dns --tail=50 --since=10m

Section 7: Ownership and Registry

ExternalDNS uses TXT records as an ownership registry to track which records it manages. Problems here cause records to be ignored, duplicated, or accidentally deleted.

# Check registry type and owner ID
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -iE 'registry|owner'

# Look for ownership conflicts in logs
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'ownership|registry|txt|skip|ignore|conflict'

Provider-Specific Gotchas

MCP Tools Available

When the appropriate MCP servers are connected, prefer these over raw kubectl where available:

• mcp__flux-operator-mcp__get_kubernetes_resources - Query ExternalDNS deployment, pods, services, ingresses
• mcp__flux-operator-mcp__get_kubernetes_logs - Retrieve ExternalDNS pod logs
• mcp__flux-operator-mcp__get_kubernetes_metrics - Check ExternalDNS resource consumption

Common Mistakes

Behavioural Guidelines

1. Check logs first — ExternalDNS is heavily log-driven; the answer is almost always in the logs.
2. Verify annotations — Most "ExternalDNS isn't working" issues are missing or malformed annotations.
3. Check the provider console — Confirm whether the record exists at the provider, not just via DNS lookup (caching).
4. Never expose provider credentials — List secret names, never decode values.
5. Understand the policy — upsert-only vs sync has dramatically different behaviour. Confirm which is set before troubleshooting deletions.
6. Check owner IDs in multi-cluster setups — Ownership conflicts are silent; records just stop updating.
7. Restart as a last resort — A rollout restart forces a full sync cycle, but diagnose the root cause first.