Pods intermittently see ~5-second DNS lookup delays in a large Kubernetes cluster running CoreDNS. Walk through how you'd diagnose and fix it.
Quick Answer
This is almost always the glibc/musl DNS resolver issuing parallel A and AAAA queries that race against a conntrack SNAT/DNAT race condition for UDP packets to the same destination, causing one of the two queries to be dropped and forcing a 5-second timeout-and-retry. Diagnose with packet captures and conntrack stats, fix with weave's `weave-fix`/`ndots` reduction, conntrack table tuning, or just switching to TCP for DNS / using NodeLocal DNSCache.
Detailed Answer
The classic '5-second DNS delay' in Kubernetes has a well-known root cause: when an application does a hostname lookup and ndots:5 is set (Kubernetes' default in /etc/resolv.conf), glibc fires off A and AAAA queries nearly simultaneously over UDP from the same source port. Both queries get SNAT'd by the same conntrack entry-creation path on the node, and if two UDP packets attempt to insert conntrack entries at the exact same time, one can fail with an EEXIST race condition and gets silently dropped. The client then waits for the OS-level UDP timeout (5 seconds) before retrying.
Diagnosis steps
1. Confirm the pattern: exactly ~5s delays, intermittent, not constant — this rules out CoreDNS being generally overloaded (check that separately via CoreDNS metrics/cache hit rate). 2. tcpdump on the node for port 53 during a reproduction, look for a query with no matching response. 3. Check conntrack insert_failed counters: conntrack -S — non-zero insert_failed confirms the race. 4. Check CoreDNS itself isn't the bottleneck separately: coredns_dns_request_duration_seconds and cache hit ratio in Prometheus. 5. Check ndots and search domain expansion — every short hostname lookup expands into multiple FQDN attempts (e.g., 5 search-domain variants) before falling through to the absolute name, multiplying the number of UDP queries fired per lookup.
Fixes, roughly in order of how production teams actually apply them: deploy NodeLocal DNSCache (runs a DNS cache as a DaemonSet on each node, removing the SNAT/conntrack race for queries answered from cache and reducing CoreDNS load); reduce ndots to 2 in pod specs for workloads making fully-qualified external calls; increase net.netfilter.nf_conntrack_max and tune nf_conntrack_udp_timeout if the table itself is near capacity; and in extreme cases force DNS over TCP to avoid the UDP race entirely.
Code Example
# Confirm conntrack insert races
conntrack -S | grep insert_failed
# Check CoreDNS resource pressure / cache behavior
kubectl -n kube-system top pod -l k8s-app=kube-dns
curl -s localhost:9153/metrics | grep -E 'coredns_dns_request_duration_seconds|coredns_cache_hits_total'
# Reduce ndots for a workload doing mostly external DNS lookups
dnsConfig:
options:
- name: ndots
value: "2"
# Deploy NodeLocal DNSCache (removes the conntrack race path)
curl -s https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml | sed 's/__PILLAR__DNS__SERVER__/<kube-dns-cluster-ip>/g' | kubectl apply -f -Interview Tip
This question lives or dies on whether you name the actual mechanism: the UDP conntrack race between simultaneous A/AAAA queries, not a vague 'CoreDNS was slow.' Citing `conntrack -S` insert_failed and NodeLocal DNSCache as the fix is the signal interviewers are listening for — it shows you've actually debugged this exact incident before.