Explain the complete lifecycle of a DNS query for a service in Kubernetes, from a pod making a request to the response being returned. What can go wrong at each step?
Quick Answer
The query goes through the container's /etc/resolv.conf → CoreDNS (or kube-dns) → cluster service lookup or upstream DNS. Failures can occur at ndots expansion, CoreDNS pod availability, or upstream resolution.
Detailed Answer
Full DNS Resolution Path
1. Application makes DNS query for my-service.my-namespace.svc.cluster.local 2. Container's /etc/resolv.conf is configured by kubelet with nameserver <CoreDNS ClusterIP>, search <namespace>.svc.cluster.local svc.cluster.local cluster.local, and ndots:5 3. ndots behavior: If the query has fewer than 5 dots, the resolver appends each search domain and tries them ALL before trying the original name. api.example.com generates 4 queries before the actual one. 4. CoreDNS receives the query via the ClusterIP service (kube-dns), which load balances to CoreDNS pods 5. CoreDNS checks its plugins: kubernetes plugin for cluster-internal names, forward plugin for external names 6. For internal services: CoreDNS queries the Kubernetes API (via watch cache) to resolve Service → ClusterIP or Endpoints 7. Response returned through the chain back to the pod
What Can Go Wrong
- ndots:5 causes 5x DNS amplification for external domains — massive performance hit - CoreDNS pods overloaded: Too few replicas, leading to timeouts (default 5s) and retries - conntrack table full: High DNS QPS can exhaust conntrack entries on nodes, causing silent drops - Race condition (CVE-2017-15129): UDP source port reuse causing intermittent SERVFAIL on some kernel versions - CoreDNS cache TTL: Stale records after service updates if cache TTL is too aggressive
Production at Scale
Use NodeLocal DNSCache (node-level DNS caching daemonset) to reduce CoreDNS load and avoid conntrack issues. Set ndots:2 for pods that primarily call external services. Monitor CoreDNS with coredns_dns_requests_total and coredns_dns_responses_total Prometheus metrics.
Code Example
# Check pod DNS configuration
kubectl exec -it <pod> -- cat /etc/resolv.conf
# Test DNS resolution with timing
kubectl exec -it <pod> -- nslookup -debug my-service.default.svc.cluster.local
# Check CoreDNS pods and logs
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Override ndots in pod spec
spec:
dnsConfig:
options:
- name: ndots
value: "2"Interview Tip
This question separates senior from staff engineers. The key insight is ndots:5 causing DNS amplification — most candidates miss this. Also mention conntrack exhaustion and NodeLocal DNSCache as the production solution.