How does Kubernetes DNS work, and how do pods discover services by name?
Quick Answer
CoreDNS runs as a Deployment in kube-system and watches the Kubernetes API for Services and Endpoints. It creates DNS records mapping Service names to ClusterIPs. Pods are configured with CoreDNS as their nameserver via /etc/resolv.conf, allowing them to resolve service-name.namespace.svc.cluster.local to the Service ClusterIP.
Detailed Answer
Think of Kubernetes DNS like a building's internal phone directory. Instead of memorizing every department's extension number (IP address), employees dial the department name ('accounting') and the PBX system (CoreDNS) translates it to the right extension. When departments move offices (pods reschedule), the directory updates automatically so callers always reach the right place without changing their speed-dial.
Kubernetes runs CoreDNS as the cluster DNS server, typically deployed as a Deployment with 2 replicas in the kube-system namespace, fronted by a Service with a well-known ClusterIP (usually 10.96.0.10). When a pod is created, the kubelet configures its /etc/resolv.conf to point to this CoreDNS Service IP as the nameserver, with search domains that allow short-name resolution.
The DNS resolution hierarchy works as follows: For a Service named payments-api in namespace production, CoreDNS creates an A record at payments-api.production.svc.cluster.local pointing to the Service's ClusterIP. Pods in the same namespace can use the short name payments-api because the search domains in /etc/resolv.conf include production.svc.cluster.local. Pods in other namespaces use payments-api.production or the full FQDN. For headless Services (clusterIP: None), CoreDNS returns the individual pod IPs instead of a single ClusterIP, which is how StatefulSets provide stable DNS names for each pod (e.g., postgres-0.postgres-headless.database.svc.cluster.local).
Internally, CoreDNS watches the Kubernetes API server for Service and Endpoints changes using its kubernetes plugin. When a new Service is created or Endpoints change, CoreDNS updates its in-memory record set within seconds. It also handles external DNS resolution: if a pod queries for api.stripe.com, CoreDNS forwards the request to upstream nameservers configured in its Corefile (typically the node's DNS or a cloud provider's resolver like 169.254.169.253 on AWS).
In production, DNS is one of the most common sources of mysterious failures. Issues include: (1) ndots:5 default causing external domain lookups to try 5 internal suffixes first (e.g., api.stripe.com.production.svc.cluster.local before the real domain), adding latency for external calls. (2) CoreDNS pod resource exhaustion under high query volume causing timeouts. (3) DNS cache TTL mismatches when Services are recreated with new ClusterIPs but clients cache the old IP. The ndots issue is particularly insidious: reducing ndots from 5 to 2 in the pod's dnsConfig can cut external DNS resolution time by 80% for applications making many external API calls.