How does CoreDNS work internally in Kubernetes and what DNS records are created?

Think of CoreDNS as the telephone directory operator for your Kubernetes city. Every time a new business (Service) opens, the operator automatically adds its name and phone number to the directory. When someone (a Pod) wants to call a business, they dial the operator first, who looks up the name and returns the correct number. Without the operator, every resident would need to memorize phone numbers (IP addresses) that might change every time a business relocates.

In Kubernetes, CoreDNS is the cluster DNS server that replaces the older kube-dns. It runs as a Deployment (typically 2 replicas for HA) in the kube-system namespace, exposed by a Service with a well-known ClusterIP (usually 10.96.0.10 or the tenth IP in the service CIDR). Every Pod created in the cluster has its /etc/resolv.conf configured to point to this ClusterIP as the nameserver, with search domains like namespace.svc.cluster.local, svc.cluster.local, and cluster.local. This means a Pod in the orders namespace can reach the payments Service by querying just 'payments' if it is in the same namespace, or 'payments.payments' for cross-namespace resolution.

Internally, CoreDNS watches the Kubernetes API server for Service and Endpoint changes using the kubernetes plugin in its Corefile configuration. When a ClusterIP Service is created, CoreDNS generates an A record mapping service-name.namespace.svc.cluster.local to the ClusterIP. For headless Services (clusterIP: None), CoreDNS creates A records for each backing Pod's IP address, allowing direct Pod-to-Pod communication. SRV records are created for named ports: _port-name._protocol.service-name.namespace.svc.cluster.local. For StatefulSets with headless Services, individual Pod DNS records are created: pod-name.service-name.namespace.svc.cluster.local. Pod A records (when enabled) use a dashed IP format: 10-244-1-5.namespace.pod.cluster.local.

CoreDNS processes queries through a plugin chain defined in the Corefile. The typical chain includes: the kubernetes plugin (for in-cluster resolution), the forward plugin (forwarding external queries to upstream DNS like 8.8.8.8 or node-level resolv.conf), the cache plugin (caching responses with configurable TTL), the loop plugin (detecting forwarding loops), the health and ready plugins (for liveness and readiness probes), and the prometheus plugin (exposing metrics). Each query enters the plugin chain and is processed in order until a plugin handles it. The cache plugin is critical: it stores successful responses for 30 seconds by default and negative responses (NXDOMAIN) for 5 seconds, dramatically reducing load on the API server.

A non-obvious gotcha is the ndots:5 setting in Pod resolv.conf. With ndots:5, any hostname with fewer than 5 dots triggers a search through all search domains before trying the absolute name. So a query for 'api.stripe.com' (2 dots, less than 5) first tries api.stripe.com.orders.svc.cluster.local, then api.stripe.com.svc.cluster.local, then api.stripe.com.cluster.local, and finally api.stripe.com -- that is four extra DNS queries for every external hostname. At scale, this generates enormous DNS traffic. The fix is to append a trailing dot (api.stripe.com.) for external hostnames or reduce ndots to 2 in the Pod's dnsConfig. Another gotcha: CoreDNS default cache does not distinguish between internal and external queries, so a thundering herd of DNS requests for an external domain after cache expiry can overwhelm the upstream resolver.