How do Kubernetes Services route traffic to pods using selectors and endpoints?

Think of a Kubernetes Service like a restaurant's main phone number. When customers call that number, a receptionist (kube-proxy) routes the call to whichever waiter (pod) is currently available. The restaurant doesn't give customers direct waiter phone numbers because waiters come and go with shift changes. The main number stays constant while the routing behind it updates automatically.

A Service is an abstraction that provides a stable network identity (ClusterIP, DNS name) for a set of pods. When you create a Service with a selector like app: payments-api, the endpoints controller (part of kube-controller-manager) watches for pods matching that label and creates an Endpoints resource listing their IP:port combinations. This Endpoints object updates automatically as pods are created, deleted, or become not-Ready.

Internally, the flow works like this: when a client pod sends a packet to the Service ClusterIP (e.g., 10.96.45.12:8080), the packet hits the node's network stack. kube-proxy has programmed iptables rules (or IPVS virtual servers) that intercept packets destined for that ClusterIP. These rules perform DNAT (Destination NAT), rewriting the destination IP from the Service ClusterIP to one of the pod IPs listed in the Endpoints object. The selection uses random probability by default (iptables mode) or round-robin/least-connections (IPVS mode). The return traffic from the pod is SNATed back so the client sees responses coming from the Service IP, not the pod IP directly.

In production, readiness probes are critical for this flow. A pod is only added to the Endpoints list when its readiness probe passes. If a pod's readiness probe fails, the endpoints controller removes it from the Endpoints object within seconds, and kube-proxy updates its rules to stop sending traffic there. This is why a missing or misconfigured readiness probe is dangerous: a pod that is starting up but not ready to serve traffic will receive requests and return errors.

A common gotcha: if your Service selector does not match any pod labels, the Endpoints object will be empty and all requests will fail. This is the number-one cause of 'Service not working' issues. Another subtle issue: headless Services (clusterIP: None) skip kube-proxy entirely and return pod IPs directly via DNS, which is used by StatefulSets for stable network identities.