How do you troubleshoot when a Kubernetes Service is not reachable?

Think of debugging an unreachable Service like troubleshooting a broken phone extension in an office. If you dial extension 5001 and nobody answers, you check: Is extension 5001 actually programmed in the phone system (does the Service exist)? Is it routed to the right desk (do selectors match labels)? Is anyone sitting at that desk (are Pods running)? Is the person at the desk taking calls (readiness probe passing)? Is the phone line physically connected (NetworkPolicy allowing traffic)? And is the internal phone directory working (DNS resolution)? You work from the outermost layer inward.

Step 1 — Verify the Service exists and has the correct configuration. Run kubectl get svc <name> to confirm the Service exists, check its type, and note the ClusterIP and port. Then run kubectl describe svc <name> to see the Selector field. A common mistake is typos in selectors: if the Service selects app: payment-api (singular) but Pods have the label app: payments-api (plural), the Service will never find any Pods.

Step 2 — Check Endpoints. Run kubectl get endpoints <service-name>. If the Endpoints list is empty, the Service found no matching Pods. This means either no Pods with matching labels exist, or Pods exist but aren't passing their readiness probe. Run kubectl get pods -l <selector-labels> to check if Pods with the right labels exist. If Pods exist but Endpoints are empty, check their readiness: kubectl describe pod <pod> and look at the Conditions section — Ready must be True. If the readiness probe is failing, the Endpoint controller removes the Pod from the Service's Endpoints, even though the Pod is Running.

Step 3 — Test DNS resolution. From a debug Pod, run nslookup <service-name>.<namespace>.svc.cluster.local. If DNS doesn't resolve, CoreDNS might be unhealthy — check kubectl get pods -n kube-system -l k8s-app=kube-dns. If DNS resolves but the connection times out, the issue is in the routing layer. Common DNS pitfalls: the client Pod uses a shortened service name (payments-api) which only works if the client is in the same namespace, or the ndots:5 setting in /etc/resolv.conf causes excessive DNS queries before finding the right suffix.

Step 4 — Check NetworkPolicies. If your cluster uses NetworkPolicies (Calico, Cilium, or any CNI that supports them), a policy might be blocking traffic. Run kubectl get networkpolicies -n <namespace> to see if any policies exist. NetworkPolicies are deny-by-default when present: if ANY NetworkPolicy selects a Pod, all traffic not explicitly allowed by some policy is blocked. A common mistake: adding a NetworkPolicy that allows ingress from the frontend namespace but forgetting to allow ingress from the monitoring namespace, breaking health checks.

Step 5 — Verify kube-proxy and node networking. If all of the above checks pass, the issue might be at the node networking level. Check if kube-proxy is running: kubectl get pods -n kube-system -l k8s-app=kube-proxy. On the node, verify iptables rules exist for your Service: sudo iptables -t nat -L KUBE-SERVICES | grep <service-clusterip>. If rules are missing, kube-proxy may not be syncing — check its logs for errors.

The production gotcha that causes the most subtle outages: readiness probes that depend on external services. If your payments-api readiness probe checks whether it can reach the database, and the database has a brief hiccup, ALL Pods simultaneously fail their readiness probes. The Endpoint controller removes all Pods from the Service. Now you have a complete outage — not because your application crashed, but because the readiness probe is too aggressive. Readiness probes should check 'can I handle requests?' not 'are all my dependencies healthy?'