Walk through your troubleshooting methodology when a user reports a Kubernetes service is down.

Think of debugging a dropped phone call. You could blame the phone, but the issue might be the SIM card, the cell tower, the network backbone, or the destination carrier. A systematic technician tests each link in the chain from the caller to the receiver. In Kubernetes, a user report of service down could mean failure at any of ten distinct layers, and guessing wastes precious minutes during an outage.

The senior troubleshooting formula follows the full request path: Application, Container, Pod, Service, Endpoints, Ingress, Load Balancer, DNS, Network, Cloud Infrastructure. Starting at the application layer, check if the process inside the container is running and responding. Inspect logs for uncaught exceptions, OOMKills, configuration errors, or dependency failures. At the container layer, check if the container started successfully, whether liveness and readiness probes are passing, and if there are restart loops. Container state shows Waiting (image pull failure, CrashLoopBackOff), Running, or Terminated (exit code reveals the reason).

At the pod layer, check scheduling status: is the pod Pending (insufficient resources, node affinity, taints), Running but not Ready (failing readiness probe), or Evicted (node pressure)? Describe the pod to see events like FailedScheduling, FailedMount, or BackOff. At the Service layer, verify the Service selector matches pod labels exactly and the Service port maps to the correct container port. At the Endpoints layer, confirm the Service has populated endpoints — empty endpoints is one of the most common causes of 503 errors that looks like an application problem but is actually a configuration issue.

At the Ingress layer, check that host and path rules are correct, the controller has synced (Address field populated), and TLS secrets are valid. At the Load Balancer layer, verify health check paths, target group registration, and security groups. At the DNS layer, confirm the domain resolves to the correct endpoint and TTLs have propagated. At the Network layer, check NetworkPolicies, security groups, NACLs, and pod-to-pod connectivity. At the Cloud Infrastructure layer, check node health, API rate limits, IAM permissions, and service quotas.

In production, this layered approach is critical because symptoms at one layer often originate from another. A 503 at the load balancer might be caused by empty endpoints, which is caused by failing readiness probes, which is caused by a database connection timeout, which is caused by a security group change. Following the chain systematically reveals the root cause in minutes rather than hours. Experienced engineers also parallelize: one person checks the application path while another checks infrastructure, then they compare findings.

The non-obvious gotcha is confirmation bias. Engineers see a recent deployment and assume the deploy caused the issue, but correlation is not causation. The deploy might be coincidental while the real cause is a cloud provider networking issue, a certificate expiration, or a resource quota being hit. The troubleshooting formula forces you to verify evidence at each layer rather than jumping to conclusions. Document what you checked, what you found, and what you ruled out — this prevents circular debugging during stressful incidents.