How do you use kubectl to debug a pod that is not starting or responding?
Quick Answer
Start with `kubectl describe pod` to read Events (scheduling failures, image pull errors, probe failures). Check `kubectl logs` for application crashes. Use `kubectl get pod -o yaml` to see status conditions and container state. For running-but-unresponsive pods, use `kubectl exec` to inspect the container or deploy an ephemeral debug container.
Detailed Answer
Think of debugging a non-starting pod like diagnosing why a car won't start. You follow a systematic order: is there fuel (image available)? Does the engine turn over (container starts)? Does it stall after starting (crash loop)? Is it running but won't move (readiness probe failing)? Each stage has different diagnostic tools and different root causes.
The debugging workflow follows the pod lifecycle phases. Phase 1 - Pending: The pod has not been scheduled to a node. Run kubectl describe pod <name> and check the Events section at the bottom. Common failures: 'Insufficient cpu/memory' (no node has enough allocatable resources -- check node capacity with kubectl describe nodes), 'no nodes available to schedule pods' (taints without matching tolerations), '0/10 nodes are available: 10 pod has unbound PersistentVolumeClaims' (PVC not bound). The fix depends on the failure: add nodes, adjust requests, fix taints, or provision storage.
Phase 2 - ImagePullBackOff: The container runtime cannot pull the image. Events show 'Failed to pull image' with reasons like authentication failure (wrong imagePullSecret), image not found (typo in tag), or registry unreachable (network policy blocking egress to registry). Debug with kubectl describe pod and check imagePullSecrets in the pod spec.
Phase 3 - CrashLoopBackOff: The container starts and immediately exits. Run kubectl logs <pod> --previous to see the output from the last crashed instance. Common causes: missing environment variables the app requires, database connection failures, misconfigured command/args in the pod spec, or the container exiting with a non-zero exit code. The --previous flag is critical because the current container may have already crashed and been replaced.
Phase 4 - Running but not Ready: The container is running but the readiness probe is failing. Check kubectl describe pod for 'Unhealthy' events showing which probe fails and the HTTP response or exec output. The pod will not receive Service traffic while not Ready. Debug by exec-ing into the pod and manually running the probe command: kubectl exec <pod> -- curl localhost:8080/healthz.
Phase 5 - Running and Ready but not responding to external traffic: The issue is network-level. Check the Service endpoints (kubectl get endpoints <svc> -- is the pod listed?), check NetworkPolicies blocking ingress, and verify the container is listening on the correct port (kubectl exec <pod> -- netstat -tlnp).
For containers that crash too quickly to exec into, use ephemeral debug containers: kubectl debug <pod> --image=busybox --target=<container> attaches a debug container that shares the pod's process namespace, letting you inspect the filesystem and network without the application needing to be running.