How can I find frequently restarting pods in a Kubernetes namespace using Python?
Quick Answer
Use the Kubernetes Python client to list pods in a namespace, iterate through each pod's container statuses, and report any container where restart_count exceeds a defined threshold. This surfaces CrashLoopBackOff and intermittent OOMKill patterns in seconds instead of manually describing dozens of pods one at a time.
Detailed Answer
This is like a hospital triage nurse walking through a ward not to check every patient's full chart, but specifically to spot anyone whose vital signs monitor has alarmed repeatedly — a single alarm might be a fluke, but a patient whose monitor has alarmed five times in an hour needs attention immediately, and the nurse doesn't need to read every chart in detail to know which bed to go to first.
This script exists because kubectl get pods shows a restart count column, but scanning that visually across a namespace with hundreds of pods during an incident is slow and error-prone, especially when the unstable pods are scattered rather than clustered. Automating the threshold check turns a manual visual scan into an instant, repeatable triage step — this matters most during incidents, when the fastest path to root cause is often "which of these 200 pods is actually the unstable one," not a general health overview.
The script authenticates via config.load_kube_config(), reading the local kubeconfig, though load_incluster_config() would be used if this ran as a pod itself, then calls list_namespaced_pod to get every pod object in the target namespace. Each pod's status.container_statuses is a list, one entry per container in the pod, that can be None if the pod is still being scheduled or its images are still pulling, which is why the script explicitly guards against that case before iterating. For each container status, restart_count is a cumulative counter that Kubernetes maintains for the life of the pod object, resetting to zero only if the pod itself is deleted and recreated, not on a simple container restart, so a high restart count is a durable signal of instability rather than something that self-clears.
In production, this kind of check is typically run both proactively, as a daily cluster health scan feeding a dashboard, and reactively, with an on-call engineer running it directly during triage to shortcut past kubectl describe on dozens of pods. What matters operationally is combining restart count with the reason for the last termination, terminated.reason such as OOMKilled versus Error, since a high restart count alone doesn't tell you whether the fix is a memory limit increase, a code bug, or a missed dependency — the threshold script is a triage filter, not a root-cause diagnosis by itself.
The non-obvious gotcha is that restart count resets to zero when a pod is deleted and recreated, for example during a rolling deployment or when a node is replaced, so a genuinely unstable workload that's been crash-looping for days can show a deceptively low restart count immediately after its most recent pod recreation, hiding the real instability pattern from a single point-in-time check. Engineers who rely purely on restart count without also checking pod age can be misled into thinking a chronic problem is resolved right after a deployment simply reset the counter, not because anything was actually fixed.