What causes a Kubernetes node to show NotReady, and how do you diagnose it?

Think of a node going NotReady like an employee who stops responding to their walkie-talkie. The manager (control plane) sends check-ins every few seconds, and the employee (kubelet) is supposed to respond. If 40 seconds pass with no response, the manager assumes something is wrong and marks the employee as unavailable on the schedule. The employee might be fine but just have a dead radio battery (network issue), or they might have actually collapsed (kubelet crash), or they might be overwhelmed handling an emergency and can't respond (resource pressure).

The kubelet sends node status updates to the API Server at a configurable interval (default: 10 seconds via node-status-update-frequency). In modern Kubernetes (1.17+), it also maintains a Lease object in the kube-node-lease namespace that acts as a lightweight heartbeat — this reduces etcd load compared to updating the full Node object. The Node controller in the kube-controller-manager monitors these heartbeats. If a node hasn't updated in 40 seconds (node-monitor-grace-period), the controller sets the node's Ready condition to Unknown. After 5 more minutes (pod-eviction-timeout), it begins evicting Pods from that node by marking them for deletion.

Disk Pressure triggers when available disk space drops below a threshold (default: 15% free or 10% free depending on the eviction signal). The kubelet starts evicting Pods with the highest disk usage — first ephemeral storage consumers, then by priority. This commonly happens when container logs fill up the disk (applications logging at DEBUG level without rotation), or when images and unused containers accumulate (lack of garbage collection). Memory Pressure triggers when available memory drops below 100Mi (default). The kubelet evicts Pods in QoS order: BestEffort first (no resource requests), then Burstable (partial requests), and Guaranteed last (requests equal limits). PID Pressure means the node is running out of process IDs — each container can spawn many processes, and the default Linux PID limit (32768) can be exhausted by applications that fork excessively.

Container runtime issues are sneaky because the kubelet process itself may be healthy and reporting heartbeats, but the runtime socket is unresponsive. Pods on the node will be stuck in ContainerCreating or Unknown state. You'll see the kubelet reporting errors like 'PLEG is not healthy' (PLEG = Pod Lifecycle Event Generator — the component that syncs container runtime state with kubelet's internal state). If PLEG hasn't completed a full relist within 3 minutes, kubelet marks itself NotReady. This is often caused by a hung containerd process, a node with hundreds of Pods causing slow container listing, or an exhausted inotify watch limit.

Network connectivity loss between the node and the API Server causes the most confusing NotReady scenarios. The node is actually perfectly healthy — all Pods are running fine, serving traffic locally — but the control plane can't reach it. After the grace period, the control plane marks the node NotReady and starts the eviction timer. If your cloud provider's load balancer health checks are tied to node condition, the load balancer removes the node from its target group, causing real traffic loss even though the applications were healthy. This is why production clusters use node-level monitoring (such as node_exporter with Prometheus) independent of Kubernetes to detect whether a node is actually down versus just disconnected from the API Server.

The most dangerous gotcha: when a node goes NotReady due to a network partition, the control plane eventually evicts its Pods and schedules replacements elsewhere. But the original Pods may still be running on the partitioned node (they don't know they're evicted). If the network heals, you can briefly have duplicate instances of stateful applications — split-brain. This is why StatefulSets have strict Pod identity guarantees and why storage systems use fencing to prevent dual-writes.