Why do Kubernetes pods get evicted, and what commands and steps do you use to diagnose and resolve pod eviction?
Quick Answer
Pods are evicted when a node is under resource pressure — disk (DiskPressure), memory (MemoryPressure), or PID exhaustion (PIDPressure). The kubelet evicts pods based on QoS class priority: BestEffort first, then Burstable, then Guaranteed last. Diagnosis starts with kubectl describe node to check conditions and kubectl get events to find eviction reasons.
Detailed Answer
Think of an overcrowded bus. When the bus exceeds its weight limit, the driver must ask some passengers to leave. Passengers without tickets (BestEffort pods) are asked first, then those with partial tickets (Burstable pods), and finally full-fare passengers (Guaranteed pods) are the last to go. The bus driver does not choose randomly — there is a clear priority order based on who has the strongest claim to stay.
In Kubernetes, pod eviction is the kubelet's mechanism for protecting node stability. When a node runs low on a critical resource — memory, temporary (ephemeral) storage, or process IDs — the kubelet begins evicting pods to reclaim that resource. This is different from preemption (which is the scheduler removing lower-priority pods to make room for higher-priority ones) and different from API-initiated eviction (which is used during node drain for maintenance).
Internally, the kubelet monitors resource usage against configurable eviction thresholds. The default soft eviction threshold for memory is memory.available < 100Mi, and for disk is nodefs.available < 10%. When a threshold is breached, the kubelet sets the corresponding node condition (MemoryPressure, DiskPressure, PIDPressure) and begins ranking pods for eviction. The ranking uses QoS class: BestEffort pods (no resource requests or limits) are evicted first, Burstable pods (requests set but lower than limits) are evicted next based on how much they exceed their requests, and Guaranteed pods (requests equal limits for all containers) are evicted last.
At production scale, the most common eviction cause is ephemeral storage exhaustion from container logs, emptyDir volumes, or container writable layers growing unbounded. Memory-based evictions happen when applications have memory leaks or when resource limits are set too low for actual workload requirements. Teams should monitor node conditions, set appropriate resource requests and limits to ensure critical pods get Guaranteed QoS, configure log rotation to prevent disk pressure, and use PodDisruptionBudgets to limit the impact of evictions on service availability.
The non-obvious gotcha is that eviction thresholds have both soft and hard variants. Soft evictions give pods a grace period to terminate cleanly, while hard evictions kill pods immediately. If the hard eviction threshold is hit (e.g., memory.available < 50Mi), the kubelet kills pods without waiting for graceful shutdown, which can cause data loss or incomplete request processing. Architects should ensure hard thresholds are never reached by setting soft thresholds with enough buffer.