What is a Pod Disruption Budget (PDB) and how can it block upgrades?
Quick Answer
A PDB defines the minimum availability an application must maintain during voluntary disruptions like node drains, upgrades, and autoscaler actions. If draining a node would violate the PDB, Kubernetes blocks the drain — which can stall cluster upgrades indefinitely.
Detailed Answer
Think of a PDB like a fire safety regulation for a building. The rule might say 'at least 2 of 3 emergency exits must remain open at all times.' During a renovation, workers can close one exit at a time. But if they try to close two simultaneously, the fire marshal blocks the work until one reopens. The regulation doesn't create more exits — it just prevents too many from being closed at once.
A Pod Disruption Budget tells Kubernetes: 'during voluntary disruptions, you must keep at least N pods of this application running (minAvailable) or take down at most M pods at a time (maxUnavailable).' Voluntary disruptions include node draining (kubectl drain), cluster upgrades, Cluster Autoscaler scale-down, and Karpenter consolidation. PDBs do NOT protect against involuntary disruptions like node crashes, OOM kills, or hardware failures — those bypass PDBs entirely.
Here's the mechanism: when Kubernetes needs to evict a Pod (during a node drain), it calls the Eviction API. The Eviction API checks all PDBs that select this Pod. If evicting it would cause the number of healthy Pods to drop below minAvailable (or the number of unavailable Pods to exceed maxUnavailable), the eviction is denied. The drain operation retries, but if the PDB condition never resolves, the drain hangs forever.
This is the #1 cause of stuck OpenShift upgrades. Real scenario: you have an application with 5 replicas and a PDB of minAvailable=5. During an upgrade, the MCO tries to drain a worker node. One of the 5 Pods is on that node. Evicting it would drop healthy Pods to 4, violating minAvailable=5. The drain is blocked. The node stays in SchedulingDisabled state. The MachineConfigPool shows UPDATING=True but never completes. The entire upgrade stalls waiting for this one node.
The fix depends on the situation: (1) if the PDB is too aggressive (minAvailable equals replica count), reduce it to allow at least one disruption, (2) if the application can scale up temporarily, increase replicas so there's headroom, (3) as a last resort in an emergency, delete the PDB temporarily, let the drain complete, then recreate it. Option 3 is risky because the application briefly loses PDB protection.
A critical interview point: PDBs do NOT create additional pods or nodes. They are purely a constraint — a gate that blocks eviction requests. They're essential for production workloads (preventing rolling updates from taking too many replicas offline), but must be carefully tuned relative to replica count. The rule of thumb: maxUnavailable should be at least 1, or minAvailable should be at most (replicas - 1).