How does ZooKeeper achieve consensus during a leader election, and why does a 4-node ensemble actually tolerate fewer failures than a 3-node one?

Picture a corporate board that requires a strict majority vote to approve any decision. A 3-person board needs 2 votes to pass anything and can survive 1 member being unreachable. A 4-person board still needs a majority — which is 3 out of 4, not 2 — so it can also only survive 1 member being unreachable, despite having a whole extra person on the payroll contributing nothing to how many absences the board can tolerate. Only when you go to 5 people does the board gain the ability to survive 2 absences.

ZooKeeper is designed around this exact majority-quorum math because it needs strong consistency guarantees for the coordination primitives (locks, leader election, configuration) that distributed systems like Kafka and HBase build on top of it. ZAB (ZooKeeper Atomic Broadcast) is the underlying protocol that ensures every write is agreed upon by a majority of the ensemble before being considered committed, and that exactly one node acts as leader accepting writes at any time — this was designed deliberately as a crash-recovery atomic broadcast protocol rather than a more general Byzantine-fault-tolerant one, trading off tolerance of malicious nodes (which ZooKeeper doesn't defend against) for a simpler, faster protocol suited to trusted internal infrastructure.

During leader election, each node proposes itself as leader and broadcasts its zxid (ZooKeeper transaction ID, a monotonically increasing number where the high bits encode the current epoch and the low bits encode the operation sequence within that epoch) to every other node. Nodes compare received zxids against their own, and the node with the highest zxid — meaning it has the most complete, up-to-date transaction history — wins, because promoting a node with an incomplete history as leader could silently lose committed writes. If multiple nodes are tied on zxid, the tiebreaker is server ID, an arbitrary but consistent way to guarantee the election always converges to a single winner rather than oscillating. Election typically completes in 200ms to 2 seconds depending on ensemble size and network conditions, and critically, no writes are accepted by the ensemble at all during this window — reads can still be served by individual followers from their local state, but nothing is committed until a new leader is established.

In production, the majority-quorum requirement means the actual fault tolerance of an ensemble is floor((N-1)/2), not simply 'how many extra nodes you have' — a 3-node ensemble tolerates 1 failure, a 5-node ensemble tolerates 2, but a 4-node ensemble still only tolerates 1, because losing 2 out of 4 nodes leaves only 2 remaining, which is not a majority of 4. This is why every ZooKeeper deployment guide insists on odd-numbered ensembles: an even-numbered ensemble pays the infrastructure and network cost of an extra node while gaining zero additional fault tolerance over the next-smaller odd number, and worse, it adds a coordination cost since every write still needs a majority of the larger total.

The gotcha that catches teams during cross-datacenter deployments: quorum math applies globally across the entire ensemble regardless of network topology, so a 5-node ensemble split 3-2 across two datacenters can survive the loss of the 2-node datacenter (the 3-node side still has majority) but cannot survive the loss of the 3-node datacenter, even though it's 'only' 3 out of 5 nodes — asymmetric datacenter splits mean your actual fault tolerance depends heavily on which specific nodes fail, not just how many, and teams that don't model this specifically can end up in a situation where their 'larger' datacenter going down is actually the survivable case while their 'smaller' one going down takes the whole ensemble offline.