Why does a healthy-looking ZooKeeper ensemble suddenly start rejecting all writes even though a majority of nodes are up, and how does the client-side fsync configuration on just one follower cause this?

Picture a notary public process that requires a majority of three witnesses to physically sign and date-stamp a document with wet ink before a contract is considered binding — not just verbally agreeing to it. If two of the three witnesses are available but their pens are drying out and it takes them a full minute each to get a working signature down, the contract-signing process backs up and slows to a crawl even though, from a 'the witnesses showed up' perspective, everything looks perfectly fine. The bottleneck isn't witness availability, it's the physical act of durably committing ink to paper.

ZooKeeper's ZAB protocol requires that a write be persisted to a transaction log and fsynced on a majority of ensemble members before the leader acknowledges it as committed, specifically because acknowledging a write that only lives in OS page cache (not yet flushed to physical disk) risks losing it if a node crashes before the OS gets around to flushing that buffer — and ZooKeeper's entire value proposition as a coordination service depends on writes it says are committed actually surviving a crash. This is a deliberate consistency-over-latency design choice: unlike systems that treat an OS-level write as sufficient, ZooKeeper insists on the disk-durability guarantee for every single write, every time, no exceptions.

Internally, this means every write's latency is bound by the slowest fsync among whichever majority subset of nodes are involved in acknowledging it — not by network round-trip time, not by CPU, specifically disk fsync latency. If even one or two nodes in the ensemble share physical disks with other noisy workloads (a common mistake when ZooKeeper is co-located on general-purpose infrastructure rather than given dedicated, ideally SSD, storage), or if the transaction log and the periodic snapshot are configured to write to the same physical disk (causing I/O contention between routine logging and periodic snapshotting), fsync calls that normally take single-digit milliseconds can spike to hundreds of milliseconds or more — and because a majority is required, even one slow node in the majority path stalls every write waiting on it.

In production, engineers dedicate separate physical disks (or at minimum separate volumes) for the ZooKeeper transaction log versus the snapshot directory specifically to eliminate this contention, monitor fsync latency directly via ZooKeeper's own exposed metrics rather than inferring it from generic disk I/O graphs, and treat any sustained fsync latency increase as a leading indicator of exactly this kind of write-stall incident, well before client-visible timeouts start. This is one of the reasons ZooKeeper deployment guides insist on dedicated hardware or at least dedicated storage volumes rather than sharing disks with other services — the failure mode isn't a crash, it's a silent, ensemble-wide latency degradation that every 'is the node up' health check will completely miss, since the node is up, reachable, and correctly reporting its status the entire time.

The gotcha that catches teams during cloud migrations: moving a previously bare-metal ZooKeeper ensemble onto cloud block storage (like network-attached SSD volumes) without re-validating fsync latency characteristics can silently introduce exactly this problem, because network-attached storage has fundamentally different and less predictable fsync latency than local NVMe, especially under the storage provider's own noisy-neighbor contention — and this kind of degradation often only shows up under real production write volume, meaning a migration that passed every functional test in staging (where write volume was low) can start exhibiting mysterious, hard-to-reproduce write stalls only once it's carrying real traffic weeks after the migration was declared successful.