Which ZooKeeper metrics actually predict an ensemble-wide outage before it happens, and why is 'is the leader reachable' the wrong question to alert on?

A toll booth attendant can be standing right there, visibly present and waving at every car (fully 'reachable'), while the actual line of cars behind the booth stretches for miles because the payment terminal itself is processing transactions too slowly. Anyone checking 'is the attendant there' gets a reassuring yes the entire time the real problem — transaction processing throughput — is actively causing a growing backup that eventually spills onto the highway.

ZooKeeper's leader is a single process handling write coordination for the entire ensemble, and basic reachability checks (can you open a TCP connection, does it respond to a ping-equivalent 'ruok' command) only confirm the process is running and accepting connections — they say nothing about whether it's keeping up with the actual write workload flowing through it. This is a critical gap because ZooKeeper's design ties every write's latency to fsync durability and majority acknowledgment, so the leader can be completely 'up' by every naive health check definition while genuinely struggling under load, backlogged on outstanding requests, or waiting on slow disk I/O from itself or from followers in the acknowledgment path.

The metrics that actually predict trouble are exposed through ZooKeeper's own four-letter-word commands and JMX: outstanding requests (the number of client requests received but not yet responded to — a growing queue here means the server is falling behind regardless of whether it's still accepting new connections), fsync latency percentiles (directly measuring the disk-durability bottleneck covered in write-stall scenarios), and watch count (the total number of active watches registered across all clients, which grows with cluster size and application usage and directly correlates with the notification fan-out cost the server has to do on every relevant write). A steadily climbing watch count without a corresponding increase in legitimate cluster growth often indicates a client-side bug — a service that registers a new watch on every poll cycle instead of reusing one, slowly leaking watches and increasing server-side notification overhead on every single write to affected paths.

In production, mature ZooKeeper monitoring treats outstanding request count and fsync latency as the primary paging signals, with basic leader-reachability checks relegated to a secondary, confirmatory role — because by the time a leader becomes fully unreachable, outstanding requests and fsync latency have almost always already been degrading for some window beforehand, and catching that earlier window is what actually gives an on-call engineer time to intervene (adding ensemble capacity, killing a runaway client, migrating to faster disks) before a full connectivity-level outage occurs. Dashboards built around 'ensemble up/down' status alone consistently miss the entire buildup phase of an incident and only alert once it's already fully materialized as a hard outage.

The gotcha that experienced platform teams learn from a real incident: watch count and outstanding requests can both spike from a single misbehaving client application — for example, a poorly written service that reconnects and re-registers thousands of watches in a tight retry loop after misinterpreting a transient error as a fatal one — and because ZooKeeper is a shared multi-tenant coordination service, that one client's bug degrades performance and durability for every other unrelated client using the same ensemble. This means alerting on ensemble-wide aggregate metrics alone isn't sufficient for fast root-causing; teams also need per-client connection and watch-count breakdowns (from echo cons) to immediately identify which specific client is the actual source of a shared-resource degradation, rather than treating the whole ensemble as uniformly at fault when really one tenant is causing it for everyone.