The standard ZooKeeper distributed lock recipe uses ephemeral sequential znodes and each client watches only its immediate predecessor — why does that specific detail matter, and what happens if a client instead watches the lock node directly?

Imagine a deli counter with a take-a-number system, except instead of a simple display showing 'now serving 42,' every single waiting customer's phone buzzes every time any number is called, and each customer has to check the display themselves to see if it's finally their turn. With 200 people waiting, calling one number causes 200 phones to buzz and 200 people to look up, for a single actual state change. The better design is a system where only the person holding ticket 43 gets notified when ticket 42 is served, since they're the only one whose status just changed.

ZooKeeper's distributed lock recipe works by having every client wanting the lock create an ephemeral sequential znode (a znode with an auto-incrementing numeric suffix appended by ZooKeeper itself, guaranteeing a strict global ordering of arrival) under a shared lock path. The client whose znode has the lowest sequence number holds the lock; everyone else must wait. The critical design detail is what each waiting client watches: instead of watching the lock path itself or the currently-lowest znode, each client watches only the znode with the sequence number immediately below its own.

This works because ZooKeeper watches are one-time, single-target notifications — setting a watch on a specific znode means you get notified only when that specific znode changes, not when any znode under a parent path changes (that would require a different kind of watch on the parent, which is exactly the design the recipe avoids). By having client with sequence number 45 watch only znode 44 (not 43, 42, or the lock root broadly), a release of lock 44 wakes up exactly one client — number 45 — which then checks if it's now the lowest remaining and either takes the lock or sets a new watch on whatever is now immediately below it. Internally this creates a linked chain of one-to-one watch relationships rather than a fan-out broadcast, which is the entire point.

In production, this distinction becomes critical at scale: with 500 clients contending for a lock and everyone watching the shared parent path or the current lock-holder directly, every single lock release triggers a notification storm where all 499 waiting clients wake up simultaneously, each re-lists the children of the lock path to determine if they're now lowest, and 498 of them immediately go back to waiting having wasted a round trip to ZooKeeper for nothing — this is the classic thundering herd problem, and at high contention it can generate enough simultaneous read load against the ZooKeeper ensemble to meaningfully degrade its performance for every other client using it for unrelated purposes, since ZooKeeper is a shared coordination service, not a per-lock isolated resource. With the predecessor-watching pattern, the same 500-client scenario generates exactly one notification per release, regardless of how many clients are queued.

The gotcha that catches engineers implementing this recipe from scratch instead of using a maintained library like Apache Curator: there's a race condition between a client checking 'am I the lowest' and setting its watch on the predecessor — if the predecessor znode is deleted in the gap between the client listing the children and setting its watch, the client can miss the deletion notification entirely and wait forever, because ZooKeeper watches only fire on changes that happen after the watch is registered, not on state that already changed before you started watching. The correct implementation re-checks whether the predecessor still exists immediately after registering the watch, in the same logical step, specifically to close this race — which is exactly the kind of subtle correctness detail that a hand-rolled implementation gets wrong far more often than a well-tested library like Curator's InterProcessMutex.