Ephemeral znodes are supposed to disappear when a service instance dies — so why do production teams see 'ghost' service instances still registered in ZooKeeper for 10-30 seconds after a real crash, and why can't you just lower the timeout to fix it?
Quick Answer
An ephemeral znode is tied to the client's session, not to the process being alive from a client-perspective health check, and a session only expires after sessionTimeout passes without any heartbeat — so a genuinely crashed process still 'exists' in ZooKeeper for up to that full timeout window, which is usually tuned to 10-30 seconds specifically to avoid false-positive deletions during normal GC pauses or brief network blips. Lowering the timeout reduces ghost duration but directly increases the risk of falsely deregistering healthy instances that are just experiencing a temporary pause, trading one failure mode for a worse one.
Detailed Answer
Think of ephemeral znodes like a hotel's 'do not disturb' sign system tied to a guest's keycard activity rather than a direct camera in the room. If a guest's keycard stops being used for a while, the hotel doesn't instantly declare the room vacated — it waits a defined grace period, because the guest might just be asleep or in the shower, not actually checked out. Only after that grace period with zero card activity does housekeeping treat the room as empty. The tradeoff is unavoidable: too short a grace period and housekeeping barges in on a sleeping guest; too long and an actually-checked-out room sits marked occupied for a while.
An ephemeral znode in ZooKeeper exists exactly as long as the client session that created it is alive, and a session is considered alive as long as the client sends heartbeats (small keep-alive pings) within the negotiated sessionTimeout window. This design was chosen deliberately as the foundation for service discovery in systems like Kafka (broker registration) and HBase (region server tracking), because it means service liveness detection comes for free from the existing TCP connection and session mechanism, rather than requiring a separate health-check subsystem — if a service instance's process dies, its TCP connection to ZooKeeper eventually drops, its session expires, and the znode is automatically cleaned up with zero extra code from the service itself.
Internally, the session timeout isn't purely client-controlled — the client requests a timeout when connecting, but the ZooKeeper server clamps it between minSessionTimeout and maxSessionTimeout configured on the ensemble (commonly a floor around 2x the tickTime and a ceiling around 20x tickTime). While the session is active, the client library sends periodic heartbeats automatically in the background, and if the server doesn't hear from the client within the session timeout window, it expires the session — deleting every ephemeral znode created by that session and notifying any watchers (clients subscribed to changes on that znode's parent path) that the node is gone. This full path — missed heartbeats, timeout expiry, deletion, watcher notification — is why there's an inherent delay between an actual crash and other services noticing.
In production, teams tune sessionTimeout as a direct tradeoff between two opposing risks: setting it too low (a few seconds) makes the system responsive to real failures but creates false-positive deregistrations whenever a JVM experiences a stop-the-world garbage collection pause longer than the timeout, or the network has a brief hiccup — because the client thread doing heartbeating gets frozen along with everything else during a GC pause, ZooKeeper sees no heartbeat and expires a session for a service instance that was actually still alive and about to resume normal operation microseconds later. Setting it too high (a minute or more) makes the system stable against transient pauses but means a genuinely dead instance stays registered and potentially still receiving traffic routed to it for a much longer window, directly increasing user-facing error rates during real outages.
The gotcha that catches teams who tune sessionTimeout too aggressively low to reduce ghost-instance duration: this doesn't just risk occasional false deregistration, it can trigger cascading failures during periods of real load, because GC pause frequency and duration typically get worse under memory pressure, which itself often correlates with high traffic — meaning the exact moments the system is under the most real load are also the moments most likely to trigger a false session expiry from an aggressive timeout, deregistering a perfectly healthy, busy instance right when it's needed most, and potentially triggering a stampede as its traffic gets redistributed to remaining instances, pushing them into GC pressure too.