How does the in-sync replica (ISR) set work in Kafka, and what exactly happens to producers and consumers when ISR shrinks below min.insync.replicas?

Think of the ISR like a bank requiring at least two of three tellers to countersign a large withdrawal before it's finalized. If one teller goes home sick, the bank can still process withdrawals with the remaining two signatures. But if a second teller also leaves and only one remains, the bank stops processing withdrawals entirely rather than accepting a single signature on a transaction that's supposed to require two — because a single point of failure at that moment could lose the record of the withdrawal entirely.

In Kafka, every partition has one leader broker and some number of follower replicas. The ISR is the subset of followers (plus the leader) that are fully caught up — not lagging behind by more than replica.lag.time.max.ms. This exists because followers that fall behind (due to slow disks, network issues, or a restart) shouldn't be trusted to safely take over as leader without losing data, so Kafka tracks in real time which replicas are actually safe fallback candidates. min.insync.replicas is the operator's way of saying 'do not accept a write unless at least this many replicas, including the leader, have it,' which is what actually prevents data loss — acks=all by itself only means 'wait for the ISR,' and if the ISR has shrunk to one broker, acks=all offers no real protection unless min.insync.replicas blocks the write outright.

Internally, followers continuously fetch from the leader, and the leader tracks each follower's fetch progress. A follower is removed from the ISR the moment it falls too far behind, and re-added once it catches back up. With a topic set to replication-factor=3 and min.insync.replicas=2, the system tolerates exactly one broker failure: if one of the three replicas goes down, the ISR drops to two, which still satisfies min.insync.replicas=2, so writes continue. If a second broker fails simultaneously, ISR drops to one, falls below the minimum, and every producer using acks=all starts getting NotEnoughReplicasException on every write attempt until a replica rejoins the ISR.

In production, engineers monitor under-replicated partition count (a partition whose ISR is smaller than its replication factor) as a leading indicator of exactly this scenario, because a partition can run for a long time with a shrunk ISR without producers noticing if they're using acks=1 or acks=0. Dashboards should alert the moment ISR size drops, not just when writes start failing, because by the time NotEnoughReplicasException fires you're already one more broker failure away from an availability outage. Broker restarts during rolling upgrades are the most common trigger — restarting brokers too quickly, one right after another, can shrink the ISR transiently on many partitions at once if the previous broker hasn't fully rejoined and caught up.

The gotcha many engineers miss: acks=all does not mean 'wait for all replicas' — it means 'wait for the ISR,' which can legitimately be smaller than the full replica set. A team that assumes acks=all always means full multi-broker durability can be surprised to learn that if the ISR has silently shrunk to just the leader (which happens if min.insync.replicas isn't set and only relies on default configuration), acks=all writes succeed even though only one broker actually has the data — the safety net only exists because min.insync.replicas was configured, not because of acks=all alone.