Kafka brokers are healthy but the __consumer_offsets topic itself is under-replicated — why does this matter more than any other topic being under-replicated, and how do you fix it?
Quick Answer
__consumer_offsets is the internal topic that stores every consumer group's committed offsets, so if it becomes under-replicated the cluster's ability to durably remember 'what has already been processed' is at risk, not just one application's data. It typically becomes under-replicated after adding brokers without ever reassigning this topic's replicas, since it was created with a small replication factor by default years earlier and never revisited.
Detailed Answer
Imagine a library that keeps one master card catalog recording exactly which books every patron has already returned. If that catalog is stored on index cards in a single filing cabinet with no backup copy, losing that cabinet doesn't just affect one book — it corrupts the library's memory of every patron's return status at once. __consumer_offsets plays that exact role for a Kafka cluster: instead of tracking application data, it tracks the read progress of every consumer group across every topic in the cluster.
Kafka creates __consumer_offsets automatically the first time any consumer group commits an offset, using the broker defaults offsets.topic.replication.factor (often left at the cluster default) and 50 partitions by default. This was a deliberate design choice to make offset storage itself just another partitioned, replicated Kafka topic rather than a separate external system — reusing Kafka's own replication and compaction machinery instead of building a bespoke offset store. The tradeoff is that this topic's health is now just as important as any user topic's, but because it's created automatically and rarely appears in a team's topic inventory, it's easy to forget it needs the same replication-factor and min.insync.replicas discipline as payments-events or order-created.
Internally, __consumer_offsets is a compacted topic (Kafka periodically removes older records for the same key, keeping only the latest offset per consumer-group/topic/partition combination) rather than a normal time-retained topic. If it was created back when the cluster had only one or two brokers, its replication factor may be stuck at 1 or 2 permanently — Kafka does not automatically increase replication factor when you add brokers later. So a cluster can look perfectly healthy on every application topic while this one internal topic silently sits under-replicated, one broker failure away from data loss on every consumer group's progress.
In production, engineers explicitly include __consumer_offsets in the same under-replicated-partition monitoring and alerting as user topics, and periodically run a replica reassignment against it just like any other undersized topic. Losing this topic's data (or a partition of it) doesn't crash consumers outright — it manifests subtly as consumer groups resetting to auto.offset.reset behavior (earliest or latest) and either reprocessing huge historical backlogs or skipping straight to the newest data, which looks like a data-correctness incident rather than an infrastructure one, making it much harder to diagnose.
The gotcha: because __consumer_offsets is compacted, not time-retained, simply waiting doesn't clean up an under-replication problem the way it might for a normal topic — you must explicitly run kafka-reassign-partitions.sh against it, and because it usually has 50 partitions, a reassignment against it touches every broker in the cluster simultaneously, so teams need to throttle the reassignment bandwidth or risk saturating inter-broker network links during business hours.