Kafka

Imagine a library that keeps one master card catalog recording exactly which books every patron has already returned. If that catalog is stored on index cards in a single filing cabinet with no backup copy, losing that cabinet doesn't just affect one book — it corrupts the library's memory of every patron's return status at once. __consumer_offsets plays that exact role for a Kafka cluster: instead of tracking application data, it tracks the read progress of every consumer group across every topic in the cluster.

Kafka creates __consumer_offsets automatically the first time any consumer group commits an offset, using the broker defaults offsets.topic.replication.factor (often left at the cluster default) and 50 partitions by default. This was a deliberate design choice to make offset storage itself just another partitioned, replicated Kafka topic rather than a separate external system — reusing Kafka's own replication and compaction machinery instead of building a bespoke offset store. The tradeoff is that this topic's health is now just as important as any user topic's, but because it's created automatically and rarely appears in a team's topic inventory, it's easy to forget it needs the same replication-factor and min.insync.replicas discipline as payments-events or order-created.

Internally, __consumer_offsets is a compacted topic (Kafka periodically removes older records for the same key, keeping only the latest offset per consumer-group/topic/partition combination) rather than a normal time-retained topic. If it was created back when the cluster had only one or two brokers, its replication factor may be stuck at 1 or 2 permanently — Kafka does not automatically increase replication factor when you add brokers later. So a cluster can look perfectly healthy on every application topic while this one internal topic silently sits under-replicated, one broker failure away from data loss on every consumer group's progress.

In production, engineers explicitly include __consumer_offsets in the same under-replicated-partition monitoring and alerting as user topics, and periodically run a replica reassignment against it just like any other undersized topic. Losing this topic's data (or a partition of it) doesn't crash consumers outright — it manifests subtly as consumer groups resetting to auto.offset.reset behavior (earliest or latest) and either reprocessing huge historical backlogs or skipping straight to the newest data, which looks like a data-correctness incident rather than an infrastructure one, making it much harder to diagnose.

The gotcha: because __consumer_offsets is compacted, not time-retained, simply waiting doesn't clean up an under-replication problem the way it might for a normal topic — you must explicitly run kafka-reassign-partitions.sh against it, and because it usually has 50 partitions, a reassignment against it touches every broker in the cluster simultaneously, so teams need to throttle the reassignment bandwidth or risk saturating inter-broker network links during business hours.

This is like a courier confirming a package was delivered to the right building, while the recipient insists it never arrived — and the real answer turns out to be that it was delivered to the mailroom of the wrong floor, or it was delivered on time but the recipient's mail slot had already been cleaned out before they checked it. The delivery succeeded; the visibility into where it landed is what's broken.

In Kafka, a producer receiving an ack only confirms the message reached the partition leader (and, depending on acks configuration, was replicated to the ISR) — it says nothing about whether any particular consumer has read it yet. 'Missing' from a consumer's perspective can mean the message was never routed to a partition that consumer is responsible for, was already deleted by retention before the consumer got to it, or the consumer's offset was moved past it by a rebalance, a manual seek, or an auto.offset.reset event. Kafka was designed to decouple producer acknowledgment completely from consumer progress — intentionally, since coupling them would mean a slow or offline consumer could block producers, which defeats the purpose of a buffered log.

The diagnostic sequence starts by confirming the message actually exists on the topic at all: use kafka-console-consumer with --from-beginning against the specific partition (derivable by hashing the key the same way the producer does) to prove existence independent of any application code. Next, check the consumer group's current assignment and offsets with kafka-consumer-groups.sh --describe — if the consumer group was never assigned the partition holding the message (common right after a rebalance mid-incident), that fully explains the gap. Then check the topic's retention.ms and whether enough time has passed for the message to have been deleted, especially on topics with aggressive retention tuned for a different use case than what's currently consuming it.

In production, the most common real cause is an auto.offset.reset misconfiguration: if a consumer group's committed offset becomes invalid (for example, the offset was itself deleted due to offsets.retention.minutes expiring for an idle consumer group that hadn't committed in a long time), the consumer falls back to either 'earliest' or 'latest' depending on config, and 'latest' silently skips every message produced while the consumer group was inactive — with zero errors logged anywhere, because from Kafka's perspective this is completely valid, expected behavior, not a fault.

The gotcha that catches teams every time: offsets.retention.minutes (default 7 days in older versions, tied to broker uptime in newer ones) can expire a consumer group's committed offset even while the actual message data is still well within the topic's own retention window. So a consumer group that goes idle for a week-long holiday can come back, find its offsets gone, silently reset to 'latest,' and skip an entire week of data that was never deleted — the messages were never missing, the group's memory of where it was just vanished first.

A building's fire alarm system has smoke detectors and sprinklers. Smoke detectors trigger on the earliest sign of trouble, before flames are even visible, giving people time to act. Sprinklers only activate once there's already real heat — by which point damage has started. Good Kafka monitoring needs both kinds of signal, but a mature operations team designs paging around the smoke detectors, not the sprinklers, because by the time the sprinkler-equivalent metric fires, users are already affected.

Kafka exposes broker-level JMX metrics that fall cleanly into these two categories. Under-replicated partition count and ISR shrink rate are 'smoke detector' metrics — they indicate replication is falling behind or brokers are struggling to keep up, well before any client-visible error occurs, because clients using acks=1 or even acks=all against a still-adequate ISR won't see failures yet. Request queue time and request handler idle ratio reveal broker-side saturation (the broker's thread pools are backed up) before latency percentiles blow out. By contrast, consumer lag spikes, elevated producer request timeouts, and NotEnoughReplicasException counts are 'sprinkler' metrics — real, but by definition something has already gone wrong for a client by the time these fire.

Internally, these metrics exist because Kafka brokers expose extensive JMX (Java Management Extensions, the standard way JVM applications publish internal metrics) counters for network request handling, log flush latency, replication fetcher lag, and controller state. A Prometheus JMX exporter sidecar scrapes these and feeds Grafana dashboards and Alertmanager rules. The design intent is that operators build layered alerting: cause-level alerts (under-replicated partitions, request queue time, network processor idle percentage below a threshold) feed into runbooks for platform engineers, while symptom-level alerts (elevated consumer lag, client error rates) are what pages the on-call for the affected application team, since that's the audience who can actually act on client-visible impact.

In production, the request handler idle ratio metric is one of the most predictive and least understood: it measures the percentage of time broker request-handling threads are idle, and a sustained drop below roughly 20-30% strongly predicts request queue buildup and latency spikes minutes before they show up in client-facing p99 latency. Teams that only alert on consumer lag or client error rate consistently get paged 5-15 minutes later than teams also watching broker-internal saturation metrics, and that gap is often the difference between a quiet mitigation and a customer-visible incident.

The non-obvious gotcha: under-replicated partition count can transiently spike to a nonzero value during completely normal operations — a rolling restart for a patch deploy, for instance — and a naive alert on 'any under-replicated partitions > 0' will page constantly during routine maintenance windows, training the on-call to ignore it. The fix experienced teams use is alerting on under-replicated partitions sustained for longer than the expected restart window (for example, more than 5 minutes outside a known maintenance window), which preserves the metric's predictive value without the noise.

Think of topic configuration automation like a building permit process for renovating a house that's currently occupied. Some changes are reversible and low-risk — repainting a room (adjusting a retention value) can be done and undone freely. Others are irreversible the moment you do them — knocking down a load-bearing wall (increasing partition count, which reshuffles which key maps to which partition forever) can't be undone by just reversing the automation, because the house is still occupied while you do it and residents (consumers) get disrupted mid-change.

Kafka's AdminClient API and CLI tools (kafka-configs.sh, kafka-topics.sh --alter) expose topic configuration as declarative key-value pairs, which makes it tempting to manage them the same way as any other infrastructure-as-code resource — diff desired state against live state, apply the delta. This works cleanly for reversible settings like retention.ms, cleanup.policy, or min.insync.replicas. It does not work safely for partition count, because Kafka only supports increasing partitions, never decreasing them, and increasing partitions changes the hash-to-partition mapping for every existing key going forward, silently breaking any consumer logic that assumed a key always lands on the same partition it always has.

A safe automation pipeline therefore treats topic config changes as two distinct categories internally: mutable-safe settings get a standard plan-diff-apply flow, similar to Terraform, run automatically in CI on every merge. Partition count changes (and any other irreversible operation) get routed through a separate path that requires a human-reviewed dry-run showing exactly which topics and consumer groups are affected, and typically a scheduled maintenance window, because the blast radius touches every consumer of that topic simultaneously, not just the automation's direct target.

In production, idempotency for this kind of automation means the pipeline can be re-run safely after a partial failure without making things worse — for example, if a run to bump retention.ms on 40 topics fails after updating 22 of them, re-running it should update only the remaining 18, not error out or reapply to the already-correct ones. This requires the automation to always re-fetch live config immediately before diffing, never trust a cached 'last known state,' since another process or engineer could have changed the topic out of band between runs.

The gotcha experienced platform teams learn the hard way: kafka-configs.sh --alter with --add-config fully replaces the config map for the keys specified but silently leaves unspecified keys untouched — which sounds safe, but means an automation script that generates its config payload from a slightly stale template can silently fail to remove a deprecated setting that a previous manual change added, because the automation only ever adds and never explicitly reconciles the full config against a source of truth, leaving configuration drift that only shows up during the next incident when someone assumes a setting isn't there anymore.

Think of a partition like a single checkout lane at a grocery store. If you want strict first-come-first-served order, you can only promise it within one lane. The moment the store opens more lanes to serve customers faster, two shoppers who arrived seconds apart might get served in a different order depending on which lane moves quicker. Kafka makes the same tradeoff deliberately: it wants both ordering and throughput, so it draws the ordering guarantee at the partition boundary, not the topic boundary.

A Kafka topic is split into partitions specifically so many consumers can read parts of the workload in parallel — parallelism is the entire reason to use a distributed log instead of a single queue. Every record appended to a partition gets a monotonically increasing offset (a per-partition sequence number), and Kafka writes records to a partition strictly in producer send order. This works because exactly one broker — the partition leader — owns the write path for that partition at any moment. Kafka's designers rejected global cross-partition ordering because it would require a global lock on every write, which would destroy the horizontal scalability the whole system is built around.

Internally, when a producer sends a keyed message (for example key=customer_id), Kafka hashes the key and deterministically routes every message with that key to the same partition — this is how per-key ordering is achieved without needing a global order. On the consume side, each consumer group has a group coordinator broker tracking which consumer owns which partition. If a consumer crashes or misses heartbeats past session.timeout.ms, the coordinator marks it dead and triggers a rebalance: it revokes that consumer's partitions and hands them to a live consumer in the group. The new owner does not start from zero — it reads the last offset committed to the internal __consumer_offsets topic and resumes exactly there.

In production, the offset-commit strategy determines what actually happens during failover. If a consumer processes a batch and crashes before committing offsets, the newly assigned consumer re-reads and reprocesses those same messages — this is at-least-once delivery, Kafka's default behavior. Engineers monitor consumer lag (the gap between the latest produced offset and the last committed offset) via kafka-consumer-groups.sh or a lag exporter feeding Prometheus, because widening lag right after a rebalance usually signals the new owner is struggling to catch up due to GC pauses or a hot key. Frequent rebalances themselves are a red flag — a 'rebalance storm' — usually caused by consumers taking longer than max.poll.interval.ms to process a batch, making the coordinator think they died even though they were just slow.

A subtle trap experienced engineers hit is assuming more consumers always fixes lag. Kafka can only actively use as many consumers as there are partitions — with 12 partitions and 20 consumers, 8 sit fully idle. An even sneakier gotcha: because ordering is per-partition, repartitioning a topic to scale out breaks order guarantees for existing keys, since a key's messages may now hash to a different partition than the older messages for that same key sitting in the original partition — anything relying on strict per-key order across a repartition event will silently see out-of-order processing with no error raised.

Picture a company newsletter mailed to 50 different departments. Every department needs its own full copy delivered to its own mailbox (a consumer group), rather than 50 departments fighting over shared mail slots. But within a single department that splits reading across several clerks (consumer instances in one group) to keep up with volume, you only need as many mail slots (partitions) as clerks working that day.

In Kafka, a consumer group is the unit of independent consumption: every group gets its own copy of every message in the topic, tracked by its own offsets in __consumer_offsets. That is why 50 microservices reading the same event stream should be modeled as 50 separate consumer groups, not 50 consumers crammed into one group — cramming them into one group would split the stream 50 ways instead of replicating it 50 times, meaning most services would silently see only a fraction of the events. Kafka was designed this way so that adding a new downstream consumer of an existing stream requires zero changes to the topic or to other consumers — it's an additive, decoupled architecture, which is the entire value proposition over a traditional message queue with single-consumer semantics.

Internally, partition count sets a hard ceiling on parallelism per group: a group can have at most as many active consumers as there are partitions, because each partition can only be owned by one consumer within a group at a time. For 50 independent services with wildly different processing speeds, you pick the partition count based on the neediest consumer group's expected peak parallelism — commonly 24 to 48 partitions for a mid-size event stream — and set replication-factor=3 so the topic survives a broker or AZ failure. A schema registry (a separate service that stores and versions the Avro/Protobuf message schema) sits alongside the topic so all 50 services can evolve their event contract independently without a synchronized deploy.

At scale, the operational risks multiply with consumer count: 50 consumer groups on one topic means 50 independent lag curves to monitor, and a single slow consumer group doesn't block the others but can mask a real problem because dashboards aggregate lag by topic instead of by group. Retention (how long Kafka keeps messages before deleting them) has to be sized for the slowest consumer group plus a safety margin — if any one of the 50 services falls behind by more than the retention window, it silently loses data rather than erroring loudly. Engineers watch min.insync.replicas violations, under-replicated partition counts, and per-group max lag as the top three signals.

The gotcha almost everyone misses: you can increase partition count later, but doing so reassigns the hash-to-partition mapping for every key, so all 50 consuming services that relied on same-key-same-partition ordering will see a discontinuity in ordering exactly at the moment you repartition — old messages for a key sit in the old partition, new messages for the same key land in a different one. Because of this, teams almost always over-provision partition count up front rather than trying to right-size it later, even though excess partitions cost more open file handles and replication overhead on every broker.

Imagine a restaurant with five equally staffed serving stations, but one station happens to be assigned every table from a bus that just arrived all at once. Adding more cooks to the other four stations does nothing for the overburdened one — the problem isn't total kitchen capacity, it's an uneven assignment of work. Kafka consumer lag has the exact same two failure modes: genuinely not enough total processing power, versus one partition getting far more traffic than the others because of a skewed partitioning key.

Consumer lag is formally the difference between a partition's log-end offset (the newest message written) and the consumer group's last committed offset (the newest message fully processed). Kafka exposes this number per partition specifically so operators can distinguish 'the whole group is behind' from 'one partition is behind.' This granularity was a deliberate design choice — early distributed queue systems that only reported aggregate backlog made it nearly impossible to tell a capacity problem from a data-skew problem, and teams would blindly add workers that never helped.

To diagnose it, run kafka-consumer-groups.sh --describe and look at the LAG column per partition rather than the sum. If lag is roughly even across all partitions and rising, that's a genuine capacity or slow-processing issue — check consumer-side metrics like max.poll.records, processing time per record, and whether downstream calls (a database write, an external API) are the bottleneck. If lag is concentrated on one or two partitions while others sit near zero, that's hot-partitioning — almost always because the message key (e.g., customer_id or tenant_id) is unevenly distributed, such as one enterprise customer generating 40% of all events.

In production, teams build alerting on both max per-partition lag and lag growth rate (the derivative, not just the absolute value), because a lag of 50,000 messages means something very different for a topic doing 10 msg/sec versus 50,000 msg/sec. Fixes differ by cause: capacity problems are solved by adding consumers up to the partition count, batching more aggressively, or moving expensive downstream work to an async queue; hot-partition problems require changing the partitioning key (adding a random suffix or sub-key to spread a hot tenant's traffic across multiple partitions) since scaling consumers cannot fix an assignment imbalance.

The gotcha that catches even experienced teams: adding partitions to fix a hot key doesn't help immediately, because the existing hot key still hashes to the same single partition it always did — only new keys benefit from the larger partition space, and the skew problem persists for that specific key until you explicitly change how it's partitioned (e.g., a custom partitioner or key-salting scheme), which usually requires a coordinated producer-side code change, not just a topic config change.

Think of the ISR like a bank requiring at least two of three tellers to countersign a large withdrawal before it's finalized. If one teller goes home sick, the bank can still process withdrawals with the remaining two signatures. But if a second teller also leaves and only one remains, the bank stops processing withdrawals entirely rather than accepting a single signature on a transaction that's supposed to require two — because a single point of failure at that moment could lose the record of the withdrawal entirely.

In Kafka, every partition has one leader broker and some number of follower replicas. The ISR is the subset of followers (plus the leader) that are fully caught up — not lagging behind by more than replica.lag.time.max.ms. This exists because followers that fall behind (due to slow disks, network issues, or a restart) shouldn't be trusted to safely take over as leader without losing data, so Kafka tracks in real time which replicas are actually safe fallback candidates. min.insync.replicas is the operator's way of saying 'do not accept a write unless at least this many replicas, including the leader, have it,' which is what actually prevents data loss — acks=all by itself only means 'wait for the ISR,' and if the ISR has shrunk to one broker, acks=all offers no real protection unless min.insync.replicas blocks the write outright.

Internally, followers continuously fetch from the leader, and the leader tracks each follower's fetch progress. A follower is removed from the ISR the moment it falls too far behind, and re-added once it catches back up. With a topic set to replication-factor=3 and min.insync.replicas=2, the system tolerates exactly one broker failure: if one of the three replicas goes down, the ISR drops to two, which still satisfies min.insync.replicas=2, so writes continue. If a second broker fails simultaneously, ISR drops to one, falls below the minimum, and every producer using acks=all starts getting NotEnoughReplicasException on every write attempt until a replica rejoins the ISR.

In production, engineers monitor under-replicated partition count (a partition whose ISR is smaller than its replication factor) as a leading indicator of exactly this scenario, because a partition can run for a long time with a shrunk ISR without producers noticing if they're using acks=1 or acks=0. Dashboards should alert the moment ISR size drops, not just when writes start failing, because by the time NotEnoughReplicasException fires you're already one more broker failure away from an availability outage. Broker restarts during rolling upgrades are the most common trigger — restarting brokers too quickly, one right after another, can shrink the ISR transiently on many partitions at once if the previous broker hasn't fully rejoined and caught up.

The gotcha many engineers miss: acks=all does not mean 'wait for all replicas' — it means 'wait for the ISR,' which can legitimately be smaller than the full replica set. A team that assumes acks=all always means full multi-broker durability can be surprised to learn that if the ISR has silently shrunk to just the leader (which happens if min.insync.replicas isn't set and only relies on default configuration), acks=all writes succeed even though only one broker actually has the data — the safety net only exists because min.insync.replicas was configured, not because of acks=all alone.

Producers append to partitions; ordering is guaranteed only within a partition. Partitions enable parallelism and are distributed across brokers. Consumers track their position by committing offsets, so they can resume where they left off. Partition count sets the max consumer parallelism per group.

Within a group, Kafka assigns each partition to exactly one consumer (rebalancing on membership changes), giving horizontal scaling and fault tolerance. Beyond the partition count, extra consumers sit idle. Offset commits plus idempotent producers/transactions enable effectively-once semantics.

acks=all with min.insync.replicas>1 means a write is acknowledged only after replicating to the required in-sync replicas, surviving a broker loss. acks=1 acknowledges after the leader only (faster, can lose data on leader failure); acks=0 is fire-and-forget. Durability vs latency is the trade-off.

Growing lag means consumers can't keep up with producers, delaying downstream processing. Monitor it per partition/group (e.g. kafka_consumergroup_lag) and alert on sustained growth. Fixes include adding consumers (up to partition count), optimizing processing, or increasing partitions.