What causes Kafka consumer lag to grow unbounded during normal load, and how do you tell a capacity problem from a hot-partition problem?

Imagine a restaurant with five equally staffed serving stations, but one station happens to be assigned every table from a bus that just arrived all at once. Adding more cooks to the other four stations does nothing for the overburdened one — the problem isn't total kitchen capacity, it's an uneven assignment of work. Kafka consumer lag has the exact same two failure modes: genuinely not enough total processing power, versus one partition getting far more traffic than the others because of a skewed partitioning key.

Consumer lag is formally the difference between a partition's log-end offset (the newest message written) and the consumer group's last committed offset (the newest message fully processed). Kafka exposes this number per partition specifically so operators can distinguish 'the whole group is behind' from 'one partition is behind.' This granularity was a deliberate design choice — early distributed queue systems that only reported aggregate backlog made it nearly impossible to tell a capacity problem from a data-skew problem, and teams would blindly add workers that never helped.

To diagnose it, run kafka-consumer-groups.sh --describe and look at the LAG column per partition rather than the sum. If lag is roughly even across all partitions and rising, that's a genuine capacity or slow-processing issue — check consumer-side metrics like max.poll.records, processing time per record, and whether downstream calls (a database write, an external API) are the bottleneck. If lag is concentrated on one or two partitions while others sit near zero, that's hot-partitioning — almost always because the message key (e.g., customer_id or tenant_id) is unevenly distributed, such as one enterprise customer generating 40% of all events.

In production, teams build alerting on both max per-partition lag and lag growth rate (the derivative, not just the absolute value), because a lag of 50,000 messages means something very different for a topic doing 10 msg/sec versus 50,000 msg/sec. Fixes differ by cause: capacity problems are solved by adding consumers up to the partition count, batching more aggressively, or moving expensive downstream work to an async queue; hot-partition problems require changing the partitioning key (adding a random suffix or sub-key to spread a hot tenant's traffic across multiple partitions) since scaling consumers cannot fix an assignment imbalance.

The gotcha that catches even experienced teams: adding partitions to fix a hot key doesn't help immediately, because the existing hot key still hashes to the same single partition it always did — only new keys benefit from the larger partition space, and the skew problem persists for that specific key until you explicitly change how it's partitioned (e.g., a custom partitioner or key-salting scheme), which usually requires a coordinated producer-side code change, not just a topic config change.