How does Kafka guarantee message ordering and what happens when a consumer fails mid-partition?

Think of a partition like a single checkout lane at a grocery store. If you want strict first-come-first-served order, you can only promise it within one lane. The moment the store opens more lanes to serve customers faster, two shoppers who arrived seconds apart might get served in a different order depending on which lane moves quicker. Kafka makes the same tradeoff deliberately: it wants both ordering and throughput, so it draws the ordering guarantee at the partition boundary, not the topic boundary.

A Kafka topic is split into partitions specifically so many consumers can read parts of the workload in parallel — parallelism is the entire reason to use a distributed log instead of a single queue. Every record appended to a partition gets a monotonically increasing offset (a per-partition sequence number), and Kafka writes records to a partition strictly in producer send order. This works because exactly one broker — the partition leader — owns the write path for that partition at any moment. Kafka's designers rejected global cross-partition ordering because it would require a global lock on every write, which would destroy the horizontal scalability the whole system is built around.

Internally, when a producer sends a keyed message (for example key=customer_id), Kafka hashes the key and deterministically routes every message with that key to the same partition — this is how per-key ordering is achieved without needing a global order. On the consume side, each consumer group has a group coordinator broker tracking which consumer owns which partition. If a consumer crashes or misses heartbeats past session.timeout.ms, the coordinator marks it dead and triggers a rebalance: it revokes that consumer's partitions and hands them to a live consumer in the group. The new owner does not start from zero — it reads the last offset committed to the internal __consumer_offsets topic and resumes exactly there.

In production, the offset-commit strategy determines what actually happens during failover. If a consumer processes a batch and crashes before committing offsets, the newly assigned consumer re-reads and reprocesses those same messages — this is at-least-once delivery, Kafka's default behavior. Engineers monitor consumer lag (the gap between the latest produced offset and the last committed offset) via kafka-consumer-groups.sh or a lag exporter feeding Prometheus, because widening lag right after a rebalance usually signals the new owner is struggling to catch up due to GC pauses or a hot key. Frequent rebalances themselves are a red flag — a 'rebalance storm' — usually caused by consumers taking longer than max.poll.interval.ms to process a batch, making the coordinator think they died even though they were just slow.

A subtle trap experienced engineers hit is assuming more consumers always fixes lag. Kafka can only actively use as many consumers as there are partitions — with 12 partitions and 20 consumers, 8 sit fully idle. An even sneakier gotcha: because ordering is per-partition, repartitioning a topic to scale out breaks order guarantees for existing keys, since a key's messages may now hash to a different partition than the older messages for that same key sitting in the original partition — anything relying on strict per-key order across a repartition event will silently see out-of-order processing with no error raised.