Producers report successful writes (acks received) but a downstream consumer swears messages are missing — what's your diagnostic sequence to find where they actually went?
Quick Answer
Start by checking whether the consumer is even subscribed to the partitions the messages landed on, since a mismatched partition assignment or a paused consumer looks identical to 'missing messages' from the application's point of view. Then check retention (the messages may have already expired), and finally check whether the consumer's offset was manually reset or auto-reset past the messages in question.
Detailed Answer
This is like a courier confirming a package was delivered to the right building, while the recipient insists it never arrived — and the real answer turns out to be that it was delivered to the mailroom of the wrong floor, or it was delivered on time but the recipient's mail slot had already been cleaned out before they checked it. The delivery succeeded; the visibility into where it landed is what's broken.
In Kafka, a producer receiving an ack only confirms the message reached the partition leader (and, depending on acks configuration, was replicated to the ISR) — it says nothing about whether any particular consumer has read it yet. 'Missing' from a consumer's perspective can mean the message was never routed to a partition that consumer is responsible for, was already deleted by retention before the consumer got to it, or the consumer's offset was moved past it by a rebalance, a manual seek, or an auto.offset.reset event. Kafka was designed to decouple producer acknowledgment completely from consumer progress — intentionally, since coupling them would mean a slow or offline consumer could block producers, which defeats the purpose of a buffered log.
The diagnostic sequence starts by confirming the message actually exists on the topic at all: use kafka-console-consumer with --from-beginning against the specific partition (derivable by hashing the key the same way the producer does) to prove existence independent of any application code. Next, check the consumer group's current assignment and offsets with kafka-consumer-groups.sh --describe — if the consumer group was never assigned the partition holding the message (common right after a rebalance mid-incident), that fully explains the gap. Then check the topic's retention.ms and whether enough time has passed for the message to have been deleted, especially on topics with aggressive retention tuned for a different use case than what's currently consuming it.
In production, the most common real cause is an auto.offset.reset misconfiguration: if a consumer group's committed offset becomes invalid (for example, the offset was itself deleted due to offsets.retention.minutes expiring for an idle consumer group that hadn't committed in a long time), the consumer falls back to either 'earliest' or 'latest' depending on config, and 'latest' silently skips every message produced while the consumer group was inactive — with zero errors logged anywhere, because from Kafka's perspective this is completely valid, expected behavior, not a fault.
The gotcha that catches teams every time: offsets.retention.minutes (default 7 days in older versions, tied to broker uptime in newer ones) can expire a consumer group's committed offset even while the actual message data is still well within the topic's own retention window. So a consumer group that goes idle for a week-long holiday can come back, find its offsets gone, silently reset to 'latest,' and skip an entire week of data that was never deleted — the messages were never missing, the group's memory of where it was just vanished first.