How do you handle Kafka consumer lag and backpressure in high-throughput payment processing?

Think of a call center during a product recall. If calls come in faster than agents can answer, the queue grows (consumer lag). You can add more agents (scale consumers), redirect overflow calls to voicemail (dead letter queue), or temporarily stop accepting new calls from the website (backpressure to producers). The key is detecting the queue growth early and responding before callers give up.

Consumer lag is the difference between the latest message offset in a partition and the consumer's current committed offset. A lag of zero means the consumer is caught up. Growing lag means messages are being produced faster than consumed. In a payment processing system, growing lag means transactions are being delayed, which can trigger timeouts, duplicate processing attempts, and compliance violations for settlement deadlines.

The response strategy has three layers. First, scale consumers: use KEDA (Kubernetes Event-Driven Autoscaler) with a Kafka trigger that watches consumer group lag and scales the Deployment from 1 to N consumers. Each consumer in the group gets assigned partitions, so the maximum parallelism equals the number of partitions. Second, handle failures: messages that fail processing after retries go to a dead letter topic for manual investigation rather than blocking the entire partition. Third, apply backpressure: if a downstream service (like the fraud detection API) is slow, implement circuit breakers so consumers pause processing rather than overwhelming the failing service.

At production scale, partition count determines your maximum consumer parallelism. If you have 12 partitions and 12 consumers, each handles one partition. Adding a 13th consumer gives it nothing to do. Teams should provision enough partitions upfront (at least 3x the expected peak consumer count) because repartitioning an existing topic requires data migration. Monitor lag per partition, not just aggregate lag, because a single hot partition with a slow consumer can cause localized delays while the aggregate looks healthy.

The non-obvious gotcha is that consumer lag spikes are normal during deployments. When consumers restart during a rolling update, partitions are reassigned and the new consumers start from the last committed offset, which may be slightly behind. This brief lag spike should resolve within minutes. Alert on lag that is continuously growing for 15+ minutes, not on momentary spikes. Also, max.poll.records and max.poll.interval.ms settings determine how many messages a consumer fetches per poll and how long it can take to process them — misconfiguring these causes unnecessary rebalances that worsen lag.