How do you monitor Kafka on Kubernetes, and what metrics and alerts matter most?

Think of monitoring a highway system. You track traffic flow (throughput), lane closures (under-replicated partitions), backup length (consumer lag), and road surface condition (disk usage). A single lane closure might not cause problems, but if multiple lanes close simultaneously, traffic grinds to a halt. The same applies to Kafka — individual metric spikes are normal, but correlated spikes indicate a systemic problem.

Kafka exposes hundreds of JMX metrics covering broker performance, topic throughput, consumer behavior, and replication health. On Kubernetes with Strimzi, the JMX Exporter runs as a sidecar container in each broker pod, converting JMX MBeans into Prometheus-compatible metrics on an HTTP endpoint. A ServiceMonitor resource tells Prometheus to scrape these endpoints, and Grafana dashboards visualize the data.

The most critical metrics fall into four categories. Replication health: kafka.server UnderReplicatedPartitions should always be zero — any non-zero value means data is at risk. Consumer health: kafka.consumer.group lag per partition shows how far behind consumers are — growing lag means consumers cannot keep up with producers. Broker performance: kafka.network RequestMetrics for produce and fetch request latency — p99 above 100ms indicates broker pressure. Resource usage: disk utilization per broker — Kafka stores messages on disk, and running out stops the broker.

At production scale, alerting should follow the symptom-based approach. Alert on under-replicated partitions greater than zero for more than 5 minutes (data durability risk), consumer lag increasing for more than 15 minutes (processing falling behind), produce request p99 latency above 200ms (client impact), and disk usage above 75% (capacity planning trigger). Avoid alerting on individual broker CPU spikes — they are often transient during rebalancing.

The non-obvious gotcha is that consumer lag metrics are only accurate when consumers are actively polling. If a consumer crashes and stops polling, the lag metric freezes at the last known value rather than showing increasing lag. Teams should also monitor consumer group state (Stable, Rebalancing, Dead) and alert on groups stuck in Rebalancing for more than 5 minutes, which indicates a consumer that keeps crashing during rebalance.