Which Kafka broker and topic metrics actually predict an outage before it happens, versus metrics that only confirm one already occurred?

A building's fire alarm system has smoke detectors and sprinklers. Smoke detectors trigger on the earliest sign of trouble, before flames are even visible, giving people time to act. Sprinklers only activate once there's already real heat — by which point damage has started. Good Kafka monitoring needs both kinds of signal, but a mature operations team designs paging around the smoke detectors, not the sprinklers, because by the time the sprinkler-equivalent metric fires, users are already affected.

Kafka exposes broker-level JMX metrics that fall cleanly into these two categories. Under-replicated partition count and ISR shrink rate are 'smoke detector' metrics — they indicate replication is falling behind or brokers are struggling to keep up, well before any client-visible error occurs, because clients using acks=1 or even acks=all against a still-adequate ISR won't see failures yet. Request queue time and request handler idle ratio reveal broker-side saturation (the broker's thread pools are backed up) before latency percentiles blow out. By contrast, consumer lag spikes, elevated producer request timeouts, and NotEnoughReplicasException counts are 'sprinkler' metrics — real, but by definition something has already gone wrong for a client by the time these fire.

Internally, these metrics exist because Kafka brokers expose extensive JMX (Java Management Extensions, the standard way JVM applications publish internal metrics) counters for network request handling, log flush latency, replication fetcher lag, and controller state. A Prometheus JMX exporter sidecar scrapes these and feeds Grafana dashboards and Alertmanager rules. The design intent is that operators build layered alerting: cause-level alerts (under-replicated partitions, request queue time, network processor idle percentage below a threshold) feed into runbooks for platform engineers, while symptom-level alerts (elevated consumer lag, client error rates) are what pages the on-call for the affected application team, since that's the audience who can actually act on client-visible impact.

In production, the request handler idle ratio metric is one of the most predictive and least understood: it measures the percentage of time broker request-handling threads are idle, and a sustained drop below roughly 20-30% strongly predicts request queue buildup and latency spikes minutes before they show up in client-facing p99 latency. Teams that only alert on consumer lag or client error rate consistently get paged 5-15 minutes later than teams also watching broker-internal saturation metrics, and that gap is often the difference between a quiet mitigation and a customer-visible incident.

The non-obvious gotcha: under-replicated partition count can transiently spike to a nonzero value during completely normal operations — a rolling restart for a patch deploy, for instance — and a naive alert on 'any under-replicated partitions > 0' will page constantly during routine maintenance windows, training the on-call to ignore it. The fix experienced teams use is alerting on under-replicated partitions sustained for longer than the expected restart window (for example, more than 5 minutes outside a known maintenance window), which preserves the metric's predictive value without the noise.