Which Logstash pipeline metrics actually indicate an emerging backlog before it becomes an outage?
Quick Answer
Watch the gap between events.in and events.out rates per pipeline — a growing, sustained gap means events are accumulating somewhere even while the process stays up — alongside persistent queue size relative to its configured max, and per-plugin duration_in_millis to spot which stage is slow. A sudden, sustained divergence between in-rate and out-rate is the earliest reliable signal of an emerging backlog, well before queue capacity is exhausted or the output starts actively rejecting writes.
Detailed Answer
Picture a bathtub with the faucet running and the drain partially clogged. The water level doesn't jump instantly — it creeps up gradually as inflow slightly exceeds outflow, and if you're only glancing at the tub occasionally, you might not notice the slow rise until it's nearly overflowing. But if you're watching the actual flow rates, how much water is coming in per minute versus draining out per minute, you'd catch the mismatch within the first few minutes, long before the water level itself becomes visually alarming.
Logstash exposes exactly these flow rates through its monitoring API and the _node/stats endpoint, because Elastic designed the pipeline to be introspectable at each stage specifically so operators don't have to wait for a hard failure — queue full, disk full, process OOM — to know something is wrong. The core insight baked into this design is that a pipeline can look perfectly healthy by a simple up/down check while quietly losing the race between ingestion rate and processing rate for many minutes before anything actually crashes.
Internally, each pipeline tracks cumulative events.in, events.filtered, and events.out counters, plus per-plugin timing where duration_in_millis divided by event count gives the average per-event processing time for each input, filter, and output plugin. Sampling these counters every 30 to 60 seconds and computing rates gives you in-rate versus out-rate directly; if in-rate consistently exceeds out-rate, the difference is accumulating somewhere — either in the persistent queue on disk, the in-memory queue, or, worst case with a memory queue and no backpressure protection configured, getting silently dropped once buffers overflow.
In production, dashboards typically graph events.in per second against events.out per second per pipeline side by side, persistent queue disk usage as a percentage of queue.max_bytes, and per-filter average duration on the same timeline. Alerting rules page when in-rate exceeds out-rate by more than some margin, say 10 percent, sustained for five or more minutes, since that's a genuine leading indicator — versus ticketing on slower creeping trends like gradually increasing per-event filter duration, which is often an early sign of a grok pattern becoming pathological as upstream data shape drifts.
The non-obvious gotcha: events.out counts events leaving Logstash's own filter and output stage — it does not necessarily mean the output destination actually persisted them successfully, especially with output configurations that don't enforce strict acknowledgment on every write. A healthy-looking in-rate to out-rate balance inside Logstash can coexist with a real data-loss problem happening one hop further downstream at the storage layer, which is why pipeline metrics need to be paired with destination-side ingestion metrics, like OpenSearch's own indexing rate and rejected-document count, to get the complete picture rather than a partial one that looks reassuring in isolation.