How do you design a Logstash pipeline for high-throughput without data loss?
Quick Answer
Enable the persistent (disk-backed) queue so events accepted from an input survive a JVM crash before the output has written them, size pipeline.workers and pipeline.batch.size to match actual filter CPU cost rather than defaults, and route unparseable events to a dead letter queue instead of blocking or silently dropping them. The core principle is decoupling ingestion rate from processing/output rate through durable buffering, so a slow downstream store causes backpressure, not silent loss.
Detailed Answer
Picture a busy restaurant kitchen during a dinner rush, with a physical order rail between the servers and the line cooks. Orders get clipped onto that rail the moment a server submits them, and they stay there, visible and durable, until a cook actually plates the dish. If the grill station gets backed up, tickets pile up on the rail instead of vanishing — the rail is a strip of steel, not a whiteboard someone could accidentally wipe. That durability is exactly the property you want between the moment data enters a pipeline and the moment it's truly, safely delivered.
Logstash's persistent queue is that durable rail. By default, Logstash buffers events in memory between the input and filter/output stages, which is fast but evaporates completely on a crash or forced restart — anything accepted from an input but not yet handed off to the output is simply gone. Elastic added queue.type: persisted specifically for pipelines where that's unacceptable, such as billing or audit trails from a service like payments-api: accepted events are written to disk and acknowledged back to the input only after that write succeeds, so a JVM crash mid-processing doesn't silently drop in-flight data.
Internally, the flow is input, then the persistent queue (disk-backed, acknowledging writes), then a pool of filter worker threads (pipeline.workers) each pulling a batch of pipeline.batch.size events through the configured filter chain (grok, mutate, dissect), then an output plugin. Events that fail every filter match aren't silently discarded by default configuration choices you make — routing them to the dead letter queue (dlq.enable: true) preserves them for later replay once you've fixed whatever broke parsing, rather than losing them or blocking the whole batch behind one bad event. Backpressure flows naturally end to end: if the output is slow or rejecting writes, workers block trying to write results, the persistent queue fills toward queue.max_bytes, and that in turn slows how fast the input reads new data — the pipeline self-regulates instead of buffering unboundedly.
At scale, the metrics worth watching are queue.events (and its percentage of max_bytes, since a queue approaching its cap means the output genuinely can't keep up and needs scaling or fixing), pipeline worker CPU (workers pegged near 100% doing grok usually means a regex pattern is too expensive), and dead letter queue growth (a growing DLQ typically means an upstream schema change broke a grok pattern and needs active draining, not just ignoring). A persistent queue sized too small for a real burst just moves the failure earlier without fixing it.
The gotcha that catches experienced engineers: the persistent queue guarantees events aren't lost between the input and being handed to the output — it does not guarantee the output itself succeeded. If Elasticsearch or OpenSearch returns a partial bulk failure (some documents indexed, others rejected for a mapping conflict), Logstash's default handling can still acknowledge the whole batch off the queue unless you've explicitly configured retry-on-failure and DLQ routing for output-level errors specifically. 'I enabled the persistent queue so I can't lose data' is a dangerously incomplete assumption — it protects one hop of the journey, not the whole trip.