Logstash

Think of a car wash with three stations in sequence: a pre-spray station that just gets cars wet, a scrub-and-dry station where people do the actual labor-intensive cleaning, and a final rinse station before cars roll out. If the scrub station has only one person working while cars arrive quickly, cars back up in the queue between spray and scrub — even though the spray and rinse stations are sitting completely idle, waiting for their turn. The bottleneck isn't spread evenly across the car wash; it's concentrated wherever the manual, expensive work happens.

Logstash was deliberately built with this three-stage separation — input plugins, filter plugins, output plugins — precisely so that acquisition, transformation, and delivery could each grow a rich, independently swappable plugin ecosystem (beats, kafka, syslog, and dozens more as inputs; elasticsearch, opensearch, s3, and dozens more as outputs) without those concerns being tangled into one monolithic script. It mirrors classic Unix pipeline philosophy: each stage does one job well, and data flows through a queue between stages rather than being handled by a single do-everything component.

Internally, an input plugin — say the beats input receiving from a Filebeat fleet shipping checkout-worker logs — hands raw events to Logstash's internal queue. A pool of filter worker threads (pipeline.workers) each pull a batch and run it through the configured filter chain in order: commonly grok (regex-based pattern extraction, notoriously CPU-expensive because it tries multiple pattern alternatives against every line), mutate (field renaming and type conversion, cheap), and date (timestamp parsing, moderate cost). Only after that chain completes does the transformed event reach an output plugin, which typically batches multiple events into one bulk API call for efficiency on the way out.

When the filter stage becomes the bottleneck in production, you see rising queue_push_duration metrics and workers pinned near 100% CPU, while the output plugin's own throughput — bulk requests per second against OpenSearch, say — stays flat or low, simply because it isn't being fed events fast enough to matter. The fix path is almost never 'add more workers' as a first move: it's reducing grok complexity (anchoring patterns with ^, avoiding greedy wildcards, ordering more specific patterns first so they can short-circuit), switching to dissect wherever the log format is fixed-position rather than genuinely needing regex flexibility, or scaling out horizontally with additional Logstash nodes behind a load balancer once the per-event cost itself is already reasonable.

The non-obvious gotcha: throwing more pipeline.workers at a filter bottleneck doesn't help if the grok patterns themselves are pathologically slow — a phenomenon resembling catastrophic backtracking in regex engines, where certain input shapes cause the pattern matcher to explore an exponential number of paths before matching or failing. You can end up with every worker thread pegged at 100% CPU, the JVM's garbage collector working harder under the memory churn, and adding more workers just means more threads simultaneously fighting for the same scarce CPU rather than actually increasing throughput. The real fix is almost always simplifying or replacing the expensive pattern, not adding parallelism around it.

Imagine an overzealous proofreader given one sloppy instruction: check every possible way this sentence could be parsed before deciding what it means. For a short, clear sentence, that's instant. But for a long, ambiguous paragraph full of repeated words and delimiters, the number of 'possible ways to read it' the proofreader tries to consider can explode combinatorially, and they get stuck rereading the same paragraph for minutes, burning all their attention on one sentence while a stack of other work piles up behind them, untouched.

That's what happens inside Logstash's grok filter, built on the Oniguruma and Joni regex engines, when a pattern is too permissive. Grok patterns are named regex shortcuts, like %{IP} or %{GREEDYDATA}, composed together, designed to make regex-based log parsing approachable without hand-writing raw regular expressions. But grok doesn't protect you from writing a combination of patterns that triggers catastrophic backtracking, where the regex engine, faced with ambiguous or nested repetition, tries an exponential number of ways to match before giving up or succeeding on a single line.

The diagnostic sequence starts with the monitoring API: GET _node/stats/pipelines/<name> shows which pipeline's filter stage has abnormally high duration_in_millis per event, correlated with top -H showing Logstash's JVM threads pegged at CPU. From there, isolate the specific pattern by testing candidate log lines against each configured grok pattern individually, using the Grok Debugger or by temporarily commenting out patterns and watching for _grokparsefailure tags disappearing or CPU dropping — looking specifically for combinations of GREEDYDATA or .* sitting next to other loosely-bounded patterns, especially where the log line contains repeated delimiter characters that create many equally-valid ways to split the string.

In production this shows up as every pipeline.workers thread pinned at 100% CPU, events.duration_in_millis climbing while events.out throughput craters, and the persistent queue (if configured) filling because filters can't drain it fast enough — for a payments-api log pipeline this cascades quickly since transaction logs are both high-volume and time-sensitive. Ongoing prevention means routing any new grok pattern through a staging pipeline fed with representative production log volume before it ever reaches the main pipeline, and preferring dissect over grok wherever the log format has a fixed, predictable structure, since dissect does simple string splitting with zero backtracking risk.

The non-obvious gotcha: a grok pattern that has run fine for months can suddenly start pegging CPU not because the pattern itself changed, but because an upstream application changed its log format slightly — checkout-worker might start emitting a longer, more repetitive error message inside a field that a GREEDYDATA pattern was matching — and the same 'safe' pattern that worked fine on short messages now triggers exponential backtracking on the new, longer ones. Grok performance is never purely a property of the pattern in isolation; it depends on the actual data shape flowing through it, which is why any pattern audit needs representative production samples, not unit-test-style short strings that never expose the edge case.

Picture a bathtub with the faucet running and the drain partially clogged. The water level doesn't jump instantly — it creeps up gradually as inflow slightly exceeds outflow, and if you're only glancing at the tub occasionally, you might not notice the slow rise until it's nearly overflowing. But if you're watching the actual flow rates, how much water is coming in per minute versus draining out per minute, you'd catch the mismatch within the first few minutes, long before the water level itself becomes visually alarming.

Logstash exposes exactly these flow rates through its monitoring API and the _node/stats endpoint, because Elastic designed the pipeline to be introspectable at each stage specifically so operators don't have to wait for a hard failure — queue full, disk full, process OOM — to know something is wrong. The core insight baked into this design is that a pipeline can look perfectly healthy by a simple up/down check while quietly losing the race between ingestion rate and processing rate for many minutes before anything actually crashes.

Internally, each pipeline tracks cumulative events.in, events.filtered, and events.out counters, plus per-plugin timing where duration_in_millis divided by event count gives the average per-event processing time for each input, filter, and output plugin. Sampling these counters every 30 to 60 seconds and computing rates gives you in-rate versus out-rate directly; if in-rate consistently exceeds out-rate, the difference is accumulating somewhere — either in the persistent queue on disk, the in-memory queue, or, worst case with a memory queue and no backpressure protection configured, getting silently dropped once buffers overflow.

In production, dashboards typically graph events.in per second against events.out per second per pipeline side by side, persistent queue disk usage as a percentage of queue.max_bytes, and per-filter average duration on the same timeline. Alerting rules page when in-rate exceeds out-rate by more than some margin, say 10 percent, sustained for five or more minutes, since that's a genuine leading indicator — versus ticketing on slower creeping trends like gradually increasing per-event filter duration, which is often an early sign of a grok pattern becoming pathological as upstream data shape drifts.

The non-obvious gotcha: events.out counts events leaving Logstash's own filter and output stage — it does not necessarily mean the output destination actually persisted them successfully, especially with output configurations that don't enforce strict acknowledgment on every write. A healthy-looking in-rate to out-rate balance inside Logstash can coexist with a real data-loss problem happening one hop further downstream at the storage layer, which is why pipeline metrics need to be paired with destination-side ingestion metrics, like OpenSearch's own indexing rate and rejected-document count, to get the complete picture rather than a partial one that looks reassuring in isolation.

Think about a surgeon who wants to try a new suturing technique. They don't test it for the first time on a live patient in the operating room — they practice on a synthetic model or a cadaver first, confirm the technique works and doesn't cause unexpected damage, and only then apply it to a real patient, usually starting with lower-risk cases before it becomes their default approach for everything they do.

Logstash config changes deserve the same discipline, because a malformed or subtly wrong filter doesn't just fail to start cleanly — depending on the specific mistake, it can either fail fast, which is the good outcome, or silently misprocess or drop events across an entire pipeline serving multiple services at once. Elastic built --config.test_and_exit specifically as an offline syntax and structural check, separate from actually running the pipeline, precisely so teams have a fast, side-effect-free way to catch obvious mistakes — unbalanced braces, invalid plugin options, unknown filter names — before touching anything live.

The safe rollout path: first run bin/logstash --config.test_and_exit -f new_pipeline.conf to catch syntax errors. Because syntax validity doesn't guarantee correct behavior, next run the candidate config in an isolated test pipeline, defined as a second entry in pipelines.yml, fed with a recorded sample of real production events — a file input replaying a saved batch of actual checkout-worker or user-auth-service log lines — and inspect the resulting output events for correct field extraction. Only after confirming expected output do you deploy to the actual production pipeline definition, ideally with config.reload.automatic: true, so Logstash picks up the new file, and critically, if the new config fails to compile at reload time, Logstash logs the error and continues running the OLD compiled pipeline rather than crashing — this fail-safe-to-old-config behavior is a major reason to prefer reload over a full process restart for routine config changes.

In production, you monitor the Logstash log for Reloading and Failed to reload messages immediately after any deploy, watch events.out rate before and after the change to confirm the new config isn't silently dropping or misrouting events, and keep the previous config version easily accessible in git, tagged by deploy, for instant rollback by re-triggering a reload with the reverted file. Teams that skip the isolated test-pipeline step often discover mapping or type mismatches only after the new config has already been indexing malformed data into OpenSearch for hours, which is a much more expensive fix than catching it beforehand.

The non-obvious gotcha: config.reload.automatic watches for file changes but does so on a debounce or polling interval, checking every few seconds by default, and if your deployment tooling writes the new config in a way that produces a transient, invalid intermediate state — an in-place multi-line edit rather than an atomic rename — Logstash can pick up a half-written config file mid-write and reload with a genuinely broken pipeline before your deployment tool has even finished writing the complete file. Always deploy config via atomic file replacement, writing to a temp file and then renaming it into place, never by streaming an in-place edit to the live config path.

Picture a busy restaurant kitchen during a dinner rush, with a physical order rail between the servers and the line cooks. Orders get clipped onto that rail the moment a server submits them, and they stay there, visible and durable, until a cook actually plates the dish. If the grill station gets backed up, tickets pile up on the rail instead of vanishing — the rail is a strip of steel, not a whiteboard someone could accidentally wipe. That durability is exactly the property you want between the moment data enters a pipeline and the moment it's truly, safely delivered.

Logstash's persistent queue is that durable rail. By default, Logstash buffers events in memory between the input and filter/output stages, which is fast but evaporates completely on a crash or forced restart — anything accepted from an input but not yet handed off to the output is simply gone. Elastic added queue.type: persisted specifically for pipelines where that's unacceptable, such as billing or audit trails from a service like payments-api: accepted events are written to disk and acknowledged back to the input only after that write succeeds, so a JVM crash mid-processing doesn't silently drop in-flight data.

Internally, the flow is input, then the persistent queue (disk-backed, acknowledging writes), then a pool of filter worker threads (pipeline.workers) each pulling a batch of pipeline.batch.size events through the configured filter chain (grok, mutate, dissect), then an output plugin. Events that fail every filter match aren't silently discarded by default configuration choices you make — routing them to the dead letter queue (dlq.enable: true) preserves them for later replay once you've fixed whatever broke parsing, rather than losing them or blocking the whole batch behind one bad event. Backpressure flows naturally end to end: if the output is slow or rejecting writes, workers block trying to write results, the persistent queue fills toward queue.max_bytes, and that in turn slows how fast the input reads new data — the pipeline self-regulates instead of buffering unboundedly.

At scale, the metrics worth watching are queue.events (and its percentage of max_bytes, since a queue approaching its cap means the output genuinely can't keep up and needs scaling or fixing), pipeline worker CPU (workers pegged near 100% doing grok usually means a regex pattern is too expensive), and dead letter queue growth (a growing DLQ typically means an upstream schema change broke a grok pattern and needs active draining, not just ignoring). A persistent queue sized too small for a real burst just moves the failure earlier without fixing it.

The gotcha that catches experienced engineers: the persistent queue guarantees events aren't lost between the input and being handed to the output — it does not guarantee the output itself succeeded. If Elasticsearch or OpenSearch returns a partial bulk failure (some documents indexed, others rejected for a mapping conflict), Logstash's default handling can still acknowledge the whole batch off the queue unless you've explicitly configured retry-on-failure and DLQ routing for output-level errors specifically. 'I enabled the persistent queue so I can't lose data' is a dangerously incomplete assumption — it protects one hop of the journey, not the whole trip.