Why does Logstash structure processing as an input to filter to output pipeline internally, and what happens when the filter stage becomes the bottleneck?
Quick Answer
Logstash separates ingestion, transformation, and delivery into distinct stages so each can use a rich, swappable plugin ecosystem independently — dozens of input types, dozens of output types — without tangling data acquisition with parsing logic. The filter stage, especially grok's regex-based parsing, is almost always the CPU bottleneck because pattern matching against unstructured text is expensive, while input and output are largely I/O-bound; when filters saturate CPU, events queue up behind them and total pipeline throughput drops to whatever the filters can chew through, regardless of how fast the input or output could otherwise go.
Detailed Answer
Think of a car wash with three stations in sequence: a pre-spray station that just gets cars wet, a scrub-and-dry station where people do the actual labor-intensive cleaning, and a final rinse station before cars roll out. If the scrub station has only one person working while cars arrive quickly, cars back up in the queue between spray and scrub — even though the spray and rinse stations are sitting completely idle, waiting for their turn. The bottleneck isn't spread evenly across the car wash; it's concentrated wherever the manual, expensive work happens.
Logstash was deliberately built with this three-stage separation — input plugins, filter plugins, output plugins — precisely so that acquisition, transformation, and delivery could each grow a rich, independently swappable plugin ecosystem (beats, kafka, syslog, and dozens more as inputs; elasticsearch, opensearch, s3, and dozens more as outputs) without those concerns being tangled into one monolithic script. It mirrors classic Unix pipeline philosophy: each stage does one job well, and data flows through a queue between stages rather than being handled by a single do-everything component.
Internally, an input plugin — say the beats input receiving from a Filebeat fleet shipping checkout-worker logs — hands raw events to Logstash's internal queue. A pool of filter worker threads (pipeline.workers) each pull a batch and run it through the configured filter chain in order: commonly grok (regex-based pattern extraction, notoriously CPU-expensive because it tries multiple pattern alternatives against every line), mutate (field renaming and type conversion, cheap), and date (timestamp parsing, moderate cost). Only after that chain completes does the transformed event reach an output plugin, which typically batches multiple events into one bulk API call for efficiency on the way out.
When the filter stage becomes the bottleneck in production, you see rising queue_push_duration metrics and workers pinned near 100% CPU, while the output plugin's own throughput — bulk requests per second against OpenSearch, say — stays flat or low, simply because it isn't being fed events fast enough to matter. The fix path is almost never 'add more workers' as a first move: it's reducing grok complexity (anchoring patterns with ^, avoiding greedy wildcards, ordering more specific patterns first so they can short-circuit), switching to dissect wherever the log format is fixed-position rather than genuinely needing regex flexibility, or scaling out horizontally with additional Logstash nodes behind a load balancer once the per-event cost itself is already reasonable.
The non-obvious gotcha: throwing more pipeline.workers at a filter bottleneck doesn't help if the grok patterns themselves are pathologically slow — a phenomenon resembling catastrophic backtracking in regex engines, where certain input shapes cause the pattern matcher to explore an exponential number of paths before matching or failing. You can end up with every worker thread pegged at 100% CPU, the JVM's garbage collector working harder under the memory churn, and adding more workers just means more threads simultaneously fighting for the same scarce CPU rather than actually increasing throughput. The real fix is almost always simplifying or replacing the expensive pattern, not adding parallelism around it.