Why is Filebeat called a 'shipper' rather than a processor, and how does that shape its harvester/input model compared to Logstash or Fluentd?

Think of the difference between a courier who picks packages up from your doorstep and drives them to a regional sorting depot, versus the depot itself. The courier's entire job is to show up reliably, pick up whatever is there, and get it to the depot intact and on time — they don't open boxes, they don't repackage contents, and they don't decide which truck a package should ultimately go on. The depot does that heavier work: scanning barcodes, sorting by destination, relabeling, consolidating shipments. You want thousands of couriers running cheaply and reliably across a city, but you only need a handful of well-resourced depots doing the complex sorting.

Filebeat is the courier. It's written in Go and compiles to a single static binary with a small, predictable memory footprint, specifically so it can run on every host, VM, or container — including resource-constrained edge nodes — without becoming a resource problem itself. It deliberately excludes a grok engine, complex conditional routing, and heavyweight transformation, because embedding that machinery into an agent that runs on every node watching every file would multiply your fleet-wide CPU and memory cost by however many nodes you operate. Logstash runs on the JVM and is built to do exactly that heavy lifting — grok parsing, mutate, ruby filters — which is fine because you run far fewer Logstash instances than Filebeat instances. Fluentd sits architecturally in between, with a plugin-based routing model and moderate processing capability.

This split is visible in Filebeat's harvester/input model. An input configuration block defines what to watch — a path glob for payments-api's log directory, a container autodiscover rule, a syslog listener — and for every matching file Filebeat starts a harvester, a lightweight per-file reader. Events pass through only libbeat's built-in processors: add_fields, drop_fields, dissect, decode_json_fields — all non-Turing-complete, bounded operations, not a full parsing language. The event is then handed to the internal queue and pushed to the configured output. There's no step anywhere in that pipeline where Filebeat evaluates an arbitrary regex-driven grok pattern against every line at scan time; that work happens later, either in Logstash's pipeline or an Elasticsearch ingest pipeline, applied centrally rather than per-node.

In production, this means you should expect Filebeat's CPU and memory usage to stay flat and low regardless of log volume nuance — if you see Filebeat's CPU spike, it's almost always a misused processor (commonly a script processor doing per-event logic it wasn't meant for) or a misconfigured multiline pattern, not the shipping itself. Teams running Filebeat across checkout-worker and fraud-detection-svc fleets typically budget well under 100MB RSS per instance and treat any deviation as a config smell, while Logstash or the ingest-pipeline layer is scaled and monitored separately as the actual processing tier.

The non-obvious trap is engineers trying to make Filebeat do Logstash's job — stacking dissect, decode_json_fields, and script processors to replicate complex conditional parsing — and then wondering why Filebeat's memory grows and file-descriptor churn increases. The fix isn't more Filebeat tuning; it's recognizing you've asked a courier to run the sorting depot, and moving that logic downstream. A related trap: people assume Filebeat ships 'whole files' the way you'd hand off a sealed package, but it actually reads line by line, so a single log line exceeding the configured line-size limit gets truncated rather than deferred — very different behavior from what the shipping analogy implies at first glance.