Filebeat

Think of the difference between a courier who picks packages up from your doorstep and drives them to a regional sorting depot, versus the depot itself. The courier's entire job is to show up reliably, pick up whatever is there, and get it to the depot intact and on time — they don't open boxes, they don't repackage contents, and they don't decide which truck a package should ultimately go on. The depot does that heavier work: scanning barcodes, sorting by destination, relabeling, consolidating shipments. You want thousands of couriers running cheaply and reliably across a city, but you only need a handful of well-resourced depots doing the complex sorting.

Filebeat is the courier. It's written in Go and compiles to a single static binary with a small, predictable memory footprint, specifically so it can run on every host, VM, or container — including resource-constrained edge nodes — without becoming a resource problem itself. It deliberately excludes a grok engine, complex conditional routing, and heavyweight transformation, because embedding that machinery into an agent that runs on every node watching every file would multiply your fleet-wide CPU and memory cost by however many nodes you operate. Logstash runs on the JVM and is built to do exactly that heavy lifting — grok parsing, mutate, ruby filters — which is fine because you run far fewer Logstash instances than Filebeat instances. Fluentd sits architecturally in between, with a plugin-based routing model and moderate processing capability.

This split is visible in Filebeat's harvester/input model. An input configuration block defines what to watch — a path glob for payments-api's log directory, a container autodiscover rule, a syslog listener — and for every matching file Filebeat starts a harvester, a lightweight per-file reader. Events pass through only libbeat's built-in processors: add_fields, drop_fields, dissect, decode_json_fields — all non-Turing-complete, bounded operations, not a full parsing language. The event is then handed to the internal queue and pushed to the configured output. There's no step anywhere in that pipeline where Filebeat evaluates an arbitrary regex-driven grok pattern against every line at scan time; that work happens later, either in Logstash's pipeline or an Elasticsearch ingest pipeline, applied centrally rather than per-node.

In production, this means you should expect Filebeat's CPU and memory usage to stay flat and low regardless of log volume nuance — if you see Filebeat's CPU spike, it's almost always a misused processor (commonly a script processor doing per-event logic it wasn't meant for) or a misconfigured multiline pattern, not the shipping itself. Teams running Filebeat across checkout-worker and fraud-detection-svc fleets typically budget well under 100MB RSS per instance and treat any deviation as a config smell, while Logstash or the ingest-pipeline layer is scaled and monitored separately as the actual processing tier.

The non-obvious trap is engineers trying to make Filebeat do Logstash's job — stacking dissect, decode_json_fields, and script processors to replicate complex conditional parsing — and then wondering why Filebeat's memory grows and file-descriptor churn increases. The fix isn't more Filebeat tuning; it's recognizing you've asked a courier to run the sorting depot, and moving that logic downstream. A related trap: people assume Filebeat ships 'whole files' the way you'd hand off a sealed package, but it actually reads line by line, so a single log line exceeding the configured line-size limit gets truncated rather than deferred — very different behavior from what the shipping analogy implies at first glance.

Imagine a metered highway on-ramp during rush hour. Cars queue up at the ramp meter when the highway ahead is congested — they're not gone, they're just later than usual, and eventually all of them get through once traffic clears. That's a normal, if annoying, backup. Now imagine the on-ramp has a hard capacity and no shoulder to wait on: once it's full, incoming cars are turned away entirely and never make the trip. From a distance, both situations look identical — traffic isn't coming through in real time — but one is a temporary delay and the other is permanent loss, and you cannot tell them apart just by watching the highway; you need to check specific gauges at the ramp itself.

Filebeat's internal queue is designed to behave like the metering ramp, not the capacity-capped one — by default it holds a bounded number of events in memory, and when that queue fills up, harvesters pause reading new lines from disk rather than discarding events. This is a deliberate at-least-once design choice: back-pressure is supposed to propagate all the way upstream to the file read itself, so overload manifests as delay, never as silent data loss. Real loss happens through a different mechanism entirely — most commonly when a log file gets rotated and deleted by logrotate faster than Filebeat's harvester can finish reading it, which is a file-lifecycle problem, not a queue problem.

To tell these apart, pull metrics from Filebeat's monitoring HTTP endpoint. A growing gap between libbeat.pipeline.events.published and libbeat.output.events.acked, with libbeat.output.events.dropped sitting at zero, means events are queued and delayed — classic backpressure, usually because the output (Elasticsearch or Logstash) is issuing bulk rejections or 429s faster than Filebeat can retry. Any nonzero value in dropped or in output write-error counters means events were actually discarded at the transport layer. Separately, compare registry offsets against the last known size of files that were rotated away: if a file was deleted while its recorded offset was still behind its final size, those unread bytes are gone permanently, with nothing in the pipeline metrics to flag it unless you're watching harvester-close and file-not-found log lines.

During checkout-worker's 10x spike, expect harvester count to climb as multiple rotated generations stay open simultaneously, the event queue to sit at its configured maximum, and the acked rate to flatline while Elasticsearch returns bulk rejections. The backpressure fix is capacity: raise queue.mem.events, tune bulk_max_size and worker count on the output, or scale the ingest tier itself. The permanent-loss fix is lifecycle coordination: make sure logrotate retains enough generations, and for long enough, that close_inactive and normal harvester read speed can always finish before a rotated file is actually deleted — and alert specifically on nonzero dropped/error metrics rather than on 'no new events,' which is ambiguous between healthy silence and a broken pipeline.

The gotcha senior engineers get bitten by: switching to Filebeat's disk-backed queue (queue.disk) to absorb bigger bursts without upstream blocking looks like it solves the backpressure problem, and for typical spikes it does — but the disk queue itself has a configurable size cap, and if nobody sizes or alerts on that, you've just moved the failure further downstream and made it quieter. Instead of harvesters visibly pausing, the host silently fills its disk with queued-but-unsent events until either the queue hits its cap (and now you're dropping) or the disk itself fills and Filebeat crash-loops — which reintroduces the exact 'file rotated out from under an unfinished harvester' loss scenario you were trying to avoid in the first place.

A nurse watching a patient overnight doesn't run a full lab panel every five minutes — she watches a small set of vitals: heart rate, blood oxygen, blood pressure. A sudden drop in blood oxygen gets a doctor paged immediately, because it predicts a crisis in progress. A slightly elevated resting heart rate over several days gets noted for the morning rounds and a follow-up test, not a 3am page. The skill isn't collecting more data, it's knowing which few signals, trending in which direction, actually precede a crisis — and reserving the alarm bell for those, because paging on every fluctuation trains everyone to ignore the alarm.

Filebeat exposes dozens of counters through its monitoring HTTP endpoint (enabled with http.enabled: true, scraped locally or shipped via Metricbeat's filebeat module into Stack Monitoring), and most of them are useful for a post-incident ticket, not for waking someone up. The reason only a handful predict outages is that Filebeat's failure modes are narrow and specific — resource exhaustion (file descriptors, memory), pipeline stall (queue not draining), and transport failure (output rejecting or dropping events) — and each has one or two counters that move meaningfully before user-visible symptoms like missing logs in Kibana show up.

The metrics worth paging on: libbeat.output.events.dropped and output write-error counts — any nonzero value means real, current data loss, not a warning sign of future loss. The gap between libbeat.pipeline.events.published and libbeat.output.events.acked, trended over 10-15 minutes rather than as an instantaneous snapshot (single spikes during normal log rotation are expected and self-correct) — a gap that keeps growing without draining means backpressure that isn't resolving on its own. filebeat.harvester.open_files climbing toward the process's file-descriptor ulimit — Filebeat will start failing to open new files and effectively stop ingesting once it hits that ceiling. Ticket-worthy signals include beat.memstats.memory_alloc creeping upward over days (possible leak, investigate on your schedule) and occasional acked-lag blips that correlate with known deploy or rotation windows.

Building this into a real on-call setup means a dashboard per fleet (payments-api, checkout-worker, fraud-detection-svc) with the paging thresholds wired to dropped-events-greater-than-zero, sustained published/acked gap beyond a chosen window, and open_files beyond roughly 80% of the configured ulimit — each backed by a runbook, because a page without a runbook just creates panic. The ticket-tier thresholds get reviewed weekly rather than fired reactively, since their value is catching slow degradation (memory creep, a harvester count that never comes back down after a rotation storm) before it becomes page-worthy.

The gotcha many teams fall into is alerting solely on 'no new events shipped in N minutes' as their entire Filebeat health signal. That metric is fundamentally ambiguous: it fires identically whether payments-api simply had a quiet night with no traffic, or Filebeat itself is completely wedged and shipping nothing from a service that's actually generating errors right now. The fix experienced teams adopt is a synthetic heartbeat — a canary log line written on a fixed interval purely so its absence downstream is an unambiguous 'the Filebeat pipeline is broken end-to-end' signal, decoupled entirely from whatever real application traffic happens to be doing.

Cities don't retime every traffic light simultaneously when rolling out a new signal-timing algorithm. They pilot it at a handful of intersections first, watch how traffic actually flows through them for a period, and only then expand to more intersections, because a bad citywide timing change causes gridlock everywhere at the exact same moment, during actual rush hour, with no way to quickly route around it. A city engineer who changed every light in town at once because the new algorithm 'tested fine on paper' would be making the same mistake as pushing an untested Filebeat config to every host in the fleet simultaneously.

Filebeat config changes are risky in a specific, non-obvious way: a bad multiline pattern (the regex-driven rule that decides whether a line continues the previous log entry, used to stitch multi-line stack traces back into one event) doesn't crash the process — it silently merges or splits events incorrectly, corrupting event boundaries across every host that has the new config, before anyone notices in Kibana. A bad new input path with wrong permissions similarly fails quietly, with harvesters unable to open files rather than throwing an obvious error. Because Filebeat can hot-reload modular input configs, a fleet-wide simultaneous push can also become a fleet-wide simultaneous failure mode within seconds, not the gradual failure you'd get from a slower deployment method.

The safe mechanics: use centralized config management (Ansible, Salt, or Fleet-managed Elastic Agent policies) with canary rings rather than a single fleet-wide push — 1%, then 10%, then 50%, then 100%, with a pause and metrics check between each ring. Run filebeat test config and filebeat test output before any push to catch syntax errors and connectivity problems without touching a live pipeline. For multiline changes, test the new regex against a sample of real historical log lines from order-fulfillment or fraud-detection-svc offline, not just a handful of synthetic examples. Ship input configs as drop-in files under an inputs.d directory with reload.enabled: true so Filebeat can hot-reload them — but verify in your specific version whether hot reload actually re-evaluates multiline logic on already-open harvesters, or only affects newly discovered files, since assuming the wrong behavior is a common source of surprises.

At fleet scale, wire the same monitoring-API metrics used for day-to-day health — harvester errors, config-reload failure counts (Filebeat logs a distinct error when a pushed config is invalid), and output.events.dropped — into the canary pipeline itself, segmented per ring, so a regression introduced in ring 1 is caught and blocks promotion before it ever reaches ring 4. Automate the rollback trigger: if error rate or dropped-events rate in the active canary ring exceeds baseline by a defined factor, halt promotion and revert that ring's config automatically rather than waiting for a human to notice a dashboard.

The failure senior engineers have genuinely been burned by: a multiline pattern change passes validation, looks correct in canary hosts for hours, then a real production incident on fraud-detection-svc produces an unusually deep, pathological stack trace that hits a regex edge case — catastrophic backtracking or a pattern that misbehaves specifically on deeply nested frames — spiking Filebeat's CPU right as the incident needs logs the most, turning a service incident into an observability blackout simultaneously. The mitigation is bounding multiline.max_lines and multiline.timeout aggressively so a pathological match can't run away, and load-testing the regex against deliberately ugly, unusual inputs before any fleet rollout, not just the typical stack traces you have on hand.

Picture a librarian who is simultaneously reading through hundreds of books, switching between them all day. Instead of trying to remember 'I was on page 42 of the blue book on the third shelf,' she keeps a notebook with one line per book, identified by that book's unique barcode rather than its title — because titles get reprinted, covers get replaced, and books get reshelved constantly. Each line records the exact page she reached. If she loses her place and picks the notebook back up, she can find the barcode and resume from that exact page, even if the book's cover looks completely different than it did yesterday. What she cannot survive is someone tearing pages out of a book faster than she can read them — those pages are simply gone before she gets to them.

In Filebeat, that notebook is the registry file, and the barcode is the combination of filesystem inode and device number rather than the file path. This design exists because production log files are rarely stable: logrotate renames access.log to access.log.1, Docker's json-log driver rotates container logs, and applications like payments-api or order-fulfillment truncate and recreate files on deploy. A filename-based bookmark would break constantly under rotation, either re-reading whole files from scratch or losing the tail of a renamed file mid-read. By keying off inode and device, Filebeat treats a renamed file as the same file it was already tracking, and can keep reading from the correct offset uninterrupted.

Internally, each configured input scans its paths and, for every matching file, spins up a harvester — a dedicated goroutine that opens the file and reads new lines as they're appended. Each line becomes an event that flows through libbeat's internal queue toward the configured output (Elasticsearch, Logstash, or Kafka). Crucially, Filebeat does not update the registry offset the moment it reads a line; it waits for an ACK (acknowledgment) from the output confirming the event was durably received. Only then does the registrar component persist the new offset to disk. This ACK-gated commit is what makes the delivery guarantee at-least-once: if Filebeat crashes between reading and receiving the ACK, it re-reads and re-sends those lines on restart rather than assuming they arrived.

At scale — say hundreds of hosts each running Filebeat against payments-api, checkout-worker, and fraud-detection-svc logs — the registry can accumulate thousands of entries, and registry write frequency becomes a real I/O concern, which is why modern Filebeat writes registry state incrementally to a directory rather than rewriting one giant file. Operators should monitor open harvester counts, the gap between events read and events acked, and registry flush latency. A registry that stops advancing while harvesters stay open is a strong signal of a stalled output or a downstream outage, not necessarily lost data — yet.

The gotcha experienced engineers eventually hit is truncation and inode reuse. If a log file is truncated in place (some naive rotation scripts do a hard truncate instead of proper copytruncate coordination) rather than renamed, Filebeat detects the file is now smaller than its recorded offset and resets to zero — but depending on timing, that can produce duplicate reads of whatever was written between truncation and detection. Worse, on filesystems or container runtimes that aggressively reuse inode numbers (fresh volumes after a pod restart, some NFS mounts), a brand-new file can inherit the same inode as a previously fully-processed and pruned file, and Filebeat will assume it's already been shipped — silently skipping real, new log data.