Under a sudden 10x spike in log volume on checkout-worker, dashboards show gaps in Kibana — how do you determine whether Filebeat is silently dropping events versus merely delayed by backpressure, and how do you fix each?

Imagine a metered highway on-ramp during rush hour. Cars queue up at the ramp meter when the highway ahead is congested — they're not gone, they're just later than usual, and eventually all of them get through once traffic clears. That's a normal, if annoying, backup. Now imagine the on-ramp has a hard capacity and no shoulder to wait on: once it's full, incoming cars are turned away entirely and never make the trip. From a distance, both situations look identical — traffic isn't coming through in real time — but one is a temporary delay and the other is permanent loss, and you cannot tell them apart just by watching the highway; you need to check specific gauges at the ramp itself.

Filebeat's internal queue is designed to behave like the metering ramp, not the capacity-capped one — by default it holds a bounded number of events in memory, and when that queue fills up, harvesters pause reading new lines from disk rather than discarding events. This is a deliberate at-least-once design choice: back-pressure is supposed to propagate all the way upstream to the file read itself, so overload manifests as delay, never as silent data loss. Real loss happens through a different mechanism entirely — most commonly when a log file gets rotated and deleted by logrotate faster than Filebeat's harvester can finish reading it, which is a file-lifecycle problem, not a queue problem.

To tell these apart, pull metrics from Filebeat's monitoring HTTP endpoint. A growing gap between libbeat.pipeline.events.published and libbeat.output.events.acked, with libbeat.output.events.dropped sitting at zero, means events are queued and delayed — classic backpressure, usually because the output (Elasticsearch or Logstash) is issuing bulk rejections or 429s faster than Filebeat can retry. Any nonzero value in dropped or in output write-error counters means events were actually discarded at the transport layer. Separately, compare registry offsets against the last known size of files that were rotated away: if a file was deleted while its recorded offset was still behind its final size, those unread bytes are gone permanently, with nothing in the pipeline metrics to flag it unless you're watching harvester-close and file-not-found log lines.

During checkout-worker's 10x spike, expect harvester count to climb as multiple rotated generations stay open simultaneously, the event queue to sit at its configured maximum, and the acked rate to flatline while Elasticsearch returns bulk rejections. The backpressure fix is capacity: raise queue.mem.events, tune bulk_max_size and worker count on the output, or scale the ingest tier itself. The permanent-loss fix is lifecycle coordination: make sure logrotate retains enough generations, and for long enough, that close_inactive and normal harvester read speed can always finish before a rotated file is actually deleted — and alert specifically on nonzero dropped/error metrics rather than on 'no new events,' which is ambiguous between healthy silence and a broken pipeline.

The gotcha senior engineers get bitten by: switching to Filebeat's disk-backed queue (queue.disk) to absorb bigger bursts without upstream blocking looks like it solves the backpressure problem, and for typical spikes it does — but the disk queue itself has a configurable size cap, and if nobody sizes or alerts on that, you've just moved the failure further downstream and made it quieter. Instead of harvesters visibly pausing, the host silently fills its disk with queued-but-unsent events until either the queue hits its cap (and now you're dropping) or the disk itself fills and Filebeat crash-loops — which reintroduces the exact 'file rotated out from under an unfinished harvester' loss scenario you were trying to avoid in the first place.