Which Filebeat internal metrics from the monitoring API actually predict an incoming outage, and what thresholds should trigger paging versus just a ticket?

A nurse watching a patient overnight doesn't run a full lab panel every five minutes — she watches a small set of vitals: heart rate, blood oxygen, blood pressure. A sudden drop in blood oxygen gets a doctor paged immediately, because it predicts a crisis in progress. A slightly elevated resting heart rate over several days gets noted for the morning rounds and a follow-up test, not a 3am page. The skill isn't collecting more data, it's knowing which few signals, trending in which direction, actually precede a crisis — and reserving the alarm bell for those, because paging on every fluctuation trains everyone to ignore the alarm.

Filebeat exposes dozens of counters through its monitoring HTTP endpoint (enabled with http.enabled: true, scraped locally or shipped via Metricbeat's filebeat module into Stack Monitoring), and most of them are useful for a post-incident ticket, not for waking someone up. The reason only a handful predict outages is that Filebeat's failure modes are narrow and specific — resource exhaustion (file descriptors, memory), pipeline stall (queue not draining), and transport failure (output rejecting or dropping events) — and each has one or two counters that move meaningfully before user-visible symptoms like missing logs in Kibana show up.

The metrics worth paging on: libbeat.output.events.dropped and output write-error counts — any nonzero value means real, current data loss, not a warning sign of future loss. The gap between libbeat.pipeline.events.published and libbeat.output.events.acked, trended over 10-15 minutes rather than as an instantaneous snapshot (single spikes during normal log rotation are expected and self-correct) — a gap that keeps growing without draining means backpressure that isn't resolving on its own. filebeat.harvester.open_files climbing toward the process's file-descriptor ulimit — Filebeat will start failing to open new files and effectively stop ingesting once it hits that ceiling. Ticket-worthy signals include beat.memstats.memory_alloc creeping upward over days (possible leak, investigate on your schedule) and occasional acked-lag blips that correlate with known deploy or rotation windows.

Building this into a real on-call setup means a dashboard per fleet (payments-api, checkout-worker, fraud-detection-svc) with the paging thresholds wired to dropped-events-greater-than-zero, sustained published/acked gap beyond a chosen window, and open_files beyond roughly 80% of the configured ulimit — each backed by a runbook, because a page without a runbook just creates panic. The ticket-tier thresholds get reviewed weekly rather than fired reactively, since their value is catching slow degradation (memory creep, a harvester count that never comes back down after a rotation storm) before it becomes page-worthy.

The gotcha many teams fall into is alerting solely on 'no new events shipped in N minutes' as their entire Filebeat health signal. That metric is fundamentally ambiguous: it fires identically whether payments-api simply had a quiet night with no traffic, or Filebeat itself is completely wedged and shipping nothing from a service that's actually generating errors right now. The fix experienced teams adopt is a synthetic heartbeat — a canary log line written on a fixed interval purely so its absence downstream is an unambiguous 'the Filebeat pipeline is broken end-to-end' signal, decoupled entirely from whatever real application traffic happens to be doing.