How do Prometheus staleness markers and the lookback window affect alerts when a target disappears?

Think of a train station display board. If the 8:10 train reported its location two minutes ago, the board can still show a useful last-known position. If the train has not reported for an hour, showing that old position would mislead passengers. Prometheus has the same problem with metrics: a recent sample is fine to use at query time, but an old sample should eventually disappear so graphs and alerts do not pretend the system is healthy.

PromQL, the Prometheus query language, evaluates instant queries at a single timestamp and range queries at many evenly spaced timestamps. For each evaluation timestamp, Prometheus looks backward for the newest sample inside the lookback window. The default lookback is 5 minutes, and it is configurable. This lets queries work even when scrapes do not land exactly on graph step boundaries. Without this behavior, normal scrape timing jitter would create broken graphs and unreliable aggregations.

Staleness adds another layer. If a target scrape no longer returns a series that previously existed, or if service discovery removes a target entirely, Prometheus can write a staleness marker for that time series. After that marker, instant queries no longer return the old value for that series. This prevents stale readings from being treated as current values in aggregations like sum, avg, or alert expressions. If fresh samples later arrive for the same label set, the series simply reappears.

Production alerting gets subtle here. An alert like up == 0 catches failed scrapes where the target is still known but unreachable. However, it may not catch a target that vanished from service discovery, because there may be no up series left to evaluate. For detecting missing services, absent() or inventory-based alerts are usually needed. Engineers also tune scrape_interval, scrape_timeout, evaluation_interval, and the alert for duration so brief network hiccups do not page people while true disappearances still get caught quickly.

The experienced gotcha is that a graph can look flat or empty for different reasons. A flat line may mean Prometheus is carrying a recent last sample inside the lookback window, while an empty graph may mean the series went stale, not that the value became zero. Exporters that attach their own timestamps can behave differently and may keep the last value visible until lookback expires. A common band-aid is using or vector(0) everywhere, which makes dashboards look tidy but hides missing telemetry. Senior engineers learn to distinguish between zero, missing, stale, and failed-scrape states explicitly rather than papering over the differences.