Why can Prometheus be unavailable for several minutes right after an OOM restart, and what does that mean for alerting during an incident?
Quick Answer
On restart, Prometheus replays its write-ahead log (WAL) to rebuild the in-memory head block before it can serve queries or evaluate alerting rules — with a large head block (millions of series) this WAL replay can take minutes, during which dashboards return no/incomplete data and alert rules aren't being evaluated at all. This means the very OOM event that indicates a problem also silences the alerting that would normally page someone about it.
Detailed Answer
Prometheus's local TSDB writes recent samples to an on-disk WAL before they're indexed into the in-memory head block, so that a crash doesn't lose recent data. On startup, Prometheus must replay this WAL from disk to reconstruct the head block state before it considers itself ready — this is a CPU- and memory-intensive step whose duration scales with how much data was in the head block (and therefore with cardinality) at the time of the crash. A Prometheus with tens of millions of active series can take several minutes to replay, during which /-/ready returns not-ready, scrapes may be delayed or skipped, and alerting rule evaluation is paused.
The operational consequence is a nasty compounding failure: the same cardinality blowup that caused the OOM also makes the WAL replay slower, and while replay is in progress, Alertmanager isn't receiving any new evaluations from this Prometheus, so a real ongoing outage can go unpaged for the entire replay window. Teams that have hit this run Alertmanager/Prometheus redundantly (two independent Prometheus + Alertmanager pairs scraping the same targets) specifically so one instance's OOM-restart cycle doesn't create an alerting blackout, and add a meta-alert (from a separate, independent monitoring path) that fires if a Prometheus instance's up or prometheus_tsdb_head_series hasn't reported recently.
Interview Tip
This question is really testing whether you understand that monitoring systems need their own monitoring — mention watching Prometheus from an independent second system as the real fix.