A team added a user_id label to a request counter. Three months later Prometheus OOM-kills. Walk through why, and how you'd both fix and prevent it.

Prometheus keeps every actively-scraped time series' recent samples in memory (the "head block") before it's compacted to disk, and each series has non-trivial fixed overhead for its labels, chunk metadata, and index entries — commonly cited estimates are 1-3KB of memory per series regardless of sample frequency. A label like user_id, session_id, or request_id has effectively unbounded cardinality: every distinct value multiplies the number of series for that metric. A counter that was a handful of series (one per service instance) becomes millions of series as the user base grows, and none of them are ever explicitly deleted while the underlying series is still being scraped (a series only becomes eligible for cleanup after it stops being reported for the retention window).

Immediate fix: identify the offending metric/label via topk(10, count by (__name__)({__name__=~".+"})) or Prometheus's own prometheus_tsdb_head_series and per-metric cardinality tooling, then add a metric_relabel_configs rule with action: labeldrop on that label, and restart or wait for the head block to shed old series. This is usually a 10-minute config change once identified — the hard part is the multi-hour investigation to realize a label, not a metric, is the culprit, since dashboards just show "Prometheus is unhealthy," not which label is responsible.

Prevention: treat label cardinality as a reviewed part of any metric addition (a label's value set should be small and bounded — status codes, regions, pod names in a bounded fleet — never user/session/request identifiers); set up cardinality alerting on scrape_series_added and prometheus_tsdb_head_series trending upward; and use --storage.tsdb.retention.size plus per-metric cardinality dashboards so a bad label is caught in days, not months.