How do you find and fix a high-cardinality metric that is killing Prometheus performance?
Quick Answer
High cardinality means a metric has too many unique label combinations, like using user_id or request_id as labels, causing millions of time series. Spot it with `prometheus_tsdb_head_series` and the TSDB status page. Fix it by dropping the bad label via relabeling or removing the metric at its source.
Detailed Answer
Think of cardinality like a filing cabinet. Each unique set of labels creates a new folder. A metric like http_requests_total{method="GET", status="200"} creates one folder -- low cardinality. But http_requests_total{method="GET", user_id="abc123"} creates a brand new folder for every single user. If you have 10 million users, that is 10 million folders, and the filing cabinet explodes.
In Prometheus, every unique combination of metric name plus label key-value pairs creates a time series. Each active series uses about 1 to 2 KB of memory in the TSDB head block. At 1 million series, that is roughly 2 GB of RAM just for the head block. At 10 million, it is 20 GB. High-cardinality metrics are the number-one cause of Prometheus running out of memory and getting killed, as well as slow queries that time out.
Detection starts with watching Prometheus itself. The metric prometheus_tsdb_head_series tells you the total active series count. If this suddenly jumps, say from 500K to 5M, you have a cardinality explosion. The TSDB status page at /tsdb-status on the Prometheus web UI shows the top 10 metrics by series count and the top 10 labels by unique value count, which immediately reveals the offender. You can also run topk(10, count by (__name__)({__name__=~".+"})) to find the metrics with the most series.
Once you find the culprit, for example a developer who added request_id as a label on a counter, you fix it at multiple levels. The immediate fix is adding a metric_relabel_config in the Prometheus scrape config to drop the high-cardinality label or the entire metric. This takes effect on the next scrape cycle. The medium-term fix is changing the application code to stop emitting the label and using histograms or summaries instead of per-ID counters. The long-term fix is setting up guardrails using tools like Prom-Label-Proxy or Mimir's per-tenant cardinality limits to prevent future explosions.
One important detail: you cannot simply delete old high-cardinality series from Prometheus. Once they are in the TSDB, they stay until the retention period expires. You can use the Admin API at /api/v1/admin/tsdb/delete_series to mark them for deletion, then run clean_tombstones to reclaim space. But this is an emergency measure -- the real fix is always upstream at the source of the metric. The 'context deadline exceeded' error in Prometheus usually means a query is trying to process too many series and timing out, which is a direct symptom of high cardinality.