How do you optimize PromQL and recording rules to avoid query-caused outages?

Think of a city tax office processing returns. If every clerk individually calculates the city's total revenue by reading every single return, the office grinds to a halt. Instead, each department pre-computes its subtotal (recording rule), and the summary report (dashboard query) just adds up the subtotals. PromQL optimization follows the same idea -- pre-compute expensive aggregations so that real-time queries stay cheap.

High cardinality in Prometheus means having many unique label combinations for a single metric. A metric like http_request_duration_seconds with labels for service, method, path, status_code, pod, and customer_id can produce millions of unique time series. When a PromQL query touches all of these series -- for example, histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) without any label filter -- Prometheus must load, decode, and process every matching series from its TSDB. This can eat gigabytes of memory and minutes of CPU, blocking other queries and potentially crashing Prometheus with out-of-memory errors.

Internally, Prometheus processes PromQL in stages: label matching loads the posting lists (inverted index entries) for each label matcher, then the storage engine fetches the actual sample data for the matched series, and the query engine evaluates the expression. The key optimization point is label matching -- narrow matchers like {service="payments-api",path=~"/api/v2/orders.*"} shrink the posting list intersection dramatically compared to {service=~".*"}. Recording rules evaluate expensive queries at a fixed interval and store the result as a new time series. Instead of every dashboard panel and alert rule independently computing histogram_quantile across all pods, a recording rule computes it once per evaluation interval and stores it as service:http_request_duration_seconds:p99, which later queries read as a simple series lookup.

At production scale, architects should build a recording rule hierarchy: L1 recording rules aggregate raw per-pod metrics into per-service metrics, L2 rules compute rates and percentiles from L1 results, and L3 rules compute SLO burn rates from L2 results. Alert rules should reference recording rules rather than raw metrics so that alerts and dashboards always see identical values. The for duration in alerting rules should be at least 2-3 evaluation intervals to absorb scrape gaps and brief spikes -- a 1-minute for duration with a 30-second evaluation interval means the condition must be true for 2-3 consecutive evaluations before it fires. Query timeout (--query.timeout flag) and maximum sample limits (--query.max-samples) protect Prometheus from runaway queries that could take down the entire monitoring stack.

The sneaky gotcha is that recording rules themselves can cause high cardinality if the aggregation does not actually reduce dimensions. A recording rule that preserves the pod label still produces one series per pod, which could be thousands. The rule should aggregate by service, environment, and cluster -- labels that shrink to tens or hundreds of series -- not by pod or instance. Another trap is rate() and histogram_quantile() behaving differently than expected when the recording rule interval does not align with the rate window. If the interval is 120 seconds but the rate window is 30s, some data points may be missed because the lookback window is shorter than the gap between evaluations.