What's the difference between Prometheus recording rules and alerting rules, and why would heavy PromQL queries be moved into recording rules?
Quick Answer
Recording rules precompute a PromQL expression on a schedule and save the result as a new time series, so expensive or frequently-viewed queries (like a fleet-wide p99 histogram_quantile or an SLO burn rate) are calculated once centrally instead of recomputed every time a dashboard or alert evaluates them. Alerting rules also evaluate a PromQL expression on a schedule, but instead of storing the result as a metric, they fire alerts to Alertmanager when the expression's result is non-empty.
Detailed Answer
Both rule types share the same rules file syntax and evaluation interval mechanism, but serve different purposes. A recording rule like record: job:http_request_duration_seconds:p99 with an expensive histogram_quantile expression runs once per evaluation interval and stores the result under that new metric name — every dashboard panel or alert that needs that p99 then reads the cheap precomputed series instead of re-running the expensive aggregation across raw histogram buckets on every page load or every alert evaluation cycle. This matters a lot at scale: a histogram_quantile over sum(rate(...)) across thousands of series, evaluated by both a dashboard refreshing every 10s and several alert rules, would otherwise redo that expensive aggregation redundantly and repeatedly.
Alerting rules use the same underlying evaluation loop, but their expr result determines whether an alert instance is created (with labels/annotations for Alertmanager) — pending, then firing after for: duration is satisfied continuously. Best practice is to have alerting rules reference recording rules rather than raw expensive expressions directly, both for performance and so the exact same computed number appears consistently in the alert, the runbook link, and the dashboard.
A common naming convention (from Prometheus docs) is level:metric:operations, e.g. job:http_inprogress_requests:sum, making it clear at a glance what aggregation level a recording rule represents.
Code Example
groups:
- name: slo_recording
rules:
- record: job:http_errors:burnrate5m
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job)Interview Tip
Mention that alert rules should reference recording rules, not raw expressions — that's the detail that shows real production experience versus textbook knowledge.