What are recording rules and alerting rules in Prometheus, and how do they differ?

Think of recording rules like a restaurant that preps ingredients before the dinner rush. Instead of chopping vegetables from scratch for every order, the kitchen pre-chops during quiet hours. Recording rules pre-compute expensive PromQL queries on a schedule so dashboards load instantly. Alerting rules are like a smoke detector: they continuously check a condition and sound the alarm when something crosses a threshold for long enough to be a real problem.

Both types of rules are defined in YAML files and loaded by Prometheus through the rule_files config. They are organized into rule groups, where each group has a name and an optional evaluation interval. Recording rules have a record field (the name of the new metric to create) and an expr field (the PromQL expression to evaluate). The naming convention follows the pattern level:metric:operations -- for example, namespace:http_requests_total:rate5m tells you the aggregation level, the base metric, and what operation was applied. Alerting rules have an alert field (the alert name), an expr field, an optional for duration, labels to attach, and annotations for human-readable descriptions.

Under the hood, Prometheus evaluates rules within each group sequentially but can run multiple groups in parallel. The evaluation interval defaults to the global setting but can be overridden per group. For recording rules, each evaluation writes a new sample to the TSDB with the current timestamp. For alerting rules, the evaluation produces one of three states: inactive (the expression returned nothing), pending (the expression matched but the for duration has not passed yet), or firing (the expression has been true for at least the for duration). When an alert hits firing state, Prometheus sends it to all configured Alertmanagers.

In production, recording rules are essential for scaling dashboards. Without them, 50 engineers opening the same Grafana dashboard during an incident would each trigger the same expensive aggregation 50 times per refresh. Recording rules compute it once and store the result. A common pattern is building a pyramid: raw metrics get aggregated into per-service rates, then those rates get aggregated into per-team totals. For alerting, the for clause is critical -- it prevents false alarms from momentary spikes. A for: 5m clause means the condition must be continuously true for 5 minutes before the alert fires.

A key gotcha with recording rules is circular dependencies. If rule A depends on the output of rule B, both must be in the same group with B listed first, because rules within a group run sequentially. Across groups, evaluation order is not guaranteed. For alerting rules, a common mistake is leaving out the for clause entirely, which causes alerts to fire on every brief spike. Another pitfall is hardcoding values in annotations instead of using template variables. Always include {{ $labels.instance }} and {{ $value }} in your annotation templates so on-call engineers can immediately see which target is affected and how bad it is.