How would you design Prometheus alerting rules to avoid both alert fatigue and missed real incidents for an SLO-based error rate?

A naive alert like "error rate > 1% for 5 minutes" either pages too often on transient blips (alert fatigue, leading to ignored pages) or, if you lengthen the window to reduce noise, delays paging on a real fast-burning outage. The multi-window burn-rate technique (popularized by Google's SRE workbook) instead defines an error budget over an SLO window (e.g. 99.9% availability over 30 days) and computes how fast you're consuming that budget: a burn rate of 14.4x means you'd exhaust a 30-day budget in about 2 days if sustained.

The alert rule requires the SAME high burn rate to be true over both a short window (catches fast, severe incidents) and a longer window (confirms it's not just a 30-second blip that self-resolved) before paging:

(sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 14.4 * 0.001) and (sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > 14.4 * 0.001)

A second, lower-urgency ticket-not-page alert typically pairs a slower burn rate (e.g. 6x) with longer windows (e.g. 6h/3d) to catch slow-burning degradations that don't need a 2am page but do need attention before the budget is exhausted.

This approach is specifically designed to be evaluated as Prometheus recording rules (precompute the burn rate ratio, then alert on the recording rule) so the alerting rule itself stays cheap to evaluate even under load.