Write a PromQL query to calculate the error rate for a service and set up a multi-window burn rate alert based on SLO. Explain why burn rate alerting is better than threshold alerting.
Quick Answer
Error rate: `sum(rate(http_requests_total{status=~'5..'}[5m])) / sum(rate(http_requests_total[5m]))`. Burn rate alerts use multiple time windows to detect SLO violations early without alerting on brief spikes.
Detailed Answer
Basic Error Rate
``promql sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ``
Why Burn Rate > Threshold
Threshold alerting (e.g., alert if error rate > 1%) has two problems
1. Too sensitive: Brief 2-minute spikes trigger pages that resolve before you investigate 2. Not SLO-aware: A 1% error rate for 5 minutes is very different from 1% for 5 hours
Burn rate measures how fast you're consuming your error budget relative to the SLO period.
If your SLO is 99.9% over 30 days, your error budget is 0.1% × 30 days = 43.2 minutes of downtime. - Burn rate 1 = consuming budget at exactly the rate to exhaust it in 30 days - Burn rate 14.4 = consuming budget so fast it'll be gone in 2 days - Burn rate 6 = gone in 5 days
Multi-Window Burn Rate Alert
Use two windows per alert: a long window (catches sustained issues) and a short window (ensures issue is still happening):
| Severity | Burn Rate | Long Window | Short Window | Budget Consumed | |----------|-----------|-------------|--------------|------------------| | Page | 14.4x | 1 hour | 5 minutes | 2% in 1h | | Page | 6x | 6 hours | 30 minutes | 5% in 6h | | Ticket | 3x | 1 day | 2 hours | 10% in 1d | | Ticket | 1x | 3 days | 6 hours | 10% in 3d |
Code Example
# PromQL: Multi-window burn rate alert
# SLO: 99.9% (error budget = 0.001)
# 1-hour burn rate
(
sum(rate(http_requests_total{status=~"5.."}[1h]))
/
sum(rate(http_requests_total[1h]))
) / 0.001 # divide by error budget to get burn rate
# Alert rule: Page if burn rate > 14.4 over 1h AND > 14.4 over 5m
groups:
- name: slo-burn-rate
rules:
- alert: HighBurnRate_Page
expr: |
(
sum(rate(http_requests_total{status=~"5.."}[1h]))
/ sum(rate(http_requests_total[1h]))
) / 0.001 > 14.4
and
(
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
) / 0.001 > 14.4
labels:
severity: pageInterview Tip
This is a Google SRE must-know topic. Explain the math: why 14.4x burn rate = budget exhausted in 2 days (30 days / 14.4 ≈ 2.08 days). The multi-window approach (long + short) eliminates both false positives (brief spikes) and stale alerts (issue already resolved).