How do you set up alerting that reduces noise — what makes a good alert vs a bad one?

Think of alerting like the alarm system in a bank vault. A well-designed system has a silent alarm for suspicious activity that notifies security discreetly, a loud alarm for an active breach that triggers lockdown, and a fire alarm that evacuates the building. Each alarm is distinct, has a specific response procedure, and only fires when human action is actually needed. A badly designed system has one alarm that goes off when a mouse crosses a sensor, when the AC cycles, when a teller opens the vault normally, and also when there's an actual robbery — and after a month, everyone ignores it. That's alert fatigue, and it's how real incidents get missed.

The fundamental principle is: alert on symptoms, not causes. A symptom is something the user experiences — slow responses, failed transactions, errors on the payment page. A cause is an internal technical event — CPU at 80%, one pod restarted, disk at 70%. If CPU is at 80% but users are happy and latency is within SLO, no human action is needed. If a pod restarts once but Kubernetes automatically schedules a replacement and the Service never drops a request, waking someone up was pointless. Alert on the impact: 'payments-api error rate exceeds 1% for 5 minutes' is actionable because it means customers are affected. 'Pod restarted' is noise because Kubernetes is designed to handle pod restarts.

SLO-based alerting using multi-window burn rate is the gold standard approach championed by Google's SRE book. Instead of alerting when latency exceeds a fixed threshold, you define your SLO (99.9% of requests complete in under 200ms over 30 days) and alert when you're burning through your error budget too quickly. The multi-window approach uses two time windows: a short window (5 minutes) catches acute incidents, and a long window (1 hour) confirms the issue is sustained and not a brief spike. You calculate the burn rate — how many times faster than normal you're consuming error budget. A burn rate of 14x means you'll exhaust your monthly error budget in just 2 days. By combining short and long windows at different burn rates, you create alerts that are both fast (catch real incidents quickly) and precise (don't fire on transient blips).

Severity levels and routing determine who gets notified and how. A three-tier model works well: P1/Critical pages the on-call engineer immediately via PagerDuty/OpsGenie (complete service outage, data loss risk, SLO burn rate > 14x), P2/Warning creates a ticket and sends a Slack notification for the team to address during business hours (elevated error rate, approaching capacity limits, burn rate 2-6x), and P3/Info goes to a monitoring channel for awareness only (canary deployment anomalies, non-critical certificate expiry in 30 days). Every alert must have an owner team routed correctly — a database alert shouldn't page the frontend team at 2 AM.

Every alert must be actionable and include a runbook link. If an engineer receives an alert and their first thought is 'what am I supposed to do about this?', the alert is broken. The runbook should contain: what this alert means in plain language, how to verify the alert is real (not a monitoring glitch), step-by-step diagnosis and remediation procedures, escalation contacts if the primary responder can't resolve it, and expected time to resolution. For banking systems, runbooks should also include regulatory notification requirements — some incidents require reporting to financial regulators within specific timeframes.

The gotcha that destroys alerting effectiveness: alert proliferation without pruning. Teams add alerts for every new incident but never remove or consolidate old ones. After a year, you have 300 alert rules, 50 of which fire weekly, and engineers develop the habit of acknowledging without investigating. Schedule quarterly alert reviews: for each alert that fired in the past 90 days, ask 'Did this alert lead to a human action that improved the situation?' If the answer is no three quarters in a row, delete it or convert it to a dashboard metric.