How do you reduce alert fatigue in Datadog while ensuring real incidents page?
⚡
Quick Answer
Composite monitors combining signals, anomaly detection, alert grouping by service not pod, maintenance windows, three-tier alerting (page/notify/dashboard).
Detailed Answer
Fixes: 1) Composite monitors: page only when BOTH error rate AND latency are high. 2) Anomaly detection instead of static thresholds. 3) Group by service not pod. 4) Maintenance windows via CI/CD. 5) Tiers: P1 page, P2 Slack, P3 dashboard only. 6) Review monitors quarterly. 7) Track MTTA.
Code Example
# Datadog composite monitor:
resource "datadog_monitor" "api_degraded" {
type = "composite"
query = "${high_error_rate.id} && ${high_latency.id}"
message = "@pagerduty-api API degraded"
}💡
Interview Tip
Show why naive alerting causes fatigue and how composite monitors fix it.
datadogalert-fatiguecomposite-monitorsanomaly-detection