How do you manage Datadog monitor sprawl and alert fatigue in a large engineering organization with 500+ services?
Quick Answer
Implement monitor-as-code (Terraform), tiered alerting strategy (SLO-based > symptom-based > cause-based), composite monitors for reducing noise, and regular monitor audits with ownership tagging.
Detailed Answer
The Problem
Organizations often accumulate thousands of monitors with no clear ownership, leading to alert fatigue where engineers ignore pages because most are noise.
Strategy
1. Monitor-as-Code: - All monitors defined in Terraform, versioned in Git - PR review for new monitors prevents duplicates and ensures quality - Teams own their monitors via team tags
2. Tiered Alerting: - Tier 1 (Page): SLO burn rate alerts only. If the SLO isn't burning, don't page. - Tier 2 (Slack): Symptom-based alerts (elevated error rate, high latency) that don't yet impact SLOs - Tier 3 (Dashboard): Cause-based alerts (high CPU, disk filling) — informational only
3. Composite Monitors: Combine multiple conditions into one alert. Instead of 5 separate monitors for each dependency, create one composite: A AND B AND NOT C — only alert when the combination indicates a real problem.
4. Regular Audits: - Monthly: Review monitors that never fired (delete or tune) - Monthly: Review monitors that fired but were never acknowledged (noise) - Quarterly: Review SLO targets and burn rate thresholds - Tag every monitor with team, service, tier
5. Downtime Management: - Scheduled downtimes for maintenance windows - Auto-muting during deployments (CI/CD integration)
Code Example
# Terraform: SLO-based monitor
resource "datadog_service_level_objective" "api_availability" {
name = "API Availability SLO"
type = "metric"
description = "99.9% of requests succeed"
query {
numerator = "sum:http.requests.success{service:api}.as_count()"
denominator = "sum:http.requests.total{service:api}.as_count()"
}
thresholds {
timeframe = "30d"
target = 99.9
warning = 99.95
}
tags = ["team:platform", "service:api", "tier:1"]
}
# Composite monitor
resource "datadog_monitor" "composite" {
name = "API degraded - multiple signals"
type = "composite"
query = "${datadog_monitor.high_error_rate.id} && ${datadog_monitor.high_latency.id}"
message = "Multiple degradation signals. @pagerduty-api-team"
}
# Mute during deploys
curl -X POST "https://api.datadoghq.com/api/v1/downtime" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-d '{"scope":"service:api","end":"'$(date -d '+30min' +%s)'"}'Interview Tip
Alert fatigue is a real problem every large organization faces. Lead with the SLO-based approach — only page on customer impact, not infrastructure symptoms. Mention monitor-as-code for governance.