How do you design incident-useful Grafana dashboards instead of vanity dashboards?
⚡
Quick Answer
Use RED method (Rate, Errors, Duration) for services and USE (Utilization, Saturation, Errors) for infra. Critical signals at top, deployment annotations, variable-driven filtering.
Detailed Answer
Rules: 1) Top row: request rate, error rate, latency p50/p99. 2) Second row: CPU, memory, disk. 3) Variables for namespace/service filtering. 4) Deployment annotations for correlation. 5) Color thresholds (green/yellow/red). 6) One service = one dashboard. 7) Link related dashboards. RED works for request-driven services. USE works for resources.
Code Example
# Key panels:
sum(rate(http_requests_total[5m])) by (service)
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))💡
Interview Tip
Mention RED and USE methods by name. Explain why deployment annotations matter.
grafanadashboardsred-methoduse-methodmonitoring