Loading...
Grafana
Everything for Grafana in one place — pick a section below. 23 reviewed items across 4 content types, plus a hands-on tutorial.
📊Hands-on TutorialGuideWire up data sources, build panels, template with variables, alert, and manage dashboards as code.→
📝Interview Questions14Q&A with detailed answers and code examples→
🏭Use Cases3Real-world production patterns and scenarios→
🔥Production Issues3On-call incidents with root causes and fixes
14 / 14
⚡
Quick Answer
Grafana Alerting evaluates queries and expressions from data sources, then routes alert instances through notification policies to contact points. Good design groups related symptoms, alerts on user impact and burn rate, and uses silences for known maintenance without muting unrelated failures.
Detailed Answer
A fire alarm system should tell you which building is burning, not ring every smoke detector as a separate emergency. Grafana alerting needs the same grouping and routing discipline.
Grafana centralizes alert rules across metrics and logs so teams can manage detection and notification from one place. The goal is not more alerts; it is faster recognition of the failure a human can act on.
A rule runs a query, applies expressions or thresholds, creates alert instances based on labels, and sends state changes to notification policies. Policies match labels, group notifications, control timing, and deliver to contact points such as PagerDuty or Slack.
Production teams standardize labels like service, team, environment, and severity. They track alert volume, pages per service, pending duration, no-data behavior, and whether dashboards include deploy and incident annotations.
The subtle problem is label cardinality. If an alert creates one instance per pod, customer, or path, a single outage can flood the on-call channel and delay diagnosis. Alert labels should identify ownership and failure class, not every raw dimension.
Code Example
apiVersion: 1
contactPoints:
- orgId: 1 # Applies to the primary production organization.
name: sre-pagerduty # Sends urgent pages to the SRE escalation path.
receivers:
- uid: sre-pd # Stable receiver id for provisioning updates.
type: pagerduty # Uses the PagerDuty notifier integration.
settings:
integrationKey: ${PAGERDUTY_INTEGRATION_KEY} # Keeps the secret outside source control.💡
Interview Tip
A junior engineer typically answers with the feature name and a happy-path command, but for a senior/architect role, the interviewer is actually looking for production judgment. Explain where the Grafana control plane stores intent, how changes are rolled out, what telemetry proves the system is healthy, and what failure mode creates the largest blast radius. Strong answers include rollback behavior, ownership boundaries, permissions, and the exact signal you would page on.
◈ Architecture Diagram
┌──────────┐
│ Signal │
└────┬─────┘
↓
┌──────────┐
│ Grafana │
└────┬─────┘
↓
┌──────────┐
│ Action │
└──────────┘💬 Comments
⚡
Quick Answer
Prometheus Alertmanager: production-grade, supports clustering, deduplication, silencing, and routing. Grafana alerting: unified across data sources, easier UI, but less mature for high-volume alerting. They can coexist.
Detailed Answer
Prometheus Alertmanager
- Alert rules defined in Prometheus (PromQL-based) - Alertmanager handles routing, grouping, deduplication, silencing - Supports clustering for HA (gossip protocol) - Mature, battle-tested at scale - Only works with Prometheus-compatible metrics
Grafana Alerting (Grafana 9+)
- Unified alerting across ANY data source (Prometheus, Elasticsearch, CloudWatch, Loki) - Visual alert rule builder in the Grafana UI - Multi-dimensional alerts (one rule, multiple series) - Contact points and notification policies (similar to Alertmanager routing) - Can use Prometheus Alertmanager as an external alertmanager
When to use each
- Use Alertmanager when: All alerts are Prometheus-based, need clustering/HA for alerting, operating at very high scale (1000s of alert rules), team prefers config-as-code (YAML) - Use Grafana Alerting when: Need to alert on non-Prometheus sources (Elasticsearch, CloudWatch), prefer UI-based management, want unified alert management across all data sources
Together: Grafana can send alerts TO Alertmanager as an external backend, giving you Grafana's multi-source alert creation with Alertmanager's routing and deduplication.
Code Example
# Prometheus alert rule (rules.yml)
groups:
- name: slo-alerts
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.01
for: 5m
labels:
severity: page
annotations:
summary: "Error rate above 1%"
# Grafana alert rule (provisioning YAML)
apiVersion: 1
groups:
- orgId: 1
name: api-alerts
folder: SRE
interval: 1m
rules:
- uid: high-error-rate
title: High Error Rate
condition: C
data:
- refId: A
datasourceUid: prometheus
model:
expr: sum(rate(http_requests_total{status=~"5.."}[5m]))
- refId: B
datasourceUid: prometheus
model:
expr: sum(rate(http_requests_total[5m]))
- refId: C
datasourceUid: __expr__
model:
type: math
expression: $A / $B > 0.01💡
Interview Tip
Show you know both systems and can articulate when each is appropriate. The hybrid approach (Grafana for multi-source alert creation, Alertmanager as backend) demonstrates architectural maturity.
💬 Comments
⚡
Quick Answer
Loki only indexes labels (not log content), stores compressed log chunks in object storage (S3/GCS). LogQL uses label matchers for stream selection and filter expressions for content search. This reduces storage costs by 10-50x vs Elasticsearch.
Detailed Answer
Loki Architecture
Write Path
1. Promtail/Grafana Agent scrapes logs, adds labels (pod, namespace, container) 2. Distributor receives log entries, validates, and hashes labels to determine ingester 3. Ingester builds compressed chunks in memory, flushes to object storage (S3/GCS) 4. Index (label→chunk mapping) stored in DynamoDB/Cassandra/BoltDB
Read Path
1. Query frontend splits query by time range 2. Querier finds relevant chunks using label index 3. Chunks are downloaded from object storage, decompressed, and grep-searched 4. Results aggregated and returned
Why it's cheaper than Elasticsearch
- No full-text indexing — labels are tiny compared to full log content - Logs stored as compressed chunks in cheap object storage - Minimal index size (labels only, not every word) - Trade-off: Content search is slower (grep vs inverted index)
LogQL
- Stream selector: {namespace="production", app="api"} (like PromQL label matchers) - Filter: |= "error" (contains), != "debug" (excludes), |~ "err(or)?" (regex) - Parser: | json or | logfmt to extract fields - Aggregation: count_over_time({app="api"} |= "error" [5m])
Code Example
# LogQL: Find errors in production API
{namespace="production", app="api-gateway"}
|= "error"
| json
| status_code >= 500
| line_format "{{.ts}} [{{.level}}] {{.msg}}"
# LogQL: Error rate over time (metric query)
sum(rate(
{namespace="production"} |= "error" [5m]
)) by (app)
# LogQL: Top 5 error messages
topk(5,
sum(count_over_time(
{app="api"} | json | level="error" [1h]
)) by (msg)
)
# Promtail config (scrapes Kubernetes pod logs)
server:
http_listen_port: 3101
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod💡
Interview Tip
The key insight is the indexing trade-off: Loki indexes labels (tiny), Elasticsearch indexes content (huge). This is WHY Loki is cheaper. Know LogQL syntax — it's increasingly replacing Elasticsearch in modern stacks.
💬 Comments
⚡
Quick Answer
Use Grafana provisioning (YAML files), Grafonnet (Jsonnet library), or Terraform grafana provider to define dashboards as code. Store in Git, deploy via CI/CD or ArgoCD.
Detailed Answer
Three Approaches
1. Native Provisioning (YAML files): - Drop YAML config files in /etc/grafana/provisioning/ - Supports data sources, dashboards, alert rules, contact points - Dashboards referenced as JSON files on disk - Best for: Kubernetes deployments with ConfigMaps/Helm
2. Grafonnet (Jsonnet): - Programmatic dashboard generation using Jsonnet - Reusable components, templates, and libraries - Generates dashboard JSON from code - Best for: Teams with many similar dashboards, DRY principle
3. Terraform Provider: - grafana_dashboard, grafana_data_source, grafana_folder resources - State-managed, drift detection, plan/apply workflow - Best for: Teams already using Terraform, multi-instance management
GitOps Workflow
1. Developer creates/modifies dashboard in Grafana UI 2. Export dashboard JSON, commit to Git 3. CI validates JSON schema and Jsonnet compilation 4. CD deploys via Helm chart with dashboard ConfigMaps 5. Grafana sidecar watches for ConfigMap changes and auto-loads
Code Example
# Provisioning data source (provisioning/datasources/prometheus.yaml)
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
jsonData:
timeInterval: 15s
exemplarTraceIdDestinations:
- name: traceID
datasourceUid: tempo
# Helm values for Grafana with sidecar
grafana:
sidecar:
dashboards:
enabled: true
searchNamespace: ALL
label: grafana_dashboard
folderAnnotation: grafana_folder
datasources:
enabled: true
# Dashboard ConfigMap (auto-loaded by sidecar)
apiVersion: v1
kind: ConfigMap
metadata:
name: api-dashboard
labels:
grafana_dashboard: "true"
annotations:
grafana_folder: "API"
data:
api-overview.json: |-
{ "dashboard": { ... } }
# Terraform
resource "grafana_dashboard" "api" {
config_json = file("dashboards/api-overview.json")
folder = grafana_folder.api.id
}💡
Interview Tip
The sidecar pattern (watch ConfigMaps, auto-load dashboards) is the standard for Kubernetes. Mention it specifically — it shows you've deployed Grafana in production K8s, not just Docker.
💬 Comments
⚡
Quick Answer
Tempo is a distributed tracing backend that stores traces in object storage. It integrates via exemplars (metrics→traces), trace-to-logs (traces→Loki), and service graphs (traces→metrics).
Detailed Answer
Grafana Tempo
- Open-source, high-scale distributed tracing backend - Stores traces in object storage (S3/GCS) — no local disk needed - Only requires trace ID for lookup — no indexing of span attributes (similar to Loki's approach for logs) - Supports OpenTelemetry, Jaeger, and Zipkin protocols
Integration Points
1. Metrics → Traces (Exemplars): - Prometheus stores exemplar data: a specific trace_id attached to a metric sample - In Grafana, clicking a data point on a metrics panel shows the exemplar - Click the exemplar → jumps directly to the trace in Tempo - Use case: High latency spike in p99 → click → see the exact slow trace
2. Traces → Logs (Trace-to-Logs): - Configure Tempo data source with Loki as the linked log source - Click a trace span → 'View Logs' → Loki query filtered by trace_id, service, and time range - Shows the exact log entries for that request
3. Traces → Metrics (Service Graph): - Tempo generates service graph metrics from traces automatically - Visualizes request rate, error rate, and latency between services - Stored as Prometheus metrics for dashboarding
The result: A fully correlated observability stack where you can seamlessly navigate between metrics, traces, and logs without context-switching tools.
Code Example
# Tempo configuration (tempo.yaml)
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
storage:
trace:
backend: s3
s3:
bucket: tempo-traces
endpoint: s3.amazonaws.com
metrics_generator:
storage:
path: /var/tempo/wal
traces_storage:
path: /var/tempo/traces
processor:
service_graphs:
dimensions: ["service"] # Generate service graph metrics
# Grafana data source provisioning with correlations
apiVersion: 1
datasources:
- name: Tempo
type: tempo
url: http://tempo:3200
jsonData:
tracesToLogs:
datasourceUid: loki
tags: ['service.name']
filterByTraceID: true
tracesToMetrics:
datasourceUid: prometheus
tags: [{ key: 'service.name', value: 'service' }]
serviceMap:
datasourceUid: prometheus💡
Interview Tip
The three-way correlation (metrics↔traces↔logs) is what makes the Grafana stack compelling. Walk through a debugging scenario: metric alert → exemplar → trace → logs. This shows you use observability tools holistically.
💬 Comments
⚡
Quick Answer
Use RED method (Rate, Errors, Duration) for services and USE (Utilization, Saturation, Errors) for infra. Critical signals at top, deployment annotations, variable-driven filtering.
Detailed Answer
Rules: 1) Top row: request rate, error rate, latency p50/p99. 2) Second row: CPU, memory, disk. 3) Variables for namespace/service filtering. 4) Deployment annotations for correlation. 5) Color thresholds (green/yellow/red). 6) One service = one dashboard. 7) Link related dashboards. RED works for request-driven services. USE works for resources.
Code Example
# Key panels:
sum(rate(http_requests_total[5m])) by (service)
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))💡
Interview Tip
Mention RED and USE methods by name. Explain why deployment annotations matter.
💬 Comments
⚡
Quick Answer
Grafana is a visualization and alerting layer; it queries data sources like Prometheus but stores no metrics itself.
Detailed Answer
Grafana connects to time-series and log backends (Prometheus, Loki, InfluxDB, SQL, cloud) and renders dashboards and alerts. It holds no metric data of its own — a slow dashboard is almost always a slow query in the underlying data source, not Grafana. Prometheus scrapes and stores; Grafana reads and displays.
💡
Interview Tip
Emphasize the separation of concerns: storage vs visualization.
💬 Comments
⚡
Quick Answer
Drop YAML/JSON provisioning files under /etc/grafana/provisioning so they load on startup and can't be edited in the UI.
Detailed Answer
Provisioning makes a Grafana instance reproducible and disposable. Data sources go in provisioning/datasources/*.yml, dashboards in provisioning/dashboards/*.yml pointing at a folder of dashboard JSON. Provisioned objects are read-only in the UI, which prevents drift. For full GitOps you can also manage dashboards with the Terraform grafana provider or Grafonnet.
Code Example
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true💡
Interview Tip
Mention that provisioned resources are immutable in the UI — a feature, not a limitation.
💬 Comments
⚡
Quick Answer
Variables turn one dashboard into many by injecting selectable values (like $service) into every panel query.
Detailed Answer
A Query variable pulls live values from a data source (e.g. label_values(http_requests_total, service)); Custom, Interval, and Data source types also exist. Reference them in queries with $var and enable Multi-value/Include All for filtering. One well-templated dashboard replaces dozens of near-duplicates and lets viewers slice by service, namespace, or environment.
Code Example
label_values(http_requests_total, service)
💡
Interview Tip
Show label_values() for Prometheus and mention the 'Data source' variable to switch prod/staging.
💬 Comments
⚡
Quick Answer
RED = Rate, Errors, Duration — the three signals that answer 'is this service up and fast?'.
Detailed Answer
For request-driven services, chart the request Rate, the Error rate (e.g. 5xx share), and Duration (latency percentiles via histogram_quantile). RED complements USE (Utilization, Saturation, Errors) which suits resources like CPU and queues. Starting every service dashboard from RED gives on-call a consistent first screen.
Code Example
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[$__rate_interval])))
💡
Interview Tip
Contrast RED (services) with USE (resources).
💬 Comments
⚡
Quick Answer
$__rate_interval scales the rate window with the dashboard's time range and scrape interval, avoiding gaps or over-smoothing.
Detailed Answer
A hard-coded [5m] breaks when zoomed out (too few points) or in (aliasing). $__rate_interval is computed by Grafana from the panel's step and the data source's scrape interval, guaranteeing at least a few samples per window at any zoom level. It's the recommended default for rate()/irate() in Grafana panels.
💡
Interview Tip
This is a common gotcha — mention it proactively.
💬 Comments
⚡
Quick Answer
An alert rule evaluates a query on a schedule; when the condition holds for the 'for' duration, it fires and routes to contact points via notification policies.
Detailed Answer
You define a query and threshold, a pending period ('for') to suppress blips, labels (e.g. severity), and a notification policy that maps labels to contact points (Slack, PagerDuty, email). Alert on user-visible symptoms (error rate, latency), not causes (CPU). Provision rules as code so they live in Git.
💡
Interview Tip
State the golden rule: alert on symptoms, not causes.
💬 Comments
⚡
Quick Answer
Add Loki as a data source alongside Prometheus and use shared labels plus data links to jump from a metric spike to the matching logs.
Detailed Answer
With Prometheus and Loki using consistent labels (service, namespace, pod), you can place a metrics panel and a logs panel on one dashboard filtered by the same variables. Grafana's split view and derived fields let you pivot from a latency spike to the exact log lines, and exemplars can link a histogram bucket to a trace.
💡
Interview Tip
Mention exemplars for metrics-to-traces linking.
💬 Comments
⚡
Quick Answer
Use Stat or Gauge for a single current value and thresholds; use Time series for trends over time.
Detailed Answer
Stat shows one big number with optional sparkline and color thresholds — good for 'current error rate'. Gauge adds a min/max arc. Time series is for evolution over the window. Table and Bar chart suit top-N breakdowns. Always set a unit and a sensible legend; an unlabeled 1.2k helps no one.
💡
Interview Tip
Always mention setting the unit and legend format.
💬 Comments