How do you provision Grafana dashboards, data sources, and alerts as code for a GitOps workflow?
Quick Answer
Use Grafana provisioning (YAML files), Grafonnet (Jsonnet library), or Terraform grafana provider to define dashboards as code. Store in Git, deploy via CI/CD or ArgoCD.
Detailed Answer
Three Approaches
1. Native Provisioning (YAML files): - Drop YAML config files in /etc/grafana/provisioning/ - Supports data sources, dashboards, alert rules, contact points - Dashboards referenced as JSON files on disk - Best for: Kubernetes deployments with ConfigMaps/Helm
2. Grafonnet (Jsonnet): - Programmatic dashboard generation using Jsonnet - Reusable components, templates, and libraries - Generates dashboard JSON from code - Best for: Teams with many similar dashboards, DRY principle
3. Terraform Provider: - grafana_dashboard, grafana_data_source, grafana_folder resources - State-managed, drift detection, plan/apply workflow - Best for: Teams already using Terraform, multi-instance management
GitOps Workflow
1. Developer creates/modifies dashboard in Grafana UI 2. Export dashboard JSON, commit to Git 3. CI validates JSON schema and Jsonnet compilation 4. CD deploys via Helm chart with dashboard ConfigMaps 5. Grafana sidecar watches for ConfigMap changes and auto-loads
Code Example
# Provisioning data source (provisioning/datasources/prometheus.yaml)
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
jsonData:
timeInterval: 15s
exemplarTraceIdDestinations:
- name: traceID
datasourceUid: tempo
# Helm values for Grafana with sidecar
grafana:
sidecar:
dashboards:
enabled: true
searchNamespace: ALL
label: grafana_dashboard
folderAnnotation: grafana_folder
datasources:
enabled: true
# Dashboard ConfigMap (auto-loaded by sidecar)
apiVersion: v1
kind: ConfigMap
metadata:
name: api-dashboard
labels:
grafana_dashboard: "true"
annotations:
grafana_folder: "API"
data:
api-overview.json: |-
{ "dashboard": { ... } }
# Terraform
resource "grafana_dashboard" "api" {
config_json = file("dashboards/api-overview.json")
folder = grafana_folder.api.id
}Interview Tip
The sidecar pattern (watch ConfigMaps, auto-load dashboards) is the standard for Kubernetes. Mention it specifically — it shows you've deployed Grafana in production K8s, not just Docker.