How do you manage Grafana dashboards and alerts as code with GitOps?

Think of it like managing a chain of restaurants where every location has to serve the same menu. Instead of calling each manager and dictating changes over the phone, which is like clicking around in Grafana's UI, you update the master menu in a shared drive, the managers review it, and an automated system prints and ships the new menus to all locations at once.

The GitOps workflow for Grafana has three main approaches, from simple to powerful. The simplest is Grafana's built-in provisioning: you put dashboard JSON files and alert rule YAML files in a directory that Grafana watches, usually mounted via a ConfigMap in Kubernetes. When the files change, Grafana reloads them. You store these files in Git, and your CI/CD pipeline updates the ConfigMap every time a change merges to main.

The second approach uses Grafonnet, a Jsonnet library for generating Grafana dashboard JSON programmatically. Instead of writing raw 500-line JSON files by hand, you write concise Jsonnet code that generates them. This is where updating 10 dashboards at once becomes easy: if all 10 share a common template, say a service dashboard with CPU, memory, error rate, and latency panels, you define the template once and pass in parameters per service. Changing the template changes all 10 dashboards in one commit. Jsonnet compiles down to JSON, which then gets provisioned into Grafana.

The third approach uses Terraform with the Grafana provider. You define dashboards, folders, alert rules, and notification channels as Terraform resources. The CI pipeline runs terraform plan on pull requests to show what would change and terraform apply on merge. This gives you state management, drift detection, and the full Terraform workflow. For large organizations managing hundreds of dashboards across multiple Grafana instances, this is the most maintainable path.

For alerts, Grafana's alerting rules and notification policies can also be defined in YAML and provisioned alongside dashboards. The entire alerting chain -- rules, routing policies, contact points, and message templates -- lives in Git, version-controlled and reviewable.

The day-to-day workflow looks like this: a developer creates a branch, modifies dashboard Jsonnet or Terraform files, opens a pull request, CI runs syntax checks like jsonnet lint or terraform validate and optionally renders a preview, a reviewer approves, the PR merges to main, and the CD pipeline applies changes to Grafana. The big win is that every change is code-reviewed, version-controlled, and reversible with a simple git revert.