How do you set up centralized Grafana dashboards across multiple Kubernetes clusters?

Think of it like a retail chain with 20 stores. Each store has its own security camera system recording locally. The head office needs to see footage from all stores on one screen. You set up a central recording server and have each store upload its footage with a store ID tag. The head office monitor can then filter by store or show all stores side by side.

The architecture has three layers. First, each Kubernetes cluster runs its own Prometheus instance, or a Prometheus Operator stack, that scrapes all metrics locally. Each Prometheus is configured with a unique external_label, something like cluster=payments-prod-us-east-1, that identifies which cluster the metrics come from. This label is critical because without it you cannot tell metrics from different clusters apart.

Second, each Prometheus uses remote-write to push metrics to a centralized metrics backend. The three main choices are Mimir from Grafana Labs for large-scale setups, Thanos Receive for accepting remote-write and storing in object storage, or VictoriaMetrics for a high-performance and simpler-to-operate option. The remote-write endpoint usually sits behind a load balancer or ingress, secured with mTLS or bearer tokens. Each cluster's Prometheus pushes metrics continuously, typically every 15 to 30 seconds, to this central system.

Third, Grafana connects to the centralized backend as a single Prometheus-compatible data source. Every PromQL query can filter by cluster label: sum(rate(http_requests_total{cluster="payments-prod"}[5m])) for one cluster, or sum by (cluster)(rate(http_requests_total[5m])) for a side-by-side comparison. Dashboard variables use label_values(cluster) to create a dropdown that lets users switch between clusters with a click.

For Grafana itself, in a multi-cluster setup it typically runs in a central management cluster rather than in every cluster. Dashboards and alerts are managed as code using Grafana's provisioning system or tools like Grafonnet and Terraform. This ensures consistency -- updating one dashboard template updates the view for all clusters at once.

The key operational concern is network reliability. If a cluster's Prometheus cannot reach the central backend because of a network partition or backend overload, metrics can be lost. Prometheus buffers remote-write data in a WAL, or write-ahead log, and retries, but the buffer has a size limit of around 300 MB by default. For critical environments, consider running a local Thanos Sidecar or VictoriaMetrics agent as a write buffer that can survive longer outages without dropping data.