Grafana: Zero to Hero

A hands-on path from your first dashboard to production-grade observability. You'll run Grafana locally, wire up a data source, build panels, template with variables, and set up alerting — all reproducible.

---

Prerequisites

No cloud account is required. If you don't have Prometheus running yet, do that tutorial first — every good Grafana panel is a good query against a real data source.

---

1. What Grafana is (and isn't)

Grafana is a visualization and alerting layer. It does not store your metrics or logs — it queries systems that do (Prometheus, Loki, InfluxDB, Elasticsearch, SQL databases, cloud providers) and renders the results as dashboards.

Mental model:

data sources (Prometheus, Loki, …)  ->  Grafana (query + visualize + alert)  ->  you

Keep that separation in mind: a "slow dashboard" is usually a slow query, not Grafana itself.

---

2. Run it in one command

docker run -d --name grafana -p 3000:3000 grafana/grafana:11.1.0

Open http://localhost:3000 and log in with admin / admin (you'll be asked to change the password). Pin the tag (:11.1.0) rather than :latest so upgrades are deliberate.

To persist dashboards and settings across restarts, mount a volume:

docker run -d --name grafana -p 3000:3000 \
  -v grafana-storage:/var/lib/grafana \
  grafana/grafana:11.1.0

---

3. Add a data source

Everything starts with a data source. In the UI: Connections → Data sources → Add data source → Prometheus, set the URL (e.g. http://prometheus:9090), and Save & test.

For reproducibility, provision it as code instead of clicking. Drop this file at /etc/grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Provisioned data sources appear automatically on startup and can't be edited in the UI — exactly what you want in production.

---

4. Your first panel

Create a dashboard (Dashboards → New → New dashboard → Add visualization), pick your Prometheus source, and enter a query:

sum by (service) (rate(http_requests_total[5m]))

Set the panel title, choose a unit (Standard options → Unit → requests/sec), and pick a visualization (Time series, Stat, Gauge, Table, Bar chart). A few habits that pay off:

• Always set a unit. "1.2k" means nothing; "1.2k req/s" does.
• Name series with Legend using {{service}} so the legend is readable.
• Use $__rate_interval instead of a hard-coded [5m] so the window scales with the dashboard's time range.

---

5. Template with variables

Variables turn one dashboard into hundreds. Add a variable (Dashboard settings → Variables → New) of type Query:

label_values(http_requests_total, service)

Name it service, enable Include All and Multi-value, then use it in panels:

sum by (service) (rate(http_requests_total{service=~"$service"}[$__rate_interval]))

Now a dropdown at the top lets viewers filter every panel at once. Common variable types:

• Query — values pulled live from a data source (label_values, label_names).
• Custom — a fixed list you type in.
• Interval — selectable time buckets (1m,5m,1h).
• Data source — switch the whole dashboard between prod and staging.

---

6. The RED and USE dashboards

Two battle-tested layouts to copy:

• RED (request-driven services): Rate, Errors, Duration.
  • Rate: sum(rate(http_requests_total[$__rate_interval]))
  • Errors: sum(rate(http_requests_total{status=~"5.."}[$__rate_interval]))
  • Duration (p99): histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[$__rate_interval])))
• USE (resources like CPU, disk, queues): Utilization, Saturation, Errors.

Start every service dashboard from RED and you'll answer "is it up and fast?" in one screen.

---

7. Alerting

Grafana's unified alerting evaluates a query on a schedule and fires when a condition holds.

1. On a time-series panel, open Alert → New alert rule.
2. Define the query (e.g. error ratio) and a threshold (IS ABOVE 0.05).
3. Set for: 5m so brief blips don't page anyone.
4. Attach labels (severity=critical) and a contact point (Slack, PagerDuty, email) via a notification policy.

Provision alert rules as code too, so they live in Git and survive a rebuild. The golden rule: alert on symptoms users feel (error rate, latency), not on causes (CPU) — causes belong on dashboards.

---

8. Dashboards as code

Click-built dashboards drift and get lost. Export the JSON (Dashboard settings → JSON Model) and commit it, then provision it:

# /etc/grafana/provisioning/dashboards/default.yml
apiVersion: 1
providers:
  - name: 'default'
    folder: 'Services'
    type: file
    options:
      path: /var/lib/grafana/dashboards

Better still, generate dashboards with the Terraform grafana provider or Grafonnet so reviews happen in pull requests. Our PromQL builder can export a ready-to-import Grafana dashboard JSON for any query.

---

9. Production hardening

• Run behind auth — disable the default admin, integrate SSO/OAuth, and set GF_AUTH_ANONYMOUS_ENABLED=false.
• Use provisioning for everything — data sources, dashboards, alerts — so the instance is disposable.
• Set data source query timeouts and caching to protect Prometheus from expensive dashboards.
• Back up the Grafana database (or use provisioning so there's nothing to back up).
• Version-pin the image and test upgrades in staging.

---

Common mistakes to avoid

• Click-building dashboards. They drift, get lost, and can't be reviewed. Export the JSON and provision it from Git.
• No unit on panels. "1.2k" is meaningless. Always set a unit so "1.2k req/s" or "340 MB" reads at a glance.
• Hard-coded [5m] ranges. They don't scale with the dashboard's time range. Use $__rate_interval.
• Alerting on causes. A "CPU > 80%" alert pages you for non-problems. Alert on symptoms users feel (error rate, latency); keep causes on dashboards.
• Treating Grafana as the metrics store. It isn't — a slow dashboard is almost always a slow query or an overloaded data source, not Grafana.
• Leaving anonymous/default admin access on. Put it behind SSO, disable the default admin, and set GF_AUTH_ANONYMOUS_ENABLED=false.
• No query timeouts. An expensive dashboard can hammer Prometheus. Set data-source timeouts and caching.

---

Practice: try it yourself

1. Wire it up. Run Grafana with a persistent volume, add Prometheus as a provisioned data source (not via clicking), and confirm it passes Save & test.
2. Build a RED row. Create three panels — request rate, 5xx error ratio, p99 latency — each with a proper unit and a {{service}} legend.
3. Template it. Add a service query variable with Include All, and make every panel filter on $service. Watch one dropdown drive the whole dashboard.
4. Alert + as-code. Add an alert rule (error ratio IS ABOVE 0.05, for: 5m) with a contact point. Then export the dashboard JSON and commit it.

Self-check:

• Grafana doesn't store metrics — so what does a "slow dashboard" usually mean?
• Why prefer $__rate_interval over a fixed [5m]?
• What turns one dashboard into hundreds without copy-paste?
• Why alert on error rate rather than CPU?

---

Where to go next

• Add Loki as a second data source and correlate logs with metrics on one dashboard.
• Learn PromQL deeply — every good Grafana panel is a good query. Try the PromQL Query Builder.
• Practice with real Prometheus interview questions to cement the concepts.

You now have the full loop: data source → panel → variables → alert → as-code. That's Grafana in production.