Prometheus: Zero to Hero

A complete, practical path from "I've never run Prometheus" to "I can instrument apps, write PromQL, and run alerting in production." Every step runs locally with Docker — no cloud needed. Work through it top to bottom, or jump to the part you need.

The journey:

1. What Prometheus is (and isn't)
2. Run your first Prometheus in 5 minutes
3. The data model — metrics, labels, and the four metric types
4. Configuration and scraping
5. Instrument your own application
6. PromQL: from selectors to rate() and quantiles
7. Recording rules
8. Alerting with Alertmanager
9. Visualize with Grafana
10. Production hardening — cardinality, retention, HA, remote storage

---

Prerequisites

No cloud account is required. A little YAML comfort helps, since config and rules are YAML. Confirm Docker works: docker run --rm hello-world.

---

Part 1 — What Prometheus is (and isn't)

Prometheus is an open-source monitoring and alerting system built around a time-series database (TSDB). Its defining design choices:

• Pull, not push. Prometheus scrapes metrics from your targets over HTTP on a fixed interval, rather than waiting for them to push. This makes targets simple (just expose a page) and lets Prometheus know when a target is down.
• Dimensional data model. Every time series is a metric name plus a set of key/value labels, so http_requests_total{method="GET", status="500"} is a distinct series you can slice and aggregate.
• PromQL. A powerful query language for selecting, aggregating, and computing over time series.
• Its own storage. A local, efficient TSDB (with optional remote write for long-term/scale).

What it is not: it is not a logging system (use Loki/ELK), not a distributed tracing system (use Tempo/Jaeger), and not built for high-cardinality event data or exact billing. It excels at numeric, time-series metrics for alerting and dashboards.

  ┌────────────┐  scrape (pull)   ┌──────────────┐
  │  targets   │◀─────────────────│  Prometheus  │──▶ PromQL / API / Grafana
  │ /metrics   │                  │   TSDB       │──▶ alerting rules
  └────────────┘                  └──────┬───────┘
                                         ▼
                                  ┌──────────────┐
                                  │ Alertmanager │──▶ Slack / PagerDuty / email
                                  └──────────────┘

---

Part 2 — Run your first Prometheus in 5 minutes

Create a minimal config, then start Prometheus with Docker.

# prometheus.yml
global:
  scrape_interval: 15s          # how often to scrape targets
  evaluation_interval: 15s      # how often to evaluate rules

scrape_configs:
  - job_name: 'prometheus'      # Prometheus scraping itself
    static_configs:
      - targets: ['localhost:9090']

docker run --rm -p 9090:9090 \
  -v "$(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml" \
  prom/prometheus

Open http://localhost:9090:

• Status → Targets — you should see the prometheus job UP.
• Graph — type up and execute. up == 1 means a target is reachable; up == 0 means it's down. This single metric is the foundation of "is my thing alive?" alerting.

You now have a working Prometheus scraping itself. Everything else is adding targets, queries, and rules.

---

Part 3 — The data model: metrics, labels, and the four types

A sample is a (timestamp, float64 value). A time series is a unique combination of a metric name and labels:

http_requests_total{job="api", method="POST", status="200"}  →  1027

Adding a label value creates a new series. That's powerful — and the source of the #1 production pitfall, cardinality (Part 10).

The four metric types

Rules of thumb:

• Never graph a raw counter — it's a monotonically increasing line. Wrap it in rate() to get per-second change.
• Prefer histograms over summaries when you need quantiles you can aggregate across instances (summaries' quantiles can't be averaged).

---

Part 4 — Configuration and scraping

prometheus.yml has three sections you'll use constantly: global, scrape_configs, and rule_files.

global:
  scrape_interval: 15s
  external_labels:
    cluster: dev            # attached to all series when federating / remote-writing

rule_files:
  - "rules/*.yml"           # recording + alerting rules

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          env: dev

  - job_name: 'api'
    metrics_path: /metrics
    scrape_interval: 10s     # override global per-job
    static_configs:
      - targets: ['api:8080']

Service discovery (don't hard-code targets in prod)

Static targets are fine for a demo. In real systems you use service discovery so Prometheus finds targets automatically as they come and go:

• Kubernetes SD (kubernetes_sd_configs) — discover pods/services/endpoints by labels/annotations.
• File SD (file_sd_configs) — a JSON/YAML file another tool writes.
• Cloud SD for EC2, GCE, Azure, Consul, etc.

relabel_configs then filter and rewrite discovered targets (keep only annotated pods, set the instance label, map ports). Relabeling is the part everyone finds confusing at first — think of it as a pipeline that transforms the label set of each target before scraping.

# Validate config before reloading
promtool check config prometheus.yml
# Hot-reload without restart (if --web.enable-lifecycle is set)
curl -X POST http://localhost:9090/-/reload

---

Part 5 — Instrument your own application

Exporters translate third-party systems into metrics (e.g., node_exporter for host metrics, blackbox_exporter for probing endpoints). For your code, use a client library.

Add node_exporter (host metrics)

docker run --rm -p 9100:9100 prom/node-exporter
# then add a scrape job for node-exporter:9100 and query, e.g.:
#   100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)   # CPU %

Instrument application code (Python example)

The pattern is identical across languages (Go, Java, Node, etc.): create metrics, update them in your handlers, and expose /metrics.

from prometheus_client import Counter, Histogram, start_http_server
import time, random

REQUESTS = Counter("app_requests_total", "Total requests", ["method", "status"])
LATENCY  = Histogram("app_request_duration_seconds", "Request latency", ["method"])

def handle(method):
    with LATENCY.labels(method).time():          # observes duration into buckets
        time.sleep(random.random() / 5)
        status = "500" if random.random() < 0.1 else "200"
        REQUESTS.labels(method, status).inc()    # increment the counter

if __name__ == "__main__":
    start_http_server(8000)                       # exposes /metrics on :8000
    while True:
        handle(random.choice(["GET", "POST"]))
        time.sleep(0.2)

Scrape localhost:8000, then query app_requests_total — you'll see one series per (method, status) pair.

Instrumentation best practices

• Name metrics <namespace>_<name>_<unit> with a base unit (seconds, bytes) — e.g. app_request_duration_seconds. Counters end in _total.
• Put things you'll aggregate or filter by in labels (method, route, status) — never put unbounded values (user IDs, emails, raw URLs) in labels.
• Instrument the RED method for services (Rate, Errors, Duration) and USE for resources (Utilization, Saturation, Errors).

---

Part 6 — PromQL: from selectors to rates and quantiles

PromQL is where Prometheus becomes powerful. Build it up in layers.

6.1 Instant selectors and filters

up                                      # all up series
app_requests_total{status="500"}        # only 500s
app_requests_total{status=~"5.."}       # regex: any 5xx
app_requests_total{status!="200"}       # negation

6.2 Rate of a counter (the single most important query)

rate(app_requests_total[5m])            # per-second rate over a 5-minute window

Use rate() for counters — it's resilient to counter resets. Use increase() for "how many over this window." Pick a range ([5m]) at least 4× your scrape interval.

6.3 Aggregation

sum(rate(app_requests_total[5m]))                      # total RPS across everything
sum by (status) (rate(app_requests_total[5m]))         # RPS grouped by status
sum without (instance) (rate(app_requests_total[5m]))  # collapse the instance label
topk(5, sum by (route)(rate(app_requests_total[5m])))  # busiest 5 routes

sum, avg, max, min, count all take by (...) / without (...).

6.4 Error ratio (a real SLO-style query)

sum(rate(app_requests_total{status=~"5.."}[5m]))
  /
sum(rate(app_requests_total[5m]))                      # fraction of requests failing

6.5 Latency quantiles from a histogram

histogram_quantile(
  0.95,
  sum by (le) (rate(app_request_duration_seconds_bucket[5m]))
)                                                       # p95 latency in seconds

The le ("less than or equal") label holds the bucket boundaries — you must keep le in the by (...) clause for histogram_quantile to work.

6.6 Common gotchas

• Don't rate() a gauge, and don't graph a raw counter.
• Match ranges to scrape interval ([5m] with 15s scrape is safe; [30s] is too short).
• Averaging summary quantiles is invalid — use histograms if you aggregate across instances.

---

Part 7 — Recording rules (precompute expensive queries)

Recording rules evaluate a query on a schedule and save the result as a new series. Use them for dashboards and alerts that would otherwise recompute a heavy expression every time.

# rules/recording.yml
groups:
  - name: api-slo
    interval: 30s
    rules:
      - record: job:app_requests:rate5m
        expr: sum by (job) (rate(app_requests_total[5m]))

      - record: job:app_errors:ratio5m
        expr: |
          sum by (job) (rate(app_requests_total{status=~"5.."}[5m]))
            /
          sum by (job) (rate(app_requests_total[5m]))

Now dashboards query the cheap job:app_errors:ratio5m instead of the full expression. Convention: name recorded series level:metric:operation.

---

Part 8 — Alerting with Alertmanager

Prometheus evaluates alert rules and fires alerts; Alertmanager dedupes, groups, silences, and routes them to receivers (Slack, PagerDuty, email).

8.1 Define alert rules

# rules/alerts.yml
groups:
  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: job:app_errors:ratio5m > 0.05      # >5% errors
        for: 10m                                  # sustained for 10 min (avoids flapping)
        labels:
          severity: page
        annotations:
          summary: "High error rate on {{ $labels.job }}"
          description: "Error ratio is {{ $value | humanizePercentage }} (>5%) for 10m."

      - alert: TargetDown
        expr: up == 0
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "Target {{ $labels.instance }} is down"

The for clause is what separates a blip from an incident — the condition must hold continuously before the alert fires.

8.2 Route with Alertmanager

# alertmanager.yml
route:
  receiver: slack-default
  group_by: ['alertname', 'job']    # collapse related alerts into one notification
  group_wait: 30s
  repeat_interval: 4h
  routes:
    - matchers: [ severity="page" ]
      receiver: pagerduty

receivers:
  - name: slack-default
    slack_configs:
      - channel: '#alerts'
        api_url: '<webhook-url>'
  - name: pagerduty
    pagerduty_configs:
      - service_key: '<key>'

Good alerts are symptom-based (users are seeing errors) not cause-based (CPU is 90%), have a sensible for:, and always link to a runbook in the annotation.

---

Part 9 — Visualize with Grafana

Prometheus's built-in graph is for exploration; Grafana is for dashboards.

docker run --rm -p 3000:3000 grafana/grafana

1. Log in (admin/admin), Add data source → Prometheus, URL http://host.docker.internal:9090.
2. Build a panel with a PromQL query (e.g. the p95 latency query from Part 6.5).
3. Import a community dashboard by ID (e.g., 1860 for node_exporter) to get a full host dashboard instantly.

Design dashboards around the RED/USE methods and use the recording rules from Part 7 so panels stay fast.

---

Part 10 — Production hardening

This is what separates "it works on my laptop" from "hero."

Cardinality — the #1 way to kill Prometheus

Each unique label-set is a series held in memory. A label with unbounded values (user ID, email, request URL, session ID) can explode series count and OOM Prometheus.

# Find your worst offenders
topk(10, count by (__name__)({__name__=~".+"}))      # biggest metrics by series count
count(app_requests_total)                             # series for one metric

Rules: keep labels bounded and low-cardinality; never label by ID/URL/timestamp; drop noisy labels/metrics with metric_relabel_configs.

Retention and storage

• Local TSDB retention: --storage.tsdb.retention.time=15d (default 15 days).
• For long-term or global querying, use remote write to a scalable backend (Thanos, Cortex/Mimir, VictoriaMetrics) — Prometheus stays the scraper; the backend stores and queries at scale.

High availability and scale

• HA: run two identical Prometheus servers scraping the same targets; Alertmanager dedupes their alerts. There's no built-in clustering — redundancy is by running pairs.
• Federation: a higher-level Prometheus scrapes aggregated series from lower-level ones (/federate) — useful for global rollups.
• Thanos/Mimir add long-term storage, global query, and downsampling on top.

Operational checklist

• Alert on up == 0 and on Prometheus itself (scrape duration, TSDB head series, WAL).
• Validate rules and config in CI with promtool check rules / promtool check config.
• Set resource requests/limits and watch memory vs series count.
• Back up or replicate long-term data; local TSDB is not durable storage.

---

Common mistakes to avoid

• High-cardinality labels. Putting a user ID, email, request URL, or timestamp in a label explodes series count and OOMs Prometheus. Keep labels bounded.
• Graphing a raw counter. It's a monotonic line and tells you nothing. Wrap counters in rate()/increase().
• A range window too close to the scrape interval. rate(...[30s]) with a 15s scrape has too few samples. Use at least 4× the interval ([1m]+), or $__rate_interval.
• Averaging summary quantiles. Quantiles don't average. If you aggregate across instances, use histograms and histogram_quantile.
• Dropping le from a histogram query. histogram_quantile needs the le label — keep it in your by (...).
• Alerting on causes, not symptoms. Paging on "CPU 90%" creates noise. Alert on what users feel (errors, latency); put causes on dashboards.
• No for: on alerts. Every transient blip pages someone. Add a for: so the condition must hold before firing.

---

Practice: try it yourself

1. Instrument and scrape. Run the Python app from Part 5, add a scrape job for it, and confirm it's UP under Status → Targets. Query app_requests_total and count the series.
2. Build the RED trio. Write three queries against app_requests_total / app_request_duration_seconds_bucket: request rate, 5xx error ratio, and p95 latency.
3. Record and alert. Turn the error-ratio query into a recording rule, then an alert that fires above 5% for: 10m. Reload with promtool check rules first.
4. Hunt cardinality. Run topk(10, count by (__name__)({__name__=~".+"})) and identify your highest-series metric. Would any of its labels be dangerous if unbounded?

Self-check:

• Why does Prometheus pull instead of accept pushes, and what does that let it detect?
• When do you reach for rate() vs increase() vs the raw value?
• What makes a label "high cardinality," and why does it matter?
• What's the difference between a recording rule and an alerting rule?

---

Where to go next

• Practice PromQL against the interview questions and use cases for Prometheus.
• Study incidents in the production issues section (scrape failures, cardinality blow-ups, alert storms).
• Handy tool: compose and validate queries with the PromQL Builder.
• Pair this with the Kubernetes monitoring guide to scrape a whole cluster.

You now have the full loop: expose metrics → scrape → query with PromQL → record → alert → visualize → harden. That's zero to hero.