What is Prometheus and how does its pull-based scraping work?

Think of Prometheus like a health inspector who visits restaurants on a schedule instead of waiting for restaurants to mail in their own reports. The inspector decides when to visit, what to check, and can immediately tell if a restaurant has closed because nobody answers the door. This pull-based approach is the opposite of push-based systems like StatsD where applications send their own metrics to a collector.

Prometheus started at SoundCloud in 2012 and is now a graduated CNCF project built for cloud-native environments. At its core, Prometheus sends HTTP GET requests to /metrics endpoints on your applications and infrastructure at regular intervals (the scrape interval, usually 15 seconds to 1 minute). Each target exposes its current metric values in a standard text format. Prometheus collects these values, timestamps them, and stores them in its local time series database (TSDB).

To find what to scrape, Prometheus uses service discovery. In Kubernetes, it automatically discovers pods, services, and endpoints through the Kubernetes API. The discovery pipeline works in stages: first, targets are found through service discovery configs. Then relabeling rules filter and modify target labels before scraping. Finally, metric relabeling can transform or drop metrics after scraping. This automation means you never need to maintain static lists of IP addresses in a world where containers come and go constantly.

The pull model has big practical advantages. You can run Prometheus on your laptop and point it at staging endpoints for debugging. Your applications do not need to know where Prometheus lives -- they just expose /metrics. Load is predictable because Prometheus controls when it scrapes. The main limitation is that Prometheus must be able to reach your targets over the network, which can be tricky with firewalls or NAT. For short-lived batch jobs that finish before Prometheus can scrape them, there is the Pushgateway as a workaround.

A common mistake is confusing scrape interval with evaluation interval. Scrape interval is how often Prometheus fetches metrics from targets. Evaluation interval is how often it runs recording rules and alerting rules. Setting the scrape interval too low (like 5 seconds) can overwhelm both Prometheus and your targets with too many HTTP requests. Another pitfall is not watching scrape duration -- if a target takes 10 seconds to respond and your scrape interval is 15 seconds, you have very little headroom before timeouts start cascading.