How does PromQL work -- selectors, rate(), and histogram_quantile()?

Think of PromQL like a spreadsheet formula language designed specifically for time-stamped data. Just as Excel lets you filter rows, apply SUM or AVERAGE, and combine cells, PromQL lets you filter metrics by labels, apply math functions over time windows, and aggregate across dimensions. The difference is that every value in PromQL has a time component attached to it.

PromQL works with two main data types: instant vectors (a set of time series, each with one sample at a single point in time) and range vectors (a set of time series, each with multiple samples over a time window). Selectors are how you choose which metrics to work with. A simple selector like http_requests_total grabs all time series with that name. Adding label matchers narrows it down: http_requests_total{service="payments-api", status_code=~"5.."} selects only 5xx errors for the payments service. The =~ operator does regex matching, while != and !~ provide negative matching.

Under the hood, when Prometheus runs a PromQL query, it first resolves the selector against its TSDB index, which maps label combinations to time series. For range vector selectors like http_requests_total[5m], it fetches all samples from the last 5 minutes for each matched series. Functions then operate on these vectors. The rate() function is essential for counters -- it calculates the per-second average rate of increase over a range vector, automatically handling counter resets (when a process restarts and the counter goes back to zero). The irate() function uses only the last two data points for a more responsive but noisier result. The histogram_quantile() function takes a quantile value between 0 and 1 and a histogram metric to estimate percentiles -- for example, the 99th percentile latency.

In production, PromQL powers both dashboards and alerting rules. A well-designed alert might use rate(http_requests_total{status_code=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01 to fire when the error rate crosses 1%. Recording rules let you pre-compute expensive queries and store the results as new time series, so dashboards load fast instead of recalculating on every refresh. Aggregation operators like sum, avg, max, and min combined with by or without clauses let you roll up metrics -- for example, summing request rates across all pods but keeping the service label.

A common mistake is using rate() on gauges instead of counters -- rate() assumes values only go up and produces garbage on gauges. Use deriv() for gauges instead. Another pitfall is using too short a range with rate(): you need at least two samples in the window, so rate(metric[15s]) with a 15-second scrape interval might return nothing. The rule of thumb is to use a range at least 4 times your scrape interval. For histogram_quantile(), remember that results are estimates based on bucket boundaries. If your buckets are spaced far apart (like 0.1, 1, 10 seconds), the interpolation between them will be inaccurate for values in the gaps.