Prometheus

Think of Prometheus like a single security camera with a local hard drive. It records great footage but the drive fills up in two weeks, and if the camera breaks, you lose everything. Thanos is like adding a cloud backup to each existing camera -- a sidecar uploads footage to cheap cloud storage, and a central monitor can search all cameras at once. Mimir is like replacing the local drives entirely -- all cameras stream footage straight to a centralized, scalable recording system in the cloud.

Thanos achieves high availability by running a sidecar container next to each Prometheus instance. The sidecar does two things: it uploads Prometheus's local two-hour TSDB blocks to object storage such as S3, GCS, or MinIO, and it exposes a gRPC Store API that the Thanos Query component can reach. Thanos Query acts as a global query layer. It fans out PromQL queries to all sidecars for recent data and to the Store Gateway for historical data in object storage, then deduplicates results from high-availability Prometheus pairs using external labels. A separate Thanos Compactor merges and downsamples blocks in object storage so long-range queries stay fast.

Mimir, originally called Cortex and donated by Grafana Labs, works differently. Instead of sidecars, each Prometheus sends metrics via remote-write to Mimir's Distributor component. Mimir is built as a set of microservices that you can scale independently. Distributors accept writes and shard them across Ingesters, which hold recent data in memory and periodically flush blocks to object storage. Queriers answer PromQL requests by reading from both Ingesters for fresh data and Store Gateways for older data. Because each piece scales on its own, Mimir handles very large workloads without bottlenecks.

The operational trade-offs come down to complexity versus capability. Thanos is easier to adopt because you keep your existing Prometheus instances and just attach sidecars -- no change to the write path. But the sidecar model means every Prometheus still needs local disk for TSDB blocks, and queries fan out to many sidecars, which can slow down in large environments. Mimir offloads storage entirely from Prometheus so it becomes almost stateless, giving you better multi-tenancy, faster global queries, and no local disk dependency. The trade-off is that you now operate a distributed system with Distributors, Ingesters, Compactors, and Store Gateways, each needing proper resource tuning.

At scale, say 50-plus Prometheus instances and 10 million or more active series, Mimir generally performs better because queries hit centralized, pre-indexed storage rather than fanning out to dozens of sidecars. For smaller setups with 5 to 10 Prometheus instances, Thanos is simpler and perfectly fine. A common migration path is to start with Thanos, then move to Mimir when query performance becomes a bottleneck.

Think of cardinality like a filing cabinet. Each unique set of labels creates a new folder. A metric like http_requests_total{method="GET", status="200"} creates one folder -- low cardinality. But http_requests_total{method="GET", user_id="abc123"} creates a brand new folder for every single user. If you have 10 million users, that is 10 million folders, and the filing cabinet explodes.

In Prometheus, every unique combination of metric name plus label key-value pairs creates a time series. Each active series uses about 1 to 2 KB of memory in the TSDB head block. At 1 million series, that is roughly 2 GB of RAM just for the head block. At 10 million, it is 20 GB. High-cardinality metrics are the number-one cause of Prometheus running out of memory and getting killed, as well as slow queries that time out.

Detection starts with watching Prometheus itself. The metric prometheus_tsdb_head_series tells you the total active series count. If this suddenly jumps, say from 500K to 5M, you have a cardinality explosion. The TSDB status page at /tsdb-status on the Prometheus web UI shows the top 10 metrics by series count and the top 10 labels by unique value count, which immediately reveals the offender. You can also run topk(10, count by (__name__)({__name__=~".+"})) to find the metrics with the most series.

Once you find the culprit, for example a developer who added request_id as a label on a counter, you fix it at multiple levels. The immediate fix is adding a metric_relabel_config in the Prometheus scrape config to drop the high-cardinality label or the entire metric. This takes effect on the next scrape cycle. The medium-term fix is changing the application code to stop emitting the label and using histograms or summaries instead of per-ID counters. The long-term fix is setting up guardrails using tools like Prom-Label-Proxy or Mimir's per-tenant cardinality limits to prevent future explosions.

One important detail: you cannot simply delete old high-cardinality series from Prometheus. Once they are in the TSDB, they stay until the retention period expires. You can use the Admin API at /api/v1/admin/tsdb/delete_series to mark them for deletion, then run clean_tombstones to reclaim space. But this is an emergency measure -- the real fix is always upstream at the source of the metric. The 'context deadline exceeded' error in Prometheus usually means a query is trying to process too many series and timing out, which is a direct symptom of high cardinality.

Think of it like a retail chain with 20 stores. Each store has its own security camera system recording locally. The head office needs to see footage from all stores on one screen. You set up a central recording server and have each store upload its footage with a store ID tag. The head office monitor can then filter by store or show all stores side by side.

The architecture has three layers. First, each Kubernetes cluster runs its own Prometheus instance, or a Prometheus Operator stack, that scrapes all metrics locally. Each Prometheus is configured with a unique external_label, something like cluster=payments-prod-us-east-1, that identifies which cluster the metrics come from. This label is critical because without it you cannot tell metrics from different clusters apart.

Second, each Prometheus uses remote-write to push metrics to a centralized metrics backend. The three main choices are Mimir from Grafana Labs for large-scale setups, Thanos Receive for accepting remote-write and storing in object storage, or VictoriaMetrics for a high-performance and simpler-to-operate option. The remote-write endpoint usually sits behind a load balancer or ingress, secured with mTLS or bearer tokens. Each cluster's Prometheus pushes metrics continuously, typically every 15 to 30 seconds, to this central system.

Third, Grafana connects to the centralized backend as a single Prometheus-compatible data source. Every PromQL query can filter by cluster label: sum(rate(http_requests_total{cluster="payments-prod"}[5m])) for one cluster, or sum by (cluster)(rate(http_requests_total[5m])) for a side-by-side comparison. Dashboard variables use label_values(cluster) to create a dropdown that lets users switch between clusters with a click.

For Grafana itself, in a multi-cluster setup it typically runs in a central management cluster rather than in every cluster. Dashboards and alerts are managed as code using Grafana's provisioning system or tools like Grafonnet and Terraform. This ensures consistency -- updating one dashboard template updates the view for all clusters at once.

The key operational concern is network reliability. If a cluster's Prometheus cannot reach the central backend because of a network partition or backend overload, metrics can be lost. Prometheus buffers remote-write data in a WAL, or write-ahead log, and retries, but the buffer has a size limit of around 300 MB by default. For critical environments, consider running a local Thanos Sidecar or VictoriaMetrics agent as a write buffer that can survive longer outages without dropping data.

Think of it like managing a chain of restaurants where every location has to serve the same menu. Instead of calling each manager and dictating changes over the phone, which is like clicking around in Grafana's UI, you update the master menu in a shared drive, the managers review it, and an automated system prints and ships the new menus to all locations at once.

The GitOps workflow for Grafana has three main approaches, from simple to powerful. The simplest is Grafana's built-in provisioning: you put dashboard JSON files and alert rule YAML files in a directory that Grafana watches, usually mounted via a ConfigMap in Kubernetes. When the files change, Grafana reloads them. You store these files in Git, and your CI/CD pipeline updates the ConfigMap every time a change merges to main.

The second approach uses Grafonnet, a Jsonnet library for generating Grafana dashboard JSON programmatically. Instead of writing raw 500-line JSON files by hand, you write concise Jsonnet code that generates them. This is where updating 10 dashboards at once becomes easy: if all 10 share a common template, say a service dashboard with CPU, memory, error rate, and latency panels, you define the template once and pass in parameters per service. Changing the template changes all 10 dashboards in one commit. Jsonnet compiles down to JSON, which then gets provisioned into Grafana.

The third approach uses Terraform with the Grafana provider. You define dashboards, folders, alert rules, and notification channels as Terraform resources. The CI pipeline runs terraform plan on pull requests to show what would change and terraform apply on merge. This gives you state management, drift detection, and the full Terraform workflow. For large organizations managing hundreds of dashboards across multiple Grafana instances, this is the most maintainable path.

For alerts, Grafana's alerting rules and notification policies can also be defined in YAML and provisioned alongside dashboards. The entire alerting chain -- rules, routing policies, contact points, and message templates -- lives in Git, version-controlled and reviewable.

The day-to-day workflow looks like this: a developer creates a branch, modifies dashboard Jsonnet or Terraform files, opens a pull request, CI runs syntax checks like jsonnet lint or terraform validate and optionally renders a preview, a reviewer approves, the PR merges to main, and the CD pipeline applies changes to Grafana. The big win is that every change is code-reviewed, version-controlled, and reversible with a simple git revert.

Think of it like a car dashboard. Cause-based alerting would mean separate warning lights for every internal part: fuel injector pressure, alternator voltage, coolant thermostat position, oxygen sensor reading. You would have 200 lights and no idea which ones matter. Symptom-based alerting gives you one light that says 'engine temperature high,' and the mechanic investigates the cause from there.

Traditional monitoring creates alerts for every possible internal state: CPU above 80%, disk above 85%, memory above 90%, Pod restarts above 3, queue depth above 1000. This leads to massive alert fatigue. A team with 50 microservices might have 500-plus alert rules, most of which fire for brief spikes that fix themselves. Engineers start ignoring alerts, and when a real outage happens, the critical signal is buried in noise.

Symptom-based alerting flips this around. You alert on what users experience: the error rate is burning through the SLO budget faster than sustainable, latency has crossed the SLO target, or availability has dropped below the threshold. These are called SLI-based alerts, where SLI stands for Service Level Indicator. If CPU is at 95% but the error rate is 0% and latency is normal, there is no user impact, so no alert is needed. If CPU is at 40% but the error rate is 5%, users are hurting, so you alert right away.

The best way to implement this is Google's multi-window, multi-burn-rate approach from the SRE book. You define an SLO such as 99.9% availability over 30 days, which gives you an error budget of 43.2 minutes of allowed downtime. Then you create burn rate alerts. A fast-burn alert fires when the error rate is consuming budget at 14.4 times the sustainable rate, meaning the entire monthly budget would be gone in 2 hours. This catches acute incidents. A slow-burn alert fires at 1 times the sustainable rate held over 3 days, catching gradual degradation. Each alert uses two time windows, a short one like 5 minutes and a long one like 1 hour, so a single brief spike does not trigger a false alarm.

In Prometheus, this translates to recording rules that calculate error ratios over multiple windows, plus alert rules that compare burn rates against thresholds. Grafana displays an SLO dashboard showing remaining error budget, burn rate trends, and alert status. The result: instead of 500 noisy alerts, you might have 10 to 20 SLO-based alerts across all services, each one actionable and tied to real user impact.

Think of a busy warehouse that receives packages all day. New boxes first land in a fast intake area, while a clerk writes every arrival into a paper ledger in case the power fails. Every few hours, the intake area is sorted into sealed pallets and moved to long-term shelves. Prometheus TSDB works the same way: fresh samples are kept hot for fast queries, protected by a write-ahead log, and later packed into efficient on-disk blocks.

In Prometheus, TSDB stands for time series database. It is the local storage engine that keeps metric samples organized by metric name and labels. Prometheus is intentionally a single-node local database, not a clustered durable store. That design keeps ingestion and querying simple, but it puts the burden on operators to size local disk, memory, and retention correctly. The local TSDB is great for recent operational data, but high availability and long-term retention typically require replicas, remote write, Thanos, Mimir, or another system layered on top.

The write path starts when a scrape returns samples. Prometheus checks the labels, appends each sample to the head block in memory, and writes a record to the WAL so a restart can replay recent data. The active head block covers the most recent samples and periodically gets frozen into an immutable block, usually covering a two-hour window. Older blocks are merged together in the background through a process called compaction. Index files map label sets to chunk data, tombstones record deletions, and retention cleanup removes blocks that have fully expired.

In production, head data and WAL data are where the pressure peaks. Operators watch active series count, ingestion rate, WAL fsync failures, checkpoint duration, available disk, compaction failures, and query latency. Retention based on time alone can be misleading when a team suddenly adds a high-cardinality label like customer_id. Size-based retention should leave headroom because the WAL and head chunks still need space while background cleanup catches up. Prometheus expects local POSIX storage; network filesystems are a reliability risk for the TSDB.

The gotcha that catches people off guard is that retention cleanup is block-oriented and delayed, but the outage is often immediate. If the disk fills because the WAL is growing or the head block is huge, lowering retention from 30 days to 15 days may not save the node right away. Deleting series through the admin API writes tombstones first and does not instantly shrink chunk files. Removing WAL files can let Prometheus start again, but you lose recent samples. Experienced operators first stop the source of excess cardinality, then recover disk, then tune retention and remote storage to prevent it from happening again.

Think of a train station display board. If the 8:10 train reported its location two minutes ago, the board can still show a useful last-known position. If the train has not reported for an hour, showing that old position would mislead passengers. Prometheus has the same problem with metrics: a recent sample is fine to use at query time, but an old sample should eventually disappear so graphs and alerts do not pretend the system is healthy.

PromQL, the Prometheus query language, evaluates instant queries at a single timestamp and range queries at many evenly spaced timestamps. For each evaluation timestamp, Prometheus looks backward for the newest sample inside the lookback window. The default lookback is 5 minutes, and it is configurable. This lets queries work even when scrapes do not land exactly on graph step boundaries. Without this behavior, normal scrape timing jitter would create broken graphs and unreliable aggregations.

Staleness adds another layer. If a target scrape no longer returns a series that previously existed, or if service discovery removes a target entirely, Prometheus can write a staleness marker for that time series. After that marker, instant queries no longer return the old value for that series. This prevents stale readings from being treated as current values in aggregations like sum, avg, or alert expressions. If fresh samples later arrive for the same label set, the series simply reappears.

Production alerting gets subtle here. An alert like up == 0 catches failed scrapes where the target is still known but unreachable. However, it may not catch a target that vanished from service discovery, because there may be no up series left to evaluate. For detecting missing services, absent() or inventory-based alerts are usually needed. Engineers also tune scrape_interval, scrape_timeout, evaluation_interval, and the alert for duration so brief network hiccups do not page people while true disappearances still get caught quickly.

The experienced gotcha is that a graph can look flat or empty for different reasons. A flat line may mean Prometheus is carrying a recent last sample inside the lookback window, while an empty graph may mean the series went stale, not that the value became zero. Exporters that attach their own timestamps can behave differently and may keep the last value visible until lookback expires. A common band-aid is using or vector(0) everywhere, which makes dashboards look tidy but hides missing telemetry. Senior engineers learn to distinguish between zero, missing, stale, and failed-scrape states explicitly rather than papering over the differences.

Think of measuring checkout wait times by placing customers into labeled bins: under 100 ms, under 300 ms, under 1 second, and so on. If the bins are chosen around the thresholds the business actually cares about, the data is useful. If every bin is too wide, too narrow, or missing the SLO boundary, the final percentile looks precise but answers the wrong question. Prometheus histograms are that binning system for measurements like request duration.

A classic Prometheus histogram exposes cumulative bucket counters using the le label, which stands for less than or equal, plus _sum and _count series. Prometheus calculates percentiles using the histogram_quantile() function over rates of those buckets. The big advantage of this design is that you can aggregate across pods, nodes, clusters, or jobs before calculating the percentile, which is why histograms are the go-to for distributed services. The cost is extra time series: each bucket boundary creates another series for every label combination.

Native histograms change the storage model by representing many bucket spans more compactly and letting Prometheus handle histogram samples directly. They reduce some of the pain of choosing bucket boundaries manually and support more flexible percentile exploration. However, they require compatible Prometheus settings, client libraries, remote write backends, and query paths, so you need to check the full chain before adopting them. Summaries are a different animal: they compute selected quantiles inside each application process. That can be useful for a single process, but averaging p95 values across replicas is statistically wrong because each process saw a different number and shape of requests.

The query path matters for getting correct results. For classic histograms, you typically apply rate() to _bucket counters, aggregate with sum by (le, service) or similar, then call histogram_quantile(). The le label must survive until the quantile function runs because it represents the bucket boundary. For SLO checks like seeing what fraction of requests finish under 300 ms, having an exact bucket at that boundary makes the calculation simple and reliable. For Apdex-style scores, you need buckets at both the satisfied and tolerated thresholds.

The gotcha is that histogram percentiles are estimates, and how good the estimate is depends entirely on where you place the buckets. A p99 alert built on buckets of 100 ms, 1 second, and 10 seconds cannot accurately tell the difference between 1.2 seconds and 8 seconds. Another common mistake is averaging per-pod p95 values in Grafana, which gives equal weight to quiet pods and busy pods. Experienced engineers pick bucket boundaries around the SLO thresholds users care about, keep labels low-cardinality, aggregate buckets before computing quantiles, and verify that the remote storage path preserves the histogram type they depend on.

Think of a shipping dock that sends packages from a factory to a central warehouse. The factory keeps making boxes, workers load them into several truck lanes, and sometimes the warehouse slows down. If the lanes fill up, boxes pile up at the dock. If the warehouse stays closed for hours, the factory has to choose between halting intake, using more dock space, or eventually throwing away boxes it can no longer hold.

Prometheus remote write is that shipping dock for metrics. Prometheus first ingests samples locally through its normal scrape and WAL path. A remote write component then reads from the WAL, maps internal series IDs to their label sets, queues samples, and sends compressed HTTP requests to the configured remote endpoint. That endpoint might be Grafana Mimir, Thanos Receive, Cortex, VictoriaMetrics, or a managed cloud service. Remote write is not a magic way to backfill historical data; it is mainly a streaming replication path from the local ingestion flow.

Backpressure shows up when the remote endpoint is slow, returning errors, rate-limiting, or totally unreachable. Prometheus uses shards, which are parallel sending workers, to improve throughput. Each shard has an in-memory queue with a capacity limit and a maximum batch size. Failed requests get retried with exponential backoff. Prometheus can automatically adjust the shard count based on the incoming sample rate and how long sends are taking. But once queues fill up, reading from the WAL for that remote write target is blocked, and pending samples start piling up.

In production, the key metrics to watch are pending samples, failed samples, retried samples, send batch duration, current shard count, and queue capacity. Tuning usually starts with the receiver side: confirm it is healthy, not throttling, and not rejecting samples due to tenant limits or bad labels. Then tune Prometheus settings like max_samples_per_send, capacity, max_shards, and backoff values. Capacity should generally be several times the batch size, but setting it too high increases Prometheus memory usage. Write relabeling can drop expensive or unnecessary samples before they even leave Prometheus.

The gotcha is that cranking every knob up can make the outage worse. More shards can overwhelm a backend that is trying to recover. More queue capacity can cause Prometheus memory pressure, especially during high series churn because remote write caches series labels. Another gotcha involves the two-hour WAL window: if remote write stays blocked longer than the WAL can hold unsent data, samples get lost when the WAL is compacted. Senior engineers treat remote write tuning as end-to-end flow control, not just a matter of making the queue bigger.

Think of a hospital emergency department during a city-wide power failure. Thousands of alarms pour in from buildings, traffic lights, and elevators. Operators do not want a separate phone call for each alarm. They want one grouped incident per affected area, with enough detail to know which buildings still need help. Alertmanager is that dispatch layer for Prometheus alerts.

Prometheus evaluates alerting rules and sends firing or resolved alerts to Alertmanager over HTTP. Alertmanager then groups alerts by chosen labels, routes each group through a routing tree to the right receiver (Slack, PagerDuty, email), deduplicates repeated notifications, applies silences for planned maintenance, and applies inhibitions when one alert makes another redundant. For example, if a ClusterDown alert is firing, an inhibition rule can suppress thousands of pod-level alerts from that same cluster because they are all symptoms of the same root cause.

Grouping is label-driven. The group_by setting picks which labels define a notification group. group_wait delays the first notification briefly so related alerts can arrive together. group_interval controls how often new alerts get added to an existing group. repeat_interval controls how frequently unresolved alerts are re-sent. Inhibition rules compare a source alert against target alerts using matchers and equality labels. Silences use matchers and time windows, and are usually created through the Alertmanager UI or API during maintenance windows.

For high availability, multiple Alertmanager instances form a cluster and share notification state through a gossip protocol. Prometheus should be configured with all Alertmanager peers listed as targets. The Prometheus docs warn against putting a load balancer between Prometheus and Alertmanager because each Prometheus instance needs to deliver alerts to the full cluster so deduplication and state replication work correctly. Teams also set external labels like cluster, region, and replica carefully so Alertmanager can tell independent environments apart while still deduplicating HA Prometheus replicas.

The gotcha is that label design can either flood your team or hide a real outage. If group_by includes pod, every single pod failure during a deployment becomes a separate page. If it only groups by alertname, unrelated production and staging incidents might collapse into one notification. Inhibition can be dangerous too -- if the source alert is too broad or fires too easily, it can silence real alerts. Senior engineers test alert routes with sample payloads, keep grouping labels tied to ownership and blast radius, and regularly review active silences to make sure planned maintenance windows have not turned into black holes for real incidents.

Think of Prometheus like a health inspector who visits restaurants on a schedule instead of waiting for restaurants to mail in their own reports. The inspector decides when to visit, what to check, and can immediately tell if a restaurant has closed because nobody answers the door. This pull-based approach is the opposite of push-based systems like StatsD where applications send their own metrics to a collector.

Prometheus started at SoundCloud in 2012 and is now a graduated CNCF project built for cloud-native environments. At its core, Prometheus sends HTTP GET requests to /metrics endpoints on your applications and infrastructure at regular intervals (the scrape interval, usually 15 seconds to 1 minute). Each target exposes its current metric values in a standard text format. Prometheus collects these values, timestamps them, and stores them in its local time series database (TSDB).

To find what to scrape, Prometheus uses service discovery. In Kubernetes, it automatically discovers pods, services, and endpoints through the Kubernetes API. The discovery pipeline works in stages: first, targets are found through service discovery configs. Then relabeling rules filter and modify target labels before scraping. Finally, metric relabeling can transform or drop metrics after scraping. This automation means you never need to maintain static lists of IP addresses in a world where containers come and go constantly.

The pull model has big practical advantages. You can run Prometheus on your laptop and point it at staging endpoints for debugging. Your applications do not need to know where Prometheus lives -- they just expose /metrics. Load is predictable because Prometheus controls when it scrapes. The main limitation is that Prometheus must be able to reach your targets over the network, which can be tricky with firewalls or NAT. For short-lived batch jobs that finish before Prometheus can scrape them, there is the Pushgateway as a workaround.

A common mistake is confusing scrape interval with evaluation interval. Scrape interval is how often Prometheus fetches metrics from targets. Evaluation interval is how often it runs recording rules and alerting rules. Setting the scrape interval too low (like 5 seconds) can overwhelm both Prometheus and your targets with too many HTTP requests. Another pitfall is not watching scrape duration -- if a target takes 10 seconds to respond and your scrape interval is 15 seconds, you have very little headroom before timeouts start cascading.

Think of PromQL like a spreadsheet formula language designed specifically for time-stamped data. Just as Excel lets you filter rows, apply SUM or AVERAGE, and combine cells, PromQL lets you filter metrics by labels, apply math functions over time windows, and aggregate across dimensions. The difference is that every value in PromQL has a time component attached to it.

PromQL works with two main data types: instant vectors (a set of time series, each with one sample at a single point in time) and range vectors (a set of time series, each with multiple samples over a time window). Selectors are how you choose which metrics to work with. A simple selector like http_requests_total grabs all time series with that name. Adding label matchers narrows it down: http_requests_total{service="payments-api", status_code=~"5.."} selects only 5xx errors for the payments service. The =~ operator does regex matching, while != and !~ provide negative matching.

Under the hood, when Prometheus runs a PromQL query, it first resolves the selector against its TSDB index, which maps label combinations to time series. For range vector selectors like http_requests_total[5m], it fetches all samples from the last 5 minutes for each matched series. Functions then operate on these vectors. The rate() function is essential for counters -- it calculates the per-second average rate of increase over a range vector, automatically handling counter resets (when a process restarts and the counter goes back to zero). The irate() function uses only the last two data points for a more responsive but noisier result. The histogram_quantile() function takes a quantile value between 0 and 1 and a histogram metric to estimate percentiles -- for example, the 99th percentile latency.

In production, PromQL powers both dashboards and alerting rules. A well-designed alert might use rate(http_requests_total{status_code=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01 to fire when the error rate crosses 1%. Recording rules let you pre-compute expensive queries and store the results as new time series, so dashboards load fast instead of recalculating on every refresh. Aggregation operators like sum, avg, max, and min combined with by or without clauses let you roll up metrics -- for example, summing request rates across all pods but keeping the service label.

A common mistake is using rate() on gauges instead of counters -- rate() assumes values only go up and produces garbage on gauges. Use deriv() for gauges instead. Another pitfall is using too short a range with rate(): you need at least two samples in the window, so rate(metric[15s]) with a 15-second scrape interval might return nothing. The rule of thumb is to use a range at least 4 times your scrape interval. For histogram_quantile(), remember that results are estimates based on bucket boundaries. If your buckets are spaced far apart (like 0.1, 1, 10 seconds), the interpolation between them will be inaccurate for values in the gaps.

Think of recording rules like a restaurant that preps ingredients before the dinner rush. Instead of chopping vegetables from scratch for every order, the kitchen pre-chops during quiet hours. Recording rules pre-compute expensive PromQL queries on a schedule so dashboards load instantly. Alerting rules are like a smoke detector: they continuously check a condition and sound the alarm when something crosses a threshold for long enough to be a real problem.

Both types of rules are defined in YAML files and loaded by Prometheus through the rule_files config. They are organized into rule groups, where each group has a name and an optional evaluation interval. Recording rules have a record field (the name of the new metric to create) and an expr field (the PromQL expression to evaluate). The naming convention follows the pattern level:metric:operations -- for example, namespace:http_requests_total:rate5m tells you the aggregation level, the base metric, and what operation was applied. Alerting rules have an alert field (the alert name), an expr field, an optional for duration, labels to attach, and annotations for human-readable descriptions.

Under the hood, Prometheus evaluates rules within each group sequentially but can run multiple groups in parallel. The evaluation interval defaults to the global setting but can be overridden per group. For recording rules, each evaluation writes a new sample to the TSDB with the current timestamp. For alerting rules, the evaluation produces one of three states: inactive (the expression returned nothing), pending (the expression matched but the for duration has not passed yet), or firing (the expression has been true for at least the for duration). When an alert hits firing state, Prometheus sends it to all configured Alertmanagers.

In production, recording rules are essential for scaling dashboards. Without them, 50 engineers opening the same Grafana dashboard during an incident would each trigger the same expensive aggregation 50 times per refresh. Recording rules compute it once and store the result. A common pattern is building a pyramid: raw metrics get aggregated into per-service rates, then those rates get aggregated into per-team totals. For alerting, the for clause is critical -- it prevents false alarms from momentary spikes. A for: 5m clause means the condition must be continuously true for 5 minutes before the alert fires.

A key gotcha with recording rules is circular dependencies. If rule A depends on the output of rule B, both must be in the same group with B listed first, because rules within a group run sequentially. Across groups, evaluation order is not guaranteed. For alerting rules, a common mistake is leaving out the for clause entirely, which causes alerts to fire on every brief spike. Another pitfall is hardcoding values in annotations instead of using template variables. Always include {{ $labels.instance }} and {{ $value }} in your annotation templates so on-call engineers can immediately see which target is affected and how bad it is.

Imagine a library that organizes books by when they were written, not by author or title. Recent books sit on a desk for quick access (the in-memory head block). Every two hours, the librarian packs the desk books into sealed boxes (on-disk blocks), compresses them, and shelves them in the archive. A journal (the WAL, or write-ahead log) records every new book as it arrives so that if the librarian is interrupted, they can replay the journal and recover. This is essentially how Prometheus TSDB works.

Prometheus TSDB is a custom-built time series database optimized for the append-heavy, label-indexed workload of metrics. Each unique combination of metric name and labels forms a time series, identified by a numeric series ID. The database has two main parts: the head block (in-memory, mutable) containing the most recent 2+ hours of data, and persistent blocks (on-disk, immutable) containing older data. When a sample arrives, it goes to the head block and the WAL simultaneously. The WAL is a sequential log that ensures no data loss if Prometheus crashes -- on restart, it replays the WAL to rebuild the head block.

Each on-disk block is a directory with three key files. The index file maps label sets to series IDs and provides posting lists for fast label-based lookups (similar to a search engine's inverted index). Chunk files store the actual timestamp-value pairs compressed using Gorilla encoding -- double-delta encoding for timestamps and XOR encoding for values, achieving roughly 1.37 bytes per sample. A meta.json file records the block's time range and statistics. The head block uses the same chunk format but keeps everything in memory. When the head block covers more than 2 hours of data, it gets compacted into an on-disk block. Background compaction then merges smaller blocks into larger ones (up to 31 hours by default), reducing block count and improving query performance.

In production, understanding these internals helps with capacity planning. Storage size depends on three things: number of active time series (cardinality), scrape interval (samples per series per day), and retention period. A rough formula: disk_size = num_series x samples_per_day x bytes_per_sample x retention_days. With 1 million active series at a 15-second scrape interval and 15-day retention, expect about 120 GB of disk. The flags --storage.tsdb.retention.time and --storage.tsdb.retention.size let you cap storage. Remote write can offload long-term data to systems like Thanos or Mimir.

The biggest gotcha is high cardinality -- creating time series with labels like user IDs, request IDs, or IP addresses explodes the index size and can crash Prometheus. The index grows proportionally with unique label combinations. Another pitfall is ignoring WAL size. If Prometheus cannot compact fast enough (usually due to CPU or disk I/O saturation), the WAL grows without bound and can fill the disk. The metric prometheus_tsdb_wal_segment_current shows the current WAL segment number, and a rapidly increasing value means compaction is falling behind.

Imagine you hire a property manager for your apartment building. Instead of personally meeting each new tenant and updating the paperwork, you tell the manager: anyone who moves into floors 3 through 5 gets set up with water, electricity, and parking. The property manager (Prometheus Operator) watches for new tenants (services), matches them against your rules (ServiceMonitors), and handles all the configuration automatically. You never edit the master list by hand.

The Prometheus Operator is a Kubernetes controller that manages Prometheus deployments and their configuration through Custom Resource Definitions (CRDs). It was originally created by CoreOS and is the backbone of the popular kube-prometheus-stack Helm chart. The Operator watches for changes to its CRDs and keeps the actual state in sync with what you declared. The main CRDs are: Prometheus (defines a Prometheus instance), ServiceMonitor (defines which Kubernetes Services to scrape), PodMonitor (scrapes pods directly without needing a Service), PrometheusRule (defines recording and alerting rules), and Alertmanager (manages Alertmanager instances).

When you create a ServiceMonitor, the Operator reads its spec, which includes a selector (label matcher for Kubernetes Services), namespaceSelector (which namespaces to look in), and endpoints (port names, paths, and scrape intervals). The Operator then generates the equivalent Prometheus scrape_config YAML and loads it into the Prometheus pod. The important part is that the Prometheus CRD has a serviceMonitorSelector field that controls which ServiceMonitors it picks up -- if your ServiceMonitor labels do not match this selector, Prometheus will silently ignore it. The Operator also supports serviceMonitorNamespaceSelector to limit which namespaces are watched.

In production, the Prometheus Operator is the standard way to run Prometheus on Kubernetes. Teams create ServiceMonitors alongside their application deployments, often in the same Helm chart. This means the team deploying payments-api also defines how it gets monitored. The Operator supports advanced features like sharding (spreading targets across multiple Prometheus instances), Thanos sidecar integration for long-term storage, and automatic TLS for scrape endpoints. It also handles version upgrades, resource management, and persistent volumes for TSDB storage.

The most common gotcha is the label mismatch problem: you create a ServiceMonitor, but Prometheus never scrapes your targets. This happens because the Prometheus CRD's serviceMonitorSelector does not match your ServiceMonitor's labels. The kube-prometheus-stack Helm chart defaults to requiring release: kube-prometheus-stack as a label, so every ServiceMonitor must include it. Debug by running kubectl get prometheus -o yaml and checking the serviceMonitorSelector field. Another pitfall is port name mismatches -- the ServiceMonitor endpoints.port field must match the port name in the Kubernetes Service spec, not the container port number.

Think of Alertmanager as a hospital triage system. Patients (alerts) arrive and are grouped by condition. Cardiac cases go to cardiology, broken bones go to orthopedics (routing rules). If the hospital is doing planned maintenance on radiology machines, they put up a sign saying ignore false alarms from 2am to 4am (silences). The triage nurse does not page the doctor twice for the same patient (deduplication), and waits a few minutes to batch patients arriving together (group_wait).

Alertmanager is a separate process that receives alerts from one or more Prometheus servers through its /api/v2/alerts endpoint. Its configuration defines receivers (notification channels like Slack or PagerDuty), a routing tree (which alerts go where), inhibition rules (suppress certain alerts when others are already firing), and templates for formatting notifications. The routing tree starts with a root route that has a default receiver. Child routes match on alert labels using matchers. Routes are evaluated top to bottom, and the first matching child wins unless continue: true is set, which lets evaluation continue to the next sibling.

Grouping is Alertmanager's most important noise reduction feature. When group_by is set to something like [service, environment], all alerts with the same service and environment label values get bundled into a single notification. Three timing settings control notification behavior: group_wait is how long to wait for more alerts before sending the first notification for a new group (default 30 seconds), group_interval is the minimum time between updates to an existing group when new alerts arrive (default 5 minutes), and repeat_interval is how long to wait before resending an unresolved alert (default 4 hours). Getting these right is the difference between a useful alert system and one that either floods your phone or misses real problems.

In production, a well-designed routing tree mirrors your organization's on-call structure. Critical payment alerts go to PagerDuty for immediate paging. Warning-level alerts for batch jobs go to a Slack channel. Silences are created through the Alertmanager UI or API and match alerts by label matchers -- they are essential during deployments and maintenance windows. Inhibition rules automatically suppress downstream alerts: when the entire cluster is unreachable (KubeAPIDown), you do not want 500 pod alerts flooding the channel. The inhibition rule says if KubeAPIDown is firing, suppress all alerts with the same cluster label.

A common mistake is setting group_by to too many labels, like [service, pod, instance]. This creates one notification per pod, which defeats the purpose of grouping. On the other hand, the special value group_by: ['...'] groups nothing -- every alert becomes its own group. Another pitfall is setting repeat_interval too low, causing alert fatigue from constant re-notifications for chronic issues. The sweet spot is usually 4 to 12 hours. Engineers also forget that silences expire. If you create a 1-hour silence for a deployment that takes 2 hours, alerts will resume halfway through. Always add generous padding to silence durations.

Think of these four metric types like different measuring tools. A counter is like a car odometer -- it only goes up, and it resets to zero when the car is new (process restart). A gauge is like a speedometer -- it moves up and down to show the current state. A histogram is like a teacher sorting exam scores into grade buckets (A, B, C, D, F) -- you know how many scores fell into each range. A summary is like having a statistician who continuously calculates the median and 90th percentile of those scores on the fly.

Counters are the most common type. They track cumulative totals that only increase or reset to zero on restart. Examples include http_requests_total, errors_total, and bytes_sent_total. You almost always use rate() or increase() with counters because the raw number is meaningless on its own -- it depends on when the process started. Gauges show a snapshot of a current value that can move up or down: memory usage, temperature, active connections, queue depth. Unlike counters, gauges are directly useful as-is -- you can use max_over_time(), min_over_time(), or avg_over_time() to track how they change.

Histograms and summaries both measure distributions (usually latencies), but they work in very different ways. A histogram tells the application to count how many observations fall into preset buckets. For example, http_request_duration_seconds with buckets [0.01, 0.05, 0.1, 0.5, 1, 5] creates a counter for each bucket boundary using the _bucket metric with le (less than or equal) labels. It also creates _count (total observations) and _sum (sum of all observed values). You must set bucket boundaries when you write the code, and changing them requires a redeploy. A summary calculates percentiles (like p50, p90, p99) inside each application process using a streaming algorithm. It gives you ready-made percentile values, but you cannot combine them across instances -- averaging p99 values from different pods gives a statistically wrong result.

In production, histograms are almost always the better choice for latency measurement. The big advantage is that histogram_quantile() can combine data across all your instances -- you get the true overall p99 for your service, not just per-pod numbers. Summaries cannot do this. However, histograms need thoughtful bucket selection. If your API has a p99 of 200ms but your smallest bucket is 1 second, the calculated percentile will be wrong because Prometheus has to guess between wide bucket gaps. The default Prometheus client buckets (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10) work well for typical HTTP latencies. Watch out for cardinality with histograms -- each label combination creates buckets_count + 2 time series.

A critical mistake is applying rate() to a gauge -- it gives nonsensical results because rate() assumes the value only goes up. Similarly, looking at a counter's raw value is almost never useful. Another common error is creating histograms with too many buckets or too many label dimensions, causing a cardinality explosion. With 20 buckets and 5 label dimensions each having 10 values, one histogram metric creates 20 x 10^5 = 2 million time series. Also remember that _bucket values are cumulative: le="0.5" counts ALL observations at or below 0.5 seconds, not just those between 0.25 and 0.5.

Think of monitoring a factory. kube-state-metrics is like the floor manager who tracks the status of every work order: how many assembly lines are running, which ones are behind schedule, which workers are idle. node-exporter is like the building engineer who monitors the physical plant: power consumption, HVAC temperature, water pressure, and structural integrity. You need both perspectives -- the logical work status and the physical infrastructure health -- to understand what is really going on.

kube-state-metrics (KSM) is a service that connects to the Kubernetes API server and turns Kubernetes object states into Prometheus metrics. It does not instrument applications or nodes -- it purely converts the API's data into time series. For example, it exposes kube_deployment_spec_replicas (how many replicas you asked for), kube_deployment_status_replicas_available (how many are actually available), kube_pod_status_phase (whether a pod is Pending, Running, Succeeded, Failed, or Unknown), and kube_pod_container_resource_requests (CPU/memory requests). KSM runs as a single deployment (or sharded for large clusters) and exposes metrics on port 8080. It puts zero load on individual nodes or pods.

node-exporter is a Prometheus exporter that runs as a DaemonSet on every Kubernetes node, exposing hardware and OS metrics. It reads from /proc and /sys on Linux to report things like node_cpu_seconds_total (CPU usage per core per mode), node_memory_MemAvailable_bytes (available memory), node_filesystem_avail_bytes (disk space), node_network_receive_bytes_total (network traffic), and node_disk_io_time_seconds_total (disk I/O). Each node-exporter instance exposes its local node's metrics on port 9100. On Kubernetes, it is deployed as a DaemonSet with hostPID, hostNetwork, and volume mounts to /proc and /sys.

In production, combining both exporters unlocks powerful correlation queries. For example, you can spot resource contention by comparing kube_pod_container_resource_requests_cpu_cores (what pods asked for) against node_cpu_seconds_total (what the node is actually using). You can build utilization dashboards showing requested versus actual versus capacity for every node. The kube-prometheus-stack Helm chart deploys both exporters along with Prometheus, Alertmanager, and pre-built Grafana dashboards and alert rules. Common alerts include KubePodNotReady (pods stuck in a non-ready state using KSM metrics), NodeMemoryHighUtilization (node running low on memory using node-exporter metrics), and KubeDeploymentReplicasMismatch (desired versus actual replicas using KSM).

A key gotcha with kube-state-metrics is that it does not show actual resource usage -- it only reports what was requested and what the limits are. To see real CPU and memory consumption, you need metrics-server or cAdvisor (container_cpu_usage_seconds_total, container_memory_usage_bytes), which are built into the kubelet. Another pitfall is node-exporter's filesystem metrics: by default, it mounts the host's /proc and /sys into the container, but if those mounts are set up wrong, metrics will reflect the container's filesystem instead of the host's. On managed Kubernetes services (EKS, GKE, AKS), some kernel-level metrics may be blocked by the cloud provider's security boundaries.

Think of a library system across a city. Each branch library (Prometheus instance) keeps recent books on its shelves, but older books go to a central warehouse (object storage). When a researcher wants to search across all branches and the warehouse at the same time, a central catalog system (Thanos Querier) knows where every book is and fetches it from the right place. Thanos does exactly this for Prometheus metrics.

Thanos was built to fix two basic Prometheus limitations: local storage is not durable (if the node dies, metrics are lost), and a single Prometheus can only query its own data. In a multi-cluster environment with 15 Kubernetes clusters, each running its own Prometheus, there is no built-in way to query across all clusters or keep metrics beyond the local retention period (typically 15-30 days). Thanos adds a sidecar to each Prometheus that uploads completed TSDB blocks to object storage (S3, GCS, or Azure Blob), a Store Gateway that serves historical blocks from object storage, a Querier that merges results from sidecars and Store Gateways, and a Compactor that downsamples and compacts blocks for faster long-range queries.

Here is the detailed flow. Prometheus writes 2-hour TSDB blocks to local disk. The Thanos Sidecar watches the Prometheus data directory and uploads completed blocks to the object storage bucket. Each block contains a meta.json describing its time range, labels, and resolution. The Querier implements the Prometheus HTTP API and receives PromQL queries. It fans out to all connected StoreAPI endpoints -- sidecars for recent data and Store Gateways for historical data -- deduplicates overlapping series using external labels, and returns merged results. The Compactor runs as a singleton (meaning exactly one instance), downloading blocks from object storage, merging overlapping blocks, creating downsampled versions at 5-minute and 1-hour resolutions, and re-uploading the compacted blocks. This cuts storage costs and speeds up queries over long time ranges.

At production scale, the Store Gateway is the most resource-hungry component because it must cache block index headers in memory to answer queries quickly. A cluster with 500 million active time series and 1 year of retention may have hundreds of thousands of blocks, requiring Store Gateway instances with 32-64 GB of memory for index caching. The Compactor must keep up with block production -- if it falls behind, queries over historical data slow down because the Querier has to open many small blocks instead of a few large ones. Architects should watch thanos_compact_group_compactions_failures_total, thanos_store_bucket_cache_hits_total, thanos_query_store_api_duration_seconds, and overall object storage bucket size and cost.

The sneaky gotcha is that Thanos deduplication depends on consistent external labels. If two Prometheus instances scrape the same targets but have different or missing external labels, the Querier cannot deduplicate correctly and returns duplicate series that produce wrong aggregation results. Another trap is that the Compactor is a singleton -- running two Compactors against the same bucket causes data corruption because they fight over the same blocks. Teams using Thanos Compactor must guarantee exactly-one behavior, typically via a Kubernetes Deployment with replicas: 1 and a PodDisruptionBudget that prevents eviction during compaction.

Think of a news organization. Hierarchical federation is like regional bureaus sending headline summaries to the national desk -- the national editor gets the big picture without reading every local article. Cross-service federation is like the sports desk borrowing a specific stat from the finance desk's data feed. In both cases, only selected information flows upward or sideways, not everything.

Prometheus federation uses the /federate endpoint to expose a subset of metrics from one Prometheus instance so another Prometheus can scrape them. In hierarchical federation, a global Prometheus sits above multiple datacenter or cluster-level Prometheus instances. Each lower-level instance runs recording rules that pre-aggregate raw metrics into summary time series -- for example, computing the 99th percentile request latency per service every minute. The global Prometheus scrapes only these aggregated metrics from the /federate endpoint of each lower-level instance, giving it a fleet-wide view without ingesting the raw per-pod or per-instance metrics.

Under the hood, the /federate endpoint accepts match[] parameters that filter which metrics are exposed. The global Prometheus configures a scrape job with honor_labels: true to keep the original labels from the source instances, and metrics_path: /federate with params that specify the match expressions. Cross-service federation uses the same mechanism but horizontally: the payments team's Prometheus scrapes specific metrics like http_requests_total or circuit_breaker_state from the user-auth team's Prometheus to monitor a critical dependency. The match expression is narrow, pulling only the exact metrics needed rather than the entire dataset.

At production scale, hierarchical federation works well when the global Prometheus only needs aggregated views -- overall error rates, cluster-level resource utilization, SLO compliance percentages. It falls short when engineers need to drill into full-resolution metrics for incident investigation because the global instance only has pre-aggregated data. That is where Thanos or remote write to a central TSDB like Cortex, Mimir, or VictoriaMetrics becomes necessary. Remote write pushes all raw metrics from every Prometheus to a central store, enabling full-resolution global queries at the cost of more storage and ingestion infrastructure. Pick hierarchical federation for cost-effective fleet-wide dashboards, cross-service federation for targeted dependency monitoring with minimal coupling, and Thanos or remote write when incident responders need full-resolution cross-cluster queries.

The sneaky gotcha is that federation creates a scrape-interval dependency. The global Prometheus scrapes the /federate endpoint at its own interval (typically 60 seconds), but the source metrics were produced at the source's scrape interval (typically 15-30 seconds). This creates staleness windows where the global view lags behind reality. If the global scrape interval is longer than twice the source's recording rule evaluation interval, data points can be missed entirely. Another trap is that /federate is expensive for the source Prometheus -- serving thousands of metrics via /federate on every scrape adds CPU and memory pressure to the source. Teams should use recording rules to keep the number of series exposed via /federate small rather than using broad match expressions that pull raw metrics.

Think of two ways to archive office documents. The Sidecar approach is like each department scanning its own files and uploading them to cloud storage -- simple and decentralized, but each department needs cloud access. The Receive approach is like all departments sending their files to a central mailroom that handles scanning, copies, and archiving -- more infrastructure in the middle, but departments need no cloud access and the mailroom can sort files by department.

Thanos Sidecar runs as a container alongside each Prometheus Pod. It watches the Prometheus data directory for completed TSDB blocks (every 2 hours) and uploads them to object storage. It also exposes a StoreAPI gRPC endpoint that the Querier uses to access recent data still in Prometheus's local TSDB. This model is pull-based: Prometheus writes to local disk as usual, and the Sidecar pulls completed blocks into object storage. The main advantages are simplicity (no extra stateful components), tight coupling with the Prometheus lifecycle, and the ability to serve recent data with zero extra latency since it reads directly from Prometheus's local TSDB.

Thanos Receive implements the Prometheus remote write API. Prometheus instances are configured with remote_write to send metrics to a Receive cluster. Receive ingests the data, applies tenant labels, replicates across Receive instances for durability (configurable replication factor, typically 3), and writes TSDB blocks to local disk before uploading to object storage. The Receive cluster is stateful and must be carefully sized for ingestion throughput, disk IOPS, and memory. It serves both recent and locally-stored historical data via StoreAPI.

At production scale, the decision depends on your infrastructure and organizational needs. Sidecar is the go-to pattern when Prometheus runs in Kubernetes with direct access to object storage (S3, GCS), when the team wants minimal extra infrastructure, and when HA is handled by running duplicate Prometheus pairs with the Querier deduplicating via external labels. Receive is the better choice when Prometheus runs in edge locations, on-premise datacenters, or environments where direct object storage access is blocked by network policy or compliance rules. Receive also enables multi-tenancy: each tenant's metrics can be routed to specific Receive instances with tenant-level resource limits and retention. The Receive hashring distributes incoming series across instances based on tenant and metric labels, allowing horizontal scaling of ingestion.

The sneaky gotcha with Sidecar is the 2-hour upload delay -- completed TSDB blocks are only uploaded after Prometheus compacts them, so there is a window where data exists only on Prometheus's local disk. If the Prometheus Pod crashes before upload, that block can be lost (though HA pairs reduce this risk). With Receive, the gotcha is operational complexity: Receive is a stateful distributed system that requires careful hashring configuration, anti-affinity scheduling, and monitoring of replication lag. If a Receive instance falls behind on ingestion, back-pressure can cause Prometheus remote write queues to grow, eventually dropping data. Architects must size Receive for peak ingestion rate plus a 30 percent buffer, and watch thanos_receive_write_failures_total and remote_storage_queue_highest_sent_timestamp_seconds.

Think of a city tax office processing returns. If every clerk individually calculates the city's total revenue by reading every single return, the office grinds to a halt. Instead, each department pre-computes its subtotal (recording rule), and the summary report (dashboard query) just adds up the subtotals. PromQL optimization follows the same idea -- pre-compute expensive aggregations so that real-time queries stay cheap.

High cardinality in Prometheus means having many unique label combinations for a single metric. A metric like http_request_duration_seconds with labels for service, method, path, status_code, pod, and customer_id can produce millions of unique time series. When a PromQL query touches all of these series -- for example, histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) without any label filter -- Prometheus must load, decode, and process every matching series from its TSDB. This can eat gigabytes of memory and minutes of CPU, blocking other queries and potentially crashing Prometheus with out-of-memory errors.

Internally, Prometheus processes PromQL in stages: label matching loads the posting lists (inverted index entries) for each label matcher, then the storage engine fetches the actual sample data for the matched series, and the query engine evaluates the expression. The key optimization point is label matching -- narrow matchers like {service="payments-api",path=~"/api/v2/orders.*"} shrink the posting list intersection dramatically compared to {service=~".*"}. Recording rules evaluate expensive queries at a fixed interval and store the result as a new time series. Instead of every dashboard panel and alert rule independently computing histogram_quantile across all pods, a recording rule computes it once per evaluation interval and stores it as service:http_request_duration_seconds:p99, which later queries read as a simple series lookup.

At production scale, architects should build a recording rule hierarchy: L1 recording rules aggregate raw per-pod metrics into per-service metrics, L2 rules compute rates and percentiles from L1 results, and L3 rules compute SLO burn rates from L2 results. Alert rules should reference recording rules rather than raw metrics so that alerts and dashboards always see identical values. The for duration in alerting rules should be at least 2-3 evaluation intervals to absorb scrape gaps and brief spikes -- a 1-minute for duration with a 30-second evaluation interval means the condition must be true for 2-3 consecutive evaluations before it fires. Query timeout (--query.timeout flag) and maximum sample limits (--query.max-samples) protect Prometheus from runaway queries that could take down the entire monitoring stack.

The sneaky gotcha is that recording rules themselves can cause high cardinality if the aggregation does not actually reduce dimensions. A recording rule that preserves the pod label still produces one series per pod, which could be thousands. The rule should aggregate by service, environment, and cluster -- labels that shrink to tens or hundreds of series -- not by pod or instance. Another trap is rate() and histogram_quantile() behaving differently than expected when the recording rule interval does not align with the rate window. If the interval is 120 seconds but the rate window is 30s, some data points may be missed because the lookback window is shorter than the gap between evaluations.

Think of a phone system that automatically adds new employees to the directory when they join a department. Instead of the IT team manually programming each extension, the system watches the HR database and configures the phone switch whenever a new hire appears. The Prometheus Operator works the same way -- it watches Kubernetes for ServiceMonitor objects and automatically reconfigures Prometheus scrape targets without anyone editing prometheus.yml.

The Prometheus Operator is a Kubernetes controller that manages Prometheus instances, Alertmanager clusters, and their configurations as native Kubernetes resources. Instead of maintaining a prometheus.yml file with static scrape targets, teams create ServiceMonitor CRDs (Custom Resource Definitions) that declare which Kubernetes Services should be scraped, which port and path to use, and what relabeling to apply. The Operator watches these CRDs and generates the corresponding scrape configuration, injects it into the Prometheus StatefulSet, and triggers a configuration reload -- all without restarting Prometheus or requiring manual work.

Under the hood, the Operator reconciliation loop runs continuously. When a new ServiceMonitor is created in any namespace, the Operator checks whether it matches the serviceMonitorSelector on any Prometheus CR (Custom Resource). If it matches, the Operator regenerates the Prometheus configuration by combining all matching ServiceMonitors into scrape_configs, updates the configuration Secret mounted into the Prometheus Pod, and sends a SIGHUP to Prometheus to reload. For target discovery, the generated scrape config uses kubernetes_sd_config with role: endpoints, which discovers all Pods behind the matched Service and adds them as scrape targets. Relabeling rules from the ServiceMonitor are translated into relabel_configs that filter, rename, or transform labels before ingestion.

At production scale with 200+ microservices, architects need several patterns. First, establish a label convention: every Service that exposes metrics must carry a label like monitoring: enabled with annotations for the metrics port and path. Second, use namespace-scoped ServiceMonitors owned by application teams rather than one central ServiceMonitor that tries to match everything -- this spreads ownership and prevents one team's misconfigured ServiceMonitor from breaking another team's monitoring. Third, when a single Prometheus cannot handle the ingestion rate (typically above 500,000 samples/second), use Prometheus sharding via the shards field in the Prometheus CR, which runs multiple Prometheus instances that each scrape a deterministic subset of targets using hashmod relabeling. Fourth, use separate PodMonitors for short-lived workloads like Jobs and CronJobs that do not have Services.

The sneaky gotcha is the serviceMonitorSelector on the Prometheus CR. By default, the Operator configures Prometheus to only discover ServiceMonitors that match specific labels. If a team creates a ServiceMonitor in a new namespace but forgets to add the required label (like release: kube-prometheus-stack), Prometheus never scrapes their targets and they assume monitoring is broken. Another trap is that ServiceMonitors generate scrape configs with kubernetes_sd_config, which requires the Prometheus Pod's service account to have RBAC permissions to list and watch Services, Endpoints, and Pods across the namespaces being monitored. Missing RBAC silently results in zero targets discovered.

Think of a car's fuel gauge. A static threshold alert says 'warn at 25% fuel' — but that ignores whether you are on a highway burning fuel fast or parked with the engine off. Multi-window burn rate is like saying 'warn when fuel consumption over the last hour would empty the tank before you reach the next gas station, AND you are still burning fast right now.' This catches real problems while ignoring brief spikes.

SLO-based alerting starts with defining an error budget. If your SLO is 99.9% availability over 30 days, your error budget is 0.1% — about 43 minutes of downtime. The burn rate is how fast you are consuming this budget. A burn rate of 1x means you will exactly exhaust the budget by the end of the period. A burn rate of 14.4x means you will exhaust the 30-day budget in just 2 days.

Multi-window burn rate uses two windows to reduce false positives. The long window (typically 1 hour) detects sustained error rates that threaten the budget. The short window (typically 5 minutes) confirms the problem is still happening right now. Both conditions must be true for the alert to fire. This prevents alerting on brief spikes that self-resolve (short window would not fire) and on historical errors that have already been fixed (long window shows the past, short window confirms the present). Google's SRE book recommends multiple severity tiers: 14.4x burn rate over 1h/5m for critical (page), 6x over 6h/30m for warning (ticket).

At production scale, teams define recording rules that pre-compute error ratios for each SLI at multiple windows. The error ratio is calculated as rate(http_requests_total{status=~"5.."}[window]) / rate(http_requests_total[window]). Recording rules at 5m, 30m, 1h, and 6h windows avoid expensive queries at alert evaluation time. Grafana dashboards show the remaining error budget as a percentage, making it visual whether the team can ship features or must focus on reliability.

The non-obvious gotcha is that burn rate alerts assume a uniform error distribution, which rarely matches reality. A 5-minute outage that burns 10% of the monthly budget followed by 29 days of perfect operation is very different from a constant 0.1% error rate. Teams should complement burn rate alerts with absolute threshold alerts for catastrophic failures (error rate > 50% for 1 minute) that would cause immediate user impact regardless of the monthly budget.

Think of investigating a crime scene. Fingerprints (metrics) tell you something happened. Witness statements (logs) describe what happened. Security camera footage (traces) shows exactly who went where and when. Each clue is useful alone, but correlating them — the fingerprint matches the person on camera who matches the witness description — solves the case. Observability correlation works the same way.

OpenTelemetry is the CNCF standard for collecting and exporting telemetry data. A single OpenTelemetry SDK instrumented in your application generates metrics, logs, and traces with shared context. The key correlation mechanism is the trace ID — a unique identifier generated when a request enters the system. This trace ID is automatically injected into every log line, every metric exemplar, and every trace span generated during that request's lifecycle across all microservices.

The Grafana stack provides the visualization and correlation layer. Prometheus stores metrics (request rate, error rate, latency histograms). Loki stores logs (structured log lines with trace IDs as labels). Tempo stores traces (distributed spans showing the full request journey). Grafana connects to all three and provides cross-signal navigation: click an anomalous point on a latency chart, see the exemplar trace ID, jump to the full trace in Tempo, then click any span to see the logs for that service during that request. This trace-to-metric-to-log correlation turns a 30-minute investigation into a 3-minute one.

At production scale, teams must manage cardinality and cost. Metrics should use bounded label values (HTTP method, status code category) not unbounded ones (user ID, request ID). Logs should be structured JSON with consistent fields across services. Traces should use head-based or tail-based sampling to reduce storage — capturing 100% of traces is prohibitively expensive at high throughput, but capturing 100% of error traces and 1% of success traces provides good coverage.

The non-obvious gotcha is that correlation only works when all three signals share the same trace ID format and propagation context. If one service uses W3C TraceContext and another uses B3 propagation, the trace breaks. Teams must standardize on one propagation format across all services and ensure the OpenTelemetry SDK is configured consistently. Also, log-to-trace correlation requires the trace ID to be a searchable field in Loki — adding it as a label enables fast lookups but increases cardinality, while using structured metadata keeps cardinality low but requires full-text search.

Think of monitoring a restaurant. Rate is how many customers you serve per hour. Errors is how many meals are sent back. Duration is how long customers wait for their food. If you track these three numbers for every section of the restaurant (bar, dining room, takeout), you know exactly where problems are and how severe they are. The RED method applies this same thinking to microservices.

The RED method was popularized by Tom Wilkie (Grafana Labs) as a simplification of Google's Four Golden Signals. While the Golden Signals include Latency, Traffic, Errors, and Saturation, RED focuses on the three most actionable signals for request-driven services. Rate tells you the traffic volume and whether it matches expectations. Errors tell you if something is broken. Duration tells you if something is slow. Together, they answer the question every on-call engineer asks: is the service healthy, and if not, what is wrong?

Implementation in Grafana uses Prometheus queries against standard HTTP metrics. Rate is computed as sum(rate(http_requests_total[5m])) by service. Error rate is sum(rate(http_requests_total{status=~"5.."}[5m])) divided by total rate. Duration uses histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) for p99 latency. Each service gets a dashboard row with three panels: a rate graph (requests/sec over time), an error percentage graph (% 5xx over time), and a latency heatmap or multi-quantile graph (p50, p95, p99 over time).

At production scale, dashboard design matters. Use template variables so one dashboard works for all services by selecting a service from a dropdown. Add annotations for deployments so you can visually correlate metric changes with code releases. Use the repeat-by-variable feature to automatically create rows for each service. Set meaningful Y-axis ranges — auto-scaling can hide problems when error rates go from 0.01% to 0.1% because the scale adjusts to make it look flat.

The non-obvious gotcha is that RED works well for request-driven services but not for batch jobs, queues, or databases. For those, the USE method (Utilization, Saturation, Errors) is more appropriate. A payments API gets RED; the Kafka consumer gets USE (CPU utilization, queue depth saturation, processing errors). Many teams make the mistake of applying RED to everything, then wondering why their database dashboard does not show useful information. Use RED for services that handle requests and USE for infrastructure components.

Think of monitoring a factory floor. You check the building infrastructure (nodes), the assembly lines (pods), the power supply (control plane), and the shipping dock (networking). Each level has its own vital signs, and a problem at one level cascades to others. A power outage (control plane issue) affects all assembly lines (pods), but a single broken machine (crashed pod) only affects one product.

Kubernetes cluster health metrics come from three sources. Node metrics from node-exporter measure hardware and OS resources: CPU usage, memory available, disk space, network throughput. Kubernetes state metrics from kube-state-metrics report the desired vs actual state of Kubernetes objects: pod phase, deployment replicas, node conditions. Control plane metrics from the API server, scheduler, and etcd report the health of cluster management components.

The critical metrics organized by layer are: Nodes — node_cpu_seconds_total (CPU usage per core), node_memory_MemAvailable_bytes (available memory), node_filesystem_avail_bytes (disk space), kube_node_status_condition (Ready/NotReady). Pods — kube_pod_container_status_restarts_total (restart count), kube_pod_status_phase (Pending/Running/Failed), container_memory_working_set_bytes (actual memory usage vs limits). Control plane — apiserver_request_total (API server request rate and errors), etcd_server_leader_changes_seen_total (leader elections indicate instability), scheduler_pending_pods (pods waiting for a node).

At production scale, alert on symptoms that indicate user impact: nodes in NotReady for more than 5 minutes, pods with more than 5 restarts in 15 minutes, pending pods for more than 10 minutes (scheduling failure), API server error rate above 1%, and etcd leader changes more than 3 in an hour. Avoid alerting on individual pod CPU usage — it fluctuates naturally. Instead, alert on the ratio of usage to request, which indicates right-sizing problems.

The non-obvious gotcha is the difference between container_memory_usage_bytes and container_memory_working_set_bytes. The usage metric includes filesystem cache that the kernel can reclaim, so it often looks higher than the actual limit. The working_set metric reflects non-reclaimable memory and is what the OOM killer uses. Alerting on usage_bytes causes false alarms; alert on working_set_bytes approaching the memory limit instead.

Why Burn Rate > Threshold

Burn rate measures how fast you're consuming your error budget relative to the SLO period.

If your SLO is 99.9% over 30 days, your error budget is 0.1% × 30 days = 43.2 minutes of downtime. - Burn rate 1 = consuming budget at exactly the rate to exhaust it in 30 days - Burn rate 14.4 = consuming budget so fast it'll be gone in 2 days - Burn rate 6 = gone in 5 days

Multi-Window Burn Rate Alert

Use two windows per alert: a long window (catches sustained issues) and a short window (ensures issue is still happening):

| Severity | Burn Rate | Long Window | Short Window | Budget Consumed | |----------|-----------|-------------|--------------|------------------| | Page | 14.4x | 1 hour | 5 minutes | 2% in 1h | | Page | 6x | 6 hours | 30 minutes | 5% in 6h | | Ticket | 3x | 1 day | 2 hours | 10% in 1d | | Ticket | 1x | 3 days | 6 hours | 10% in 3d |

In the pull model, Prometheus's scrape config defines a set of targets (statically or via service discovery — Kubernetes, Consul, EC2, file_sd, etc.), and the server initiates an HTTP GET against each target's metrics endpoint on a fixed interval (commonly 15s-60s). The target's client library (e.g. client_golang, client_python) exposes current metric values in the Prometheus text exposition format or the newer OpenMetrics format.

This has several practical consequences interviewers probe for: (1) you can scrape from your laptop to debug a target directly with curl, since the target doesn't need to know Prometheus exists; (2) a target that's down simply fails to be scraped, which Prometheus surfaces as the synthetic up metric going to 0 — a push model needs a separate heartbeat/dead-man's-switch mechanism to detect this; (3) Prometheus, not the application, controls cardinality and scrape frequency, which caps the blast radius of a runaway metrics emitter; (4) short-lived batch jobs don't fit this model well, which is exactly why the Pushgateway exists as an explicit escape hatch — metrics are pushed to the gateway, which Prometheus then scrapes like any other target.

The tradeoff: pull doesn't work well through NAT/firewalls without extra plumbing (hence remote_write and Pushgateway), and very large fleets need service discovery to keep the target list current rather than static configs.