What is Prometheus federation, and when should you use it instead of remote_write to a system like Thanos or Cortex?
Quick Answer
Federation lets one Prometheus server scrape a filtered subset of another Prometheus server's already-computed metrics via the /federate endpoint — it's designed for pulling aggregated or a small number of important series up to a global view, not for shipping all raw data. remote_write to Thanos/Cortex/Mimir is the right choice when you need full-resolution long-term storage, global querying across all raw series, and high availability — federation doesn't solve long-term storage or full-fidelity global queries at all.
Detailed Answer
Federation works by having a "global" Prometheus scrape the /federate HTTP endpoint of "leaf" Prometheus servers, using a match[] parameter to select only specific metrics (typically pre-aggregated recording rules, not raw high-cardinality series). This keeps the federated dataset small and is genuinely useful for hierarchical rollups — e.g., each region's Prometheus computes SLO recording rules locally, and a global Prometheus federates just those recording-rule outputs for a company-wide dashboard.
What federation does NOT give you: it doesn't provide long-term retention beyond what each leaf Prometheus already stores, it's not designed to federate ALL raw metrics (doing so just recreates the cardinality/storage problem one level up plus scrape overhead), and it doesn't provide deduplication for HA Prometheus pairs.
remote_write-based systems (Thanos, Cortex, Grafana Mimir) solve a different problem: durable, long-term, full-resolution storage with a single global query interface across every raw series from every Prometheus instance, plus (in Thanos/Cortex) deduplication when you run HA Prometheus pairs. Thanos in particular is popular because it's additive — a sidecar next to each existing Prometheus uploads TSDB blocks to object storage, and a global Querier serves PromQL across all of them plus long-term data in object storage, without replacing Prometheus's ingestion path. Cortex/Mimir instead replace local storage with prometheus writing directly to a distributed ingestion tier.
Rule of thumb interviewers listen for: federation = selective, aggregated, hierarchical pull; remote_write+Thanos/Cortex/Mimir = complete, long-term, centralized push.
Interview Tip
If asked to design multi-cluster monitoring, don't default to federation for everything — say you'd federate only SLO/recording-rule outputs and use remote_write/Thanos for full-fidelity long-term storage.