How does Prometheus federation work, and when should you use it versus Thanos or remote write?

Think of a news organization. Hierarchical federation is like regional bureaus sending headline summaries to the national desk -- the national editor gets the big picture without reading every local article. Cross-service federation is like the sports desk borrowing a specific stat from the finance desk's data feed. In both cases, only selected information flows upward or sideways, not everything.

Prometheus federation uses the /federate endpoint to expose a subset of metrics from one Prometheus instance so another Prometheus can scrape them. In hierarchical federation, a global Prometheus sits above multiple datacenter or cluster-level Prometheus instances. Each lower-level instance runs recording rules that pre-aggregate raw metrics into summary time series -- for example, computing the 99th percentile request latency per service every minute. The global Prometheus scrapes only these aggregated metrics from the /federate endpoint of each lower-level instance, giving it a fleet-wide view without ingesting the raw per-pod or per-instance metrics.

Under the hood, the /federate endpoint accepts match[] parameters that filter which metrics are exposed. The global Prometheus configures a scrape job with honor_labels: true to keep the original labels from the source instances, and metrics_path: /federate with params that specify the match expressions. Cross-service federation uses the same mechanism but horizontally: the payments team's Prometheus scrapes specific metrics like http_requests_total or circuit_breaker_state from the user-auth team's Prometheus to monitor a critical dependency. The match expression is narrow, pulling only the exact metrics needed rather than the entire dataset.

At production scale, hierarchical federation works well when the global Prometheus only needs aggregated views -- overall error rates, cluster-level resource utilization, SLO compliance percentages. It falls short when engineers need to drill into full-resolution metrics for incident investigation because the global instance only has pre-aggregated data. That is where Thanos or remote write to a central TSDB like Cortex, Mimir, or VictoriaMetrics becomes necessary. Remote write pushes all raw metrics from every Prometheus to a central store, enabling full-resolution global queries at the cost of more storage and ingestion infrastructure. Pick hierarchical federation for cost-effective fleet-wide dashboards, cross-service federation for targeted dependency monitoring with minimal coupling, and Thanos or remote write when incident responders need full-resolution cross-cluster queries.

The sneaky gotcha is that federation creates a scrape-interval dependency. The global Prometheus scrapes the /federate endpoint at its own interval (typically 60 seconds), but the source metrics were produced at the source's scrape interval (typically 15-30 seconds). This creates staleness windows where the global view lags behind reality. If the global scrape interval is longer than twice the source's recording rule evaluation interval, data points can be missed entirely. Another trap is that /federate is expensive for the source Prometheus -- serving thousands of metrics via /federate on every scrape adds CPU and memory pressure to the source. Teams should use recording rules to keep the number of series exposed via /federate small rather than using broad match expressions that pull raw metrics.