How does Thanos enable long-term Prometheus storage and global queries across clusters?

Think of a library system across a city. Each branch library (Prometheus instance) keeps recent books on its shelves, but older books go to a central warehouse (object storage). When a researcher wants to search across all branches and the warehouse at the same time, a central catalog system (Thanos Querier) knows where every book is and fetches it from the right place. Thanos does exactly this for Prometheus metrics.

Thanos was built to fix two basic Prometheus limitations: local storage is not durable (if the node dies, metrics are lost), and a single Prometheus can only query its own data. In a multi-cluster environment with 15 Kubernetes clusters, each running its own Prometheus, there is no built-in way to query across all clusters or keep metrics beyond the local retention period (typically 15-30 days). Thanos adds a sidecar to each Prometheus that uploads completed TSDB blocks to object storage (S3, GCS, or Azure Blob), a Store Gateway that serves historical blocks from object storage, a Querier that merges results from sidecars and Store Gateways, and a Compactor that downsamples and compacts blocks for faster long-range queries.

Here is the detailed flow. Prometheus writes 2-hour TSDB blocks to local disk. The Thanos Sidecar watches the Prometheus data directory and uploads completed blocks to the object storage bucket. Each block contains a meta.json describing its time range, labels, and resolution. The Querier implements the Prometheus HTTP API and receives PromQL queries. It fans out to all connected StoreAPI endpoints -- sidecars for recent data and Store Gateways for historical data -- deduplicates overlapping series using external labels, and returns merged results. The Compactor runs as a singleton (meaning exactly one instance), downloading blocks from object storage, merging overlapping blocks, creating downsampled versions at 5-minute and 1-hour resolutions, and re-uploading the compacted blocks. This cuts storage costs and speeds up queries over long time ranges.

At production scale, the Store Gateway is the most resource-hungry component because it must cache block index headers in memory to answer queries quickly. A cluster with 500 million active time series and 1 year of retention may have hundreds of thousands of blocks, requiring Store Gateway instances with 32-64 GB of memory for index caching. The Compactor must keep up with block production -- if it falls behind, queries over historical data slow down because the Querier has to open many small blocks instead of a few large ones. Architects should watch thanos_compact_group_compactions_failures_total, thanos_store_bucket_cache_hits_total, thanos_query_store_api_duration_seconds, and overall object storage bucket size and cost.

The sneaky gotcha is that Thanos deduplication depends on consistent external labels. If two Prometheus instances scrape the same targets but have different or missing external labels, the Querier cannot deduplicate correctly and returns duplicate series that produce wrong aggregation results. Another trap is that the Compactor is a singleton -- running two Compactors against the same bucket causes data corruption because they fight over the same blocks. Teams using Thanos Compactor must guarantee exactly-one behavior, typically via a Kubernetes Deployment with replicas: 1 and a PodDisruptionBudget that prevents eviction during compaction.