How does the Prometheus TSDB store time series data under the hood?

Imagine a library that organizes books by when they were written, not by author or title. Recent books sit on a desk for quick access (the in-memory head block). Every two hours, the librarian packs the desk books into sealed boxes (on-disk blocks), compresses them, and shelves them in the archive. A journal (the WAL, or write-ahead log) records every new book as it arrives so that if the librarian is interrupted, they can replay the journal and recover. This is essentially how Prometheus TSDB works.

Prometheus TSDB is a custom-built time series database optimized for the append-heavy, label-indexed workload of metrics. Each unique combination of metric name and labels forms a time series, identified by a numeric series ID. The database has two main parts: the head block (in-memory, mutable) containing the most recent 2+ hours of data, and persistent blocks (on-disk, immutable) containing older data. When a sample arrives, it goes to the head block and the WAL simultaneously. The WAL is a sequential log that ensures no data loss if Prometheus crashes -- on restart, it replays the WAL to rebuild the head block.

Each on-disk block is a directory with three key files. The index file maps label sets to series IDs and provides posting lists for fast label-based lookups (similar to a search engine's inverted index). Chunk files store the actual timestamp-value pairs compressed using Gorilla encoding -- double-delta encoding for timestamps and XOR encoding for values, achieving roughly 1.37 bytes per sample. A meta.json file records the block's time range and statistics. The head block uses the same chunk format but keeps everything in memory. When the head block covers more than 2 hours of data, it gets compacted into an on-disk block. Background compaction then merges smaller blocks into larger ones (up to 31 hours by default), reducing block count and improving query performance.

In production, understanding these internals helps with capacity planning. Storage size depends on three things: number of active time series (cardinality), scrape interval (samples per series per day), and retention period. A rough formula: disk_size = num_series x samples_per_day x bytes_per_sample x retention_days. With 1 million active series at a 15-second scrape interval and 15-day retention, expect about 120 GB of disk. The flags --storage.tsdb.retention.time and --storage.tsdb.retention.size let you cap storage. Remote write can offload long-term data to systems like Thanos or Mimir.

The biggest gotcha is high cardinality -- creating time series with labels like user IDs, request IDs, or IP addresses explodes the index size and can crash Prometheus. The index grows proportionally with unique label combinations. Another pitfall is ignoring WAL size. If Prometheus cannot compact fast enough (usually due to CPU or disk I/O saturation), the WAL grows without bound and can fill the disk. The metric prometheus_tsdb_wal_segment_current shows the current WAL segment number, and a rapidly increasing value means compaction is falling behind.