How does Prometheus TSDB write data through the WAL, head block, and compaction, and what breaks when sizing is wrong?

Think of a busy warehouse that receives packages all day. New boxes first land in a fast intake area, while a clerk writes every arrival into a paper ledger in case the power fails. Every few hours, the intake area is sorted into sealed pallets and moved to long-term shelves. Prometheus TSDB works the same way: fresh samples are kept hot for fast queries, protected by a write-ahead log, and later packed into efficient on-disk blocks.

In Prometheus, TSDB stands for time series database. It is the local storage engine that keeps metric samples organized by metric name and labels. Prometheus is intentionally a single-node local database, not a clustered durable store. That design keeps ingestion and querying simple, but it puts the burden on operators to size local disk, memory, and retention correctly. The local TSDB is great for recent operational data, but high availability and long-term retention typically require replicas, remote write, Thanos, Mimir, or another system layered on top.

The write path starts when a scrape returns samples. Prometheus checks the labels, appends each sample to the head block in memory, and writes a record to the WAL so a restart can replay recent data. The active head block covers the most recent samples and periodically gets frozen into an immutable block, usually covering a two-hour window. Older blocks are merged together in the background through a process called compaction. Index files map label sets to chunk data, tombstones record deletions, and retention cleanup removes blocks that have fully expired.

In production, head data and WAL data are where the pressure peaks. Operators watch active series count, ingestion rate, WAL fsync failures, checkpoint duration, available disk, compaction failures, and query latency. Retention based on time alone can be misleading when a team suddenly adds a high-cardinality label like customer_id. Size-based retention should leave headroom because the WAL and head chunks still need space while background cleanup catches up. Prometheus expects local POSIX storage; network filesystems are a reliability risk for the TSDB.

The gotcha that catches people off guard is that retention cleanup is block-oriented and delayed, but the outage is often immediate. If the disk fills because the WAL is growing or the head block is huge, lowering retention from 30 days to 15 days may not save the node right away. Deleting series through the admin API writes tombstones first and does not instantly shrink chunk files. Removing WAL files can let Prometheus start again, but you lose recent samples. Experienced operators first stop the source of excess cardinality, then recover disk, then tune retention and remote storage to prevent it from happening again.