How does Prometheus remote write handle backpressure when the receiving backend is slow or down?

Think of a shipping dock that sends packages from a factory to a central warehouse. The factory keeps making boxes, workers load them into several truck lanes, and sometimes the warehouse slows down. If the lanes fill up, boxes pile up at the dock. If the warehouse stays closed for hours, the factory has to choose between halting intake, using more dock space, or eventually throwing away boxes it can no longer hold.

Prometheus remote write is that shipping dock for metrics. Prometheus first ingests samples locally through its normal scrape and WAL path. A remote write component then reads from the WAL, maps internal series IDs to their label sets, queues samples, and sends compressed HTTP requests to the configured remote endpoint. That endpoint might be Grafana Mimir, Thanos Receive, Cortex, VictoriaMetrics, or a managed cloud service. Remote write is not a magic way to backfill historical data; it is mainly a streaming replication path from the local ingestion flow.

Backpressure shows up when the remote endpoint is slow, returning errors, rate-limiting, or totally unreachable. Prometheus uses shards, which are parallel sending workers, to improve throughput. Each shard has an in-memory queue with a capacity limit and a maximum batch size. Failed requests get retried with exponential backoff. Prometheus can automatically adjust the shard count based on the incoming sample rate and how long sends are taking. But once queues fill up, reading from the WAL for that remote write target is blocked, and pending samples start piling up.

In production, the key metrics to watch are pending samples, failed samples, retried samples, send batch duration, current shard count, and queue capacity. Tuning usually starts with the receiver side: confirm it is healthy, not throttling, and not rejecting samples due to tenant limits or bad labels. Then tune Prometheus settings like max_samples_per_send, capacity, max_shards, and backoff values. Capacity should generally be several times the batch size, but setting it too high increases Prometheus memory usage. Write relabeling can drop expensive or unnecessary samples before they even leave Prometheus.

The gotcha is that cranking every knob up can make the outage worse. More shards can overwhelm a backend that is trying to recover. More queue capacity can cause Prometheus memory pressure, especially during high series churn because remote write caches series labels. Another gotcha involves the two-hour WAL window: if remote write stays blocked longer than the WAL can hold unsent data, samples get lost when the WAL is compacted. Senior engineers treat remote write tuning as end-to-end flow control, not just a matter of making the queue bigger.