How does OpenTelemetry sampling work and what strategies are available (head-based vs tail-based)?

Think of sampling like a quality control inspector at a factory. Head-based sampling is like randomly pulling every 10th item off the conveyor belt for inspection before it is even assembled. Tail-based sampling is like letting all items go through the entire assembly line, then inspecting only the finished products that have defects or took too long to build. Both approaches reduce the volume you need to examine, but tail-based gives you much smarter filtering at the cost of temporarily storing everything.

Head-based sampling is the default and simplest approach in OpenTelemetry. The sampling decision is made at the root span of a trace, before any work is done. The SDK provides several built-in samplers: AlwaysOn (sample everything), AlwaysOff (sample nothing), TraceIdRatioBased (sample a percentage based on the trace ID hash), and ParentBased (respect the sampling decision of the parent span). The ParentBased sampler is particularly important in microservice architectures because it ensures that if the payments-api starts a trace and decides to sample it, all downstream services like order-processing-service and inventory-sync will also sample their spans for that trace. Without ParentBased, you end up with broken traces where some services recorded spans and others did not. The trace ID hash ensures deterministic sampling, meaning the same trace ID always produces the same sampling decision across all services.

Tail-based sampling requires the OpenTelemetry Collector because the decision cannot be made at the SDK level. The Collector receives all spans from all services, assembles complete traces in memory, and then applies policies to decide which traces to keep. Common policies include keeping all traces that contain an error status, keeping traces where the total duration exceeds a threshold like 2 seconds, keeping traces that touch specific services like checkout-service, or applying a probabilistic ratio to the remaining traces. The tail_sampling processor in the Collector supports combining multiple policies with AND and OR logic. For example, you might keep 100% of error traces, 100% of traces slower than 5 seconds, and 1% of everything else.

In production, the choice between head-based and tail-based sampling has significant infrastructure implications. Head-based sampling is lightweight because services discard unsampled spans immediately, reducing CPU, memory, and network overhead at the source. However, it means you lose visibility into rare errors that happen to fall in the unsampled portion. Tail-based sampling captures every interesting trace but requires the Collector to buffer all spans in memory until the trace is complete, which demands significant RAM. A typical tail-sampling Collector handling the traffic from services like payments-api, user-auth-service, and order-processing-service might need 4-8 GB of RAM. The wait_duration parameter controls how long the Collector waits for late-arriving spans before making a decision, typically 30-60 seconds. If a span arrives after the decision window, it is either dropped or sampled probabilistically.

A critical production gotcha is the interaction between head-based and tail-based sampling. If you configure a 10% head-based sampler in your SDKs and also run tail-based sampling in the Collector, you are sampling from an already-sampled stream, which means your tail-based policies only see 10% of traces. The correct setup for tail-based sampling is to use AlwaysOn in the SDK and let the Collector make all sampling decisions. Another common mistake is running a single tail-sampling Collector instance, which creates a single point of failure and limits throughput. In production, you need a load-balancing Collector tier in front of tail-sampling Collectors, using the loadbalancing exporter to route all spans from the same trace to the same tail-sampling instance. Without this, spans from the same trace land on different Collectors, and neither has the complete picture to make an accurate decision.