How do you implement tail-based sampling with the OpenTelemetry Collector?

Think of tail-based sampling like a film editor reviewing complete movie takes. Head-based sampling is like randomly deciding before filming whether to keep the take or not, meaning you might discard a take that turns out to contain a brilliant performance. Tail-based sampling waits until the entire take is filmed, watches it through, and then decides whether to keep it based on its content. You keep all the takes with great performances (errors, high latency) and discard a percentage of the routine takes (successful, fast requests).

Head-based sampling, which is the default in most SDK configurations, makes the sampling decision at the root span before any processing occurs. You set a probability (say 10%) and randomly keep 1 in 10 traces. The problem is that errors and slow requests happen in less than 1% of traffic, so a 10% head-based sample might miss critical traces entirely. You could set sampling to 100% to catch everything, but that generates enormous volume and cost. Tail-based sampling solves this by deferring the decision until all spans of a trace have been collected, then applying intelligent policies.

The tail_sampling processor in the OpenTelemetry Collector works by buffering spans in memory for a configurable decision_wait period (typically 10-30 seconds). During this window, all spans arriving with the same trace ID are grouped together. After the wait period expires, the processor evaluates the complete trace against a set of policies: did any span have an ERROR status code? Did the total trace duration exceed a latency threshold? Does the trace contain spans from a specific service? Based on these policy evaluations, the trace is either kept or dropped. Composite policies allow you to combine rules, for example keeping 100% of error traces, 100% of traces over 2 seconds, 50% of traces from the payments-api service, and only 5% of everything else.

The critical architectural requirement is that all spans belonging to a single trace must arrive at the same Collector instance. If span A of trace-123 goes to Gateway-1 and span B goes to Gateway-2, neither Collector sees the complete trace and both make incorrect sampling decisions. This is solved using the loadbalancing exporter in the agent-mode Collector, which implements consistent hashing on the trace ID to route all spans of the same trace to the same gateway. The load balancing exporter discovers gateway instances via DNS or Kubernetes service endpoints and distributes traces deterministically.

Memory management is the biggest operational challenge. The Collector must buffer all spans during the decision_wait period. If your system generates 50,000 spans per second and the decision wait is 30 seconds, the Collector buffers 1.5 million spans simultaneously. Each span consumes roughly 1-3KB of memory, so you need 1.5-4.5GB of RAM just for the sampling buffer. You must tune decision_wait (shorter wait means less memory but risk of incomplete traces), num_traces (maximum concurrent traces in the buffer), and expected_new_traces_per_sec. Monitoring otelcol_processor_tail_sampling_count_traces_sampled versus otelcol_processor_tail_sampling_count_traces_dropped is essential to verify your policies are working correctly.