How do you configure the OpenTelemetry Collector pipeline (receivers, processors, exporters)?

Think of the OpenTelemetry Collector pipeline like a water treatment plant. Water comes in from multiple sources through intake pipes (receivers), passes through filtration and purification stages (processors), and is distributed to homes and businesses through output pipes (exporters). Each stage is independent and modular. You can add new intake sources without changing the treatment stages, or add new distribution targets without modifying the intake. The Collector acts as a vendor-agnostic telemetry processing hub.

Receivers are the entry points for telemetry data. The Collector supports multiple receiver types simultaneously. The OTLP receiver accepts data via the OpenTelemetry Protocol over gRPC (port 4317) and HTTP (port 4318), and is the primary receiver for applications instrumented with OpenTelemetry SDKs. The Jaeger receiver accepts Jaeger-native formats for legacy applications. The Prometheus receiver scrapes Prometheus-format metrics endpoints, allowing the Collector to replace or augment a Prometheus server. The Kafka receiver consumes telemetry from Kafka topics for high-throughput buffered ingestion. The hostmetrics receiver collects system-level metrics like CPU, memory, and disk from the host. Each receiver can be configured independently with its own ports, TLS settings, and protocol options. The payments-api might send traces via OTLP while the legacy order-processing-service sends Jaeger-format spans.

Processors sit between receivers and exporters and form the core transformation layer. The batch processor is almost always required in production because it groups individual spans or metrics into batches before export, dramatically reducing the number of outbound network calls. The memory_limiter processor prevents the Collector from consuming unbounded memory by applying backpressure when usage exceeds a threshold. The attributes processor adds, modifies, or deletes span attributes, which is useful for injecting environment metadata or stripping sensitive data like user email addresses. The filter processor drops entire spans or metrics matching specific criteria, such as health check endpoints that generate noise. The resource processor enriches all telemetry with resource attributes like deployment environment or cluster name. Processors execute in the order they are listed in the pipeline configuration, so ordering matters. The memory_limiter should always be first to prevent OOM before other processors consume memory.

Exporters send processed telemetry to one or more backends. The OTLP exporter sends data to any OTLP-compatible backend like Grafana Tempo, Jaeger, or another Collector in a multi-tier deployment. The Prometheus exporter exposes metrics in Prometheus format on an HTTP endpoint for scraping. The logging exporter writes telemetry to stdout, useful for debugging. The loadbalancing exporter distributes traces across multiple downstream Collectors by routing spans with the same trace ID to the same instance, essential for tail-based sampling. Multiple exporters can be configured in a single pipeline, allowing you to fan out traces to both Jaeger for real-time viewing and a data lake for long-term analysis.

In production, Collector deployment follows two common patterns. The Agent pattern deploys a Collector as a DaemonSet or sidecar on every node, receiving telemetry from local applications with minimal network latency. The Gateway pattern deploys a centralized Collector cluster that receives data from agents, performs heavy processing like tail-based sampling, and exports to backends. Many organizations use both in a two-tier architecture: lightweight agents on every node forward to a gateway cluster. The Collector configuration is typically managed via ConfigMaps in Kubernetes, and the Collector supports hot-reloading when the ConfigMap changes. A common pitfall is forgetting to configure the batch processor, which results in one HTTP request per span and overwhelms both the Collector and the backend. Another gotcha is not setting memory_limiter, causing the Collector to OOM under traffic spikes from services like checkout-service during flash sales.