How do you design a production-grade OpenTelemetry Collector deployment (gateway vs agent mode, scaling)?

Think of the OpenTelemetry Collector like a postal system. Agent-mode Collectors are the local mailboxes on every street corner (every Kubernetes node) where residents (applications) drop off their letters (telemetry data). Gateway-mode Collectors are the central post offices that sort mail by destination, apply business rules (like filtering junk mail), and route to the final destinations (backends like Jaeger, Prometheus, or a cloud vendor). You would never have every house send mail directly to the national sorting center, and you would never have just one mailbox for the entire city.

The agent-mode Collector runs as a Kubernetes DaemonSet, meaning one instance per node. Applications on that node send telemetry (traces, metrics, logs) to localhost or the node's IP, which is fast and avoids cross-network hops. The agent handles lightweight tasks: receiving data via OTLP, adding resource attributes like node name and pod name using the k8sattributes processor, performing basic filtering to drop health-check spans, and batching data before forwarding. Because it runs on every node, it must be resource-efficient, typically limited to 200-500MB RAM and 0.5 CPU. If the agent crashes, only that node's telemetry is affected, not the entire pipeline.

The gateway-mode Collector runs as a Kubernetes Deployment with multiple replicas behind a ClusterIP or LoadBalancer service. This tier handles heavy processing: tail-based sampling decisions across the entire fleet (which requires seeing all spans of a trace in one place), complex attribute transformations, routing to multiple backends based on signal type, and rate limiting. Gateway replicas can be autoscaled with HPA based on CPU, memory, or custom metrics like the queue size of the exporter. A typical production setup uses 3-5 gateway replicas, each with 2-4GB RAM.

Scaling is the critical design concern. Agent-mode Collectors scale automatically with nodes (DaemonSet), so you only worry about per-node resource limits. Gateway-mode Collectors need careful capacity planning. The key bottleneck is usually the exporter queue: if the backend (say Tempo or Jaeger) is slow, the queue fills up, backpressure propagates to agents, and telemetry is dropped. You mitigate this with the sending_queue configuration (persistent queue backed by file storage so data survives restarts), retry_on_failure, and proper timeout settings. Monitoring the Collector itself is essential: expose its internal metrics (otelcol_exporter_queue_size, otelcol_receiver_refused_spans) to Prometheus and alert on queue saturation.

A common production pattern at companies like the ones running payments-api, order-processing-service, and checkout-service is to have separate gateway pools per signal type: one gateway Deployment for traces (which need tail-based sampling), another for metrics (which need different batching and aggregation), and a third for logs (which have different volume characteristics). This prevents a log spike from starving trace processing. Each pool has its own HPA policy tuned to its specific workload profile.