How do you implement OpenTelemetry in a Kubernetes environment with the OpenTelemetry Operator?

Think of the OpenTelemetry Operator like a building superintendent who manages the heating system and installs smart thermometers in every apartment. Without the superintendent, every tenant (development team) would need to buy, install, and maintain their own thermometer (SDK instrumentation) and connect it to the building's heating system (Collector). With the superintendent, tenants just put a sign on their door saying 'please install a thermometer' (add an annotation), and the superintendent handles everything: installs the right thermometer model, wires it to the central system, and manages the heating infrastructure.

The OpenTelemetry Operator is installed via Helm and runs as a controller in the cluster. It watches for two custom resources: OpenTelemetryCollector and Instrumentation. The OpenTelemetryCollector CRD lets you declare Collector deployments in Kubernetes-native YAML instead of manually managing Deployments, DaemonSets, ConfigMaps, and Services. You specify the mode (deployment, daemonset, statefulset, or sidecar), the Collector configuration, resource limits, replicas, and autoscaling parameters. The Operator creates and manages all underlying Kubernetes objects, handles configuration updates with rolling restarts, and supports version upgrades.

The Instrumentation CRD is the most powerful feature. It defines auto-instrumentation configuration for a specific language: which OTLP endpoint to send to, which propagators to use (W3C Trace Context, B3), resource attributes to add, and sampler configuration. Once created, you enable instrumentation for any pod by adding an annotation like instrumentation.opentelemetry.io/inject-java: "true" to the pod spec. The Operator's mutating webhook intercepts pod creation and injects an init container that downloads the appropriate instrumentation agent (like the Java JAVAAGENT JAR), modifies environment variables to activate it, and mounts the agent into the application container. The payments-api service written in Java gets full distributed tracing, metric collection, and log correlation without changing a single line of application code.

For a production Kubernetes environment running the checkout-service, order-processing-service, user-auth-service, and inventory-sync, the typical setup involves three Operator-managed components. First, an OpenTelemetryCollector in daemonset mode for agent-level collection on every node, configured with the k8sattributes processor for Kubernetes metadata enrichment. Second, an OpenTelemetryCollector in deployment mode with 3 replicas for the gateway tier, configured with tail-based sampling and multi-backend export. Third, multiple Instrumentation resources, one per language (Java, Python, Node.js), each configured with the appropriate SDK settings. Teams onboard by adding a single annotation to their Deployment spec, and within minutes their service appears in the tracing backend.

The sidecar mode is worth special mention. When you set mode: sidecar on an OpenTelemetryCollector resource and annotate a pod with sidecar.opentelemetry.io/inject: "true", the Operator injects a Collector container directly into the application pod. This gives each pod its own dedicated Collector, which is useful for multi-tenant clusters where teams want isolated telemetry pipelines, or for services with extremely high telemetry volume that would overwhelm a shared DaemonSet agent. The trade-off is higher resource consumption since every pod runs an additional container. For the inventory-sync service that generates 100,000 spans per minute during peak sync operations, a sidecar Collector prevents it from starving other services of telemetry bandwidth on the node.

Operational considerations include RBAC configuration (the Operator needs permissions to mutate pods and create resources), certificate management for webhook TLS, and careful resource limit tuning on both Collectors and injected init containers. Upgrades should be tested in a staging cluster first, as Operator version changes can affect how instrumentation agents are injected.