OpenTelemetry

Think of OpenTelemetry as a universal adapter for observability. Before it existed, every monitoring vendor had its own proprietary agent and data format, so switching from one tool to another meant re-instrumenting your entire codebase. OpenTelemetry removes that lock-in by providing a single, vendor-neutral way to generate and export telemetry data.

OpenTelemetry (often abbreviated as OTel) is a Cloud Native Computing Foundation (CNCF) incubating project that provides a comprehensive observability framework. It defines a standard for how telemetry data (traces, metrics, and logs) is generated, collected, processed, and exported. The project delivers language-specific SDKs for Go, Java, Python, JavaScript, .NET, Ruby, and others, along with the OpenTelemetry Collector, a standalone binary that receives, processes, and forwards telemetry data to any supported backend. In a production environment running services like payments-api, user-auth-service, and order-processing-service, OpenTelemetry instruments each service identically regardless of whether you send data to Jaeger, Prometheus, Datadog, or Grafana Cloud.

The history behind OpenTelemetry is important context for interviews. Two earlier projects tackled distributed tracing independently: OpenTracing (a vendor-neutral tracing API) and OpenCensus (a Google-led project covering tracing and metrics). Both gained significant adoption, but having two competing standards fragmented the ecosystem. Library authors had to choose which one to support, and users sometimes needed both in the same application. In 2019, the two projects merged to form OpenTelemetry, combining the best ideas from each. OpenTracing contributed its flexible span-based tracing model, while OpenCensus brought metrics support and the Collector architecture. This merger gave the community a single project to rally around.

In production environments, OpenTelemetry solves a critical problem for platform teams managing microservices. Consider a company running checkout-service, inventory-sync, and payments-api across Kubernetes clusters. Without a standard, each team might instrument differently: one uses the Datadog agent, another uses Prometheus client libraries, and a third uses custom logging. OpenTelemetry unifies this by providing a single instrumentation layer that every service adopts. The data can then be routed to any combination of backends through the Collector, meaning the observability backend becomes a deployment decision rather than a code decision.

A key architectural principle is the separation of instrumentation from export. Your application code uses OpenTelemetry APIs to create spans, record metrics, and emit logs. The SDK handles sampling, batching, and exporting. If you later switch from Jaeger to Tempo for trace storage, you change a configuration file in the Collector rather than modifying application code. This decoupling is why platform teams at companies running services like user-auth-service and order-processing-service adopt OpenTelemetry as their internal standard, even when individual teams might prefer different analysis tools.

Imagine diagnosing a car problem. Traces are like the GPS route showing where the car went and how long each leg took. Metrics are the dashboard gauges showing speed, RPM, and fuel level over time. Logs are the mechanic's notes recording specific events like 'engine light turned on at 45,000 miles.' You need all three together to understand what happened and why.

Traces in OpenTelemetry represent the end-to-end journey of a request as it flows through a distributed system. When a user clicks 'Place Order' on checkout-service, that request might flow to order-processing-service, then to inventory-sync to check stock, then to payments-api to charge the card. A trace captures this entire journey as a tree of spans, where each span represents a unit of work in a specific service. The trace gives you latency breakdowns showing that order-processing-service took 50ms, inventory-sync took 120ms, and payments-api took 300ms. Without traces, debugging a slow request across 15 microservices becomes a guessing game of searching through logs on each service independently.

Metrics are numerical values that measure system behavior over time. OpenTelemetry supports three primary metric instruments: counters (monotonically increasing values like total_requests or total_errors), gauges (values that go up and down like active_connections or queue_depth), and histograms (distributions of values like request_duration or response_size). Metrics are lightweight and designed for aggregation. Instead of recording every single request, you record summaries: the average latency, the 99th percentile response time, the error rate per minute. In a production environment, metrics from payments-api might show that the p99 latency increased from 200ms to 800ms over the last hour, which triggers an alert. Metrics tell you that something is wrong and help you identify the affected service.

Logs are timestamped text records of discrete events. A log entry might say '2024-03-15T10:23:45Z ERROR payments-api: Failed to charge card ending in 4242, timeout after 30s.' OpenTelemetry's logging support correlates logs with traces by embedding trace IDs and span IDs into log entries. This correlation is the key differentiator: when user-auth-service logs an authentication failure, you can click the trace ID and see the full request path that led to that failure. Without correlation, logs from checkout-service, order-processing-service, and payments-api exist in separate silos, and connecting them requires manual timestamp matching and guesswork.

The power of OpenTelemetry comes from unifying all three pillars under a single framework with shared context. A trace ID generated when a request enters checkout-service propagates through every downstream service. Metrics recorded during that request carry the same service metadata. Logs emitted during processing embed the trace ID. This means you can start with a metric alert (p99 latency spike on inventory-sync), drill into traces to find slow requests, and then examine the logs for those specific traces to understand the root cause. This workflow, moving seamlessly from metrics to traces to logs, is what observability practitioners call the 'three pillars' working together.

Think of a span as a single entry in a detailed travel itinerary. If a trace is the complete trip from New York to London, then each span is a leg: taxi to JFK airport, check-in at the counter, flight to Heathrow, customs processing. Each leg has a start time, end time, and notes about what happened. Some legs contain sub-legs, like the flight leg containing takeoff, cruising, and landing as child spans.

A span is the fundamental building block of distributed tracing in OpenTelemetry. Every span has a unique span ID and belongs to a trace identified by a trace ID. When checkout-service receives a user request, it creates a root span. When checkout-service calls order-processing-service, a new child span is created with the checkout-service span as its parent. This parent-child relationship forms a tree structure that visualizes the entire request flow. The span's start and end timestamps tell you exactly how long each operation took, making it straightforward to identify which service or operation is the bottleneck.

Each span carries several categories of information. The span name identifies the operation, like 'POST /api/orders' or 'SELECT orders WHERE user_id = ?' for a database query. Attributes are key-value pairs that add context: http.method=POST, http.status_code=200, db.system=postgresql, or custom attributes like order.total=149.99. The span kind indicates the role: CLIENT for outgoing calls, SERVER for incoming requests, PRODUCER for async message sends, CONSUMER for message processing, and INTERNAL for in-process operations. The status field indicates whether the operation succeeded (OK), encountered an error (ERROR), or is unset. When an error occurs, the status description provides a human-readable explanation.

Spans also support events and links. Events are timestamped annotations within a span, useful for recording notable moments without creating a child span. For example, a span in inventory-sync might record an event 'cache miss for SKU-45678' at timestamp T1 and 'fetched from database' at timestamp T2. Links connect spans across different traces, which is valuable for batch processing: a span in order-processing-service that processes 50 orders might link to the 50 individual order creation spans from checkout-service. Events and links are often overlooked by beginners but are essential for modeling real-world distributed systems.

In production, span attributes are critical for debugging. When payments-api returns a 500 error, the span attributes tell you the payment method (credit_card), the provider (stripe), the customer ID, and the error message. Platform teams typically define attribute naming conventions following OpenTelemetry semantic conventions, which standardize attribute names across services. For example, all HTTP spans use http.request.method instead of each team inventing their own attribute name. This standardization enables consistent querying across services: 'show me all spans where http.response.status_code >= 500' works across checkout-service, payments-api, and user-auth-service because they all use the same attribute names.

Think of the OpenTelemetry Collector as a mail sorting facility. Letters (telemetry data) arrive from thousands of homes (services), get sorted, filtered, and routed to the correct destinations (backends). The sorting facility can handle mail from any post office (receiver) and deliver to any carrier (exporter), and you can add processing steps like removing junk mail (filtering) or combining small packages into larger shipments (batching).

The OpenTelemetry Collector is a standalone binary that forms the backbone of most production OpenTelemetry deployments. It is built around three core components: receivers, processors, and exporters, connected through pipelines. Receivers accept telemetry data in various formats. The OTLP receiver handles OpenTelemetry's native protocol, but the Collector also supports Jaeger, Zipkin, Prometheus, and other formats. This means you can migrate from Jaeger to OpenTelemetry incrementally: existing services send Jaeger-format spans to the Collector, which converts and exports them alongside native OTLP data. In a microservices environment with payments-api, checkout-service, and user-auth-service, every service sends its telemetry to the Collector rather than directly to the backend.

Processors sit between receivers and exporters and transform telemetry data in flight. The batch processor groups telemetry into larger payloads to reduce network overhead, which is essential in high-throughput environments. The memory_limiter processor prevents the Collector from consuming too much memory during traffic spikes. The filter processor drops unwanted telemetry, like health check spans that add noise without value. The attributes processor can add, modify, or delete attributes, such as injecting the Kubernetes namespace or removing sensitive data like user email addresses before export. In production at scale, processors are where platform teams implement their telemetry governance policies.

Exporters send processed telemetry to one or more backends. The OTLP exporter sends data to any OTLP-compatible backend like Grafana Tempo, Jaeger, or Datadog. The Prometheus exporter exposes metrics in Prometheus format for scraping. The logging exporter prints telemetry to stdout for debugging. A single Collector instance can export the same data to multiple backends simultaneously: traces to Jaeger for developers, metrics to Prometheus for SREs, and everything to an S3 bucket for long-term storage. This fan-out capability is one of the strongest reasons to use the Collector rather than exporting directly from applications.

The Collector can be deployed in two patterns: as an agent (sidecar or DaemonSet alongside applications) and as a gateway (centralized cluster receiving from multiple agents). In Kubernetes, a common pattern deploys the Collector as a DaemonSet where each node runs an agent that receives telemetry from pods on that node. The agents forward data to a gateway Collector deployment that handles heavy processing, sampling, and export. This two-tier architecture reduces the blast radius of Collector failures and keeps resource-intensive processing off application nodes. For a team running order-processing-service and inventory-sync across a 50-node Kubernetes cluster, the agent handles lightweight batching on each node while the gateway handles tail-based sampling and export to Grafana Cloud.

Think of automatic instrumentation like a dashcam that records everything your car does without you pressing any buttons. Manual instrumentation is like a passenger taking notes about specific landmarks and events along the route. The dashcam captures the broad picture, but the passenger's notes add context and meaning that a camera cannot capture on its own.

Automatic instrumentation (also called auto-instrumentation or zero-code instrumentation) works by hooking into popular libraries and frameworks to capture telemetry without modifying application code. In Python, the opentelemetry-instrument command wraps your application and automatically instruments libraries like Flask, Django, requests, SQLAlchemy, and psycopg2. In Java, a Java agent JAR is attached to the JVM at startup, intercepting method calls in frameworks like Spring Boot, gRPC, and JDBC. In Node.js, the @opentelemetry/auto-instrumentations-node package registers instrumentation hooks before your application loads its dependencies. The result is that a payments-api built with Flask and PostgreSQL immediately gets spans for every HTTP request and every database query without a single line of instrumentation code.

Manual instrumentation gives developers explicit control over what telemetry is generated. Using the OpenTelemetry API, a developer in order-processing-service can create a span named 'validate_inventory' that wraps the specific business logic for checking stock levels against inventory-sync. They can add custom attributes like order.item_count=5 and order.total=249.99, record events like 'backorder_triggered' when stock is insufficient, and set the span status to ERROR with a descriptive message when validation fails. None of this business-specific context can be captured by automatic instrumentation because the auto-instrumenter only sees HTTP calls and database queries, not the business meaning behind them.

In production, the standard approach is to layer both techniques. Automatic instrumentation provides the baseline: every HTTP request to checkout-service generates a span, every database query in user-auth-service is traced, and every outgoing HTTP call to payments-api creates a client span. Manual instrumentation then enriches this baseline with business context. A developer adds a custom span around the fraud detection logic in payments-api, sets attributes for the risk score and decision, and records events for each fraud rule evaluated. The auto-generated spans provide the structural skeleton of the trace, while manual spans fill in the business logic details that matter for debugging.

There are important tradeoffs to understand. Automatic instrumentation requires minimal effort but produces generic telemetry. Every Flask endpoint gets a span named 'POST /api/payments,' but without manual enrichment, you cannot tell whether the payment was for a subscription renewal or a one-time purchase. Manual instrumentation captures exactly what you need but requires developer discipline and ongoing maintenance. If a team adds a new payment provider to payments-api but forgets to add manual spans, that code path becomes a blind spot. Platform teams address this by establishing instrumentation guidelines and reviewing spans as part of the code review process, treating observability as a first-class concern alongside testing.

Think of exporters as shipping carriers for your telemetry data. Your application packages the data (spans, metrics, logs), and the exporter is the carrier (FedEx, UPS, DHL) that delivers it to the destination. Different destinations accept different carriers, but OTLP is like a universal carrier that most modern destinations accept. You can even ship to multiple destinations simultaneously using different carriers for the same package.

Exporters in OpenTelemetry are the final stage of the telemetry pipeline, responsible for serializing and transmitting data to external systems. They exist in two places: within the application SDK and within the OpenTelemetry Collector. SDK exporters are configured in your application code and send data directly to a backend or to a Collector. Collector exporters are configured in the Collector's YAML file and send processed data to one or more backends. In a production setup with payments-api, checkout-service, and order-processing-service, each application typically uses the OTLP exporter to send data to the Collector, and the Collector uses backend-specific exporters to forward data to its final destinations.

The OTLP (OpenTelemetry Protocol) exporter is the recommended default. OTLP is OpenTelemetry's native protocol, supporting traces, metrics, and logs over both gRPC (port 4317) and HTTP/protobuf (port 4318). All major observability backends now support OTLP natively: Grafana Tempo, Jaeger, Datadog, New Relic, Honeycomb, Splunk, and Elastic APM. Using OTLP means you are sending data in OpenTelemetry's own format without any conversion, which preserves all metadata and avoids lossy translations. When inventory-sync sends a span with 20 attributes and 5 events via OTLP, every attribute and event arrives intact at the backend.

Beyond OTLP, several protocol-specific exporters exist for backward compatibility and specific use cases. The Jaeger exporter sends traces in Jaeger's Thrift or gRPC format, useful when migrating from a Jaeger-native setup. The Zipkin exporter sends traces in Zipkin's JSON format. The Prometheus exporter is unique because it does not push data; instead, it exposes a metrics endpoint that Prometheus scrapes at regular intervals. This pull-based model matches how Prometheus fundamentally works. For cloud vendors, there are exporters for AWS X-Ray, Google Cloud Trace, Azure Monitor, Datadog, and New Relic. These vendor exporters handle the format translation and authentication required by each platform.

In production, the choice of exporter is usually made at the Collector level, not in application code. Applications export to the Collector using OTLP, and the Collector handles the fan-out. This pattern is powerful because changing your observability backend requires updating only the Collector configuration, not redeploying every service. Consider a team running user-auth-service and checkout-service that decides to switch from self-hosted Jaeger to Grafana Cloud. With the Collector in place, they change the exporter in the Collector config from Jaeger to OTLP pointing at Grafana Cloud's endpoint, redeploy the Collector, and every service is migrated without touching application code. During migration, they can even export to both backends simultaneously by adding two exporters to the same pipeline, running old and new systems in parallel until validation is complete.

Think of context propagation like a relay race baton. When the first runner (checkout-service) starts the race (trace), they carry a baton (trace context) with the race ID written on it. When they hand off to the second runner (order-processing-service), the baton transfers, and the second runner continues the same race. Without the baton handoff, each runner would think they are running a separate race, and you would never know they were part of the same event.

Context propagation is the foundation that makes distributed tracing work across service boundaries. When checkout-service receives a user request, the OpenTelemetry SDK generates a trace ID (a 128-bit identifier like 4bf92f3577b34da6a3ce929d0e0e4736) and creates a root span. When checkout-service makes an HTTP call to order-processing-service, the SDK's propagator injects the trace ID and current span ID into the outgoing HTTP headers. The most common format is W3C Trace Context, which uses two headers: traceparent (containing the trace ID, parent span ID, and trace flags) and tracestate (carrying vendor-specific data). When order-processing-service receives the request, its SDK extracts the trace context from the headers and creates a new span with the propagated trace ID and the incoming span ID as the parent.

OpenTelemetry supports multiple propagation formats through its propagator API. The W3C Trace Context format is the default and the recommended standard, defined in a W3C specification. The B3 format (used by Zipkin) is supported for backward compatibility with systems already using B3 headers. The Jaeger propagation format uses the uber-trace-id header. You can configure the SDK to use multiple propagators simultaneously, which is essential during migrations: if payments-api uses W3C format but a legacy inventory-sync still sends B3 headers, the Collector or SDK can understand both. The propagation format is typically set globally for all services in an organization to ensure consistency.

Context propagation is not limited to HTTP. OpenTelemetry propagates context through gRPC metadata, message queue headers (Kafka record headers, RabbitMQ message properties), and even custom transports. When order-processing-service publishes a message to a Kafka topic, the producer instrumentation injects the trace context into the Kafka record headers. When inventory-sync consumes that message, the consumer instrumentation extracts the context and creates a span linked to the original trace. This means the entire flow from user click through synchronous HTTP calls through asynchronous message processing appears as a single connected trace.

In production, broken context propagation is one of the most common reasons traces appear fragmented. If a proxy, API gateway, or load balancer strips custom headers, the trace context is lost and downstream services start new traces instead of continuing the existing one. Platform teams at companies running services like user-auth-service and checkout-service debug this by checking whether the traceparent header survives through every hop. Another common issue is mixing propagation formats: if checkout-service injects W3C headers but payments-api only extracts B3 format, the trace breaks at that boundary. Setting a consistent propagation format across all services and ensuring that infrastructure components (Nginx, Envoy, AWS ALB) forward trace headers are essential operational steps.

Setting up OpenTelemetry is like installing a security camera system. You choose the cameras (SDK and instrumentation libraries), connect them to a recording station (the Collector), and configure where the footage is stored (the backend). The cameras start recording automatically once installed, but you can also add extra cameras (manual instrumentation) in areas where you need more detailed coverage.

For a Python application like payments-api built with Flask and PostgreSQL, the setup involves three steps. First, install the core packages: opentelemetry-api and opentelemetry-sdk provide the tracing and metrics foundation, opentelemetry-exporter-otlp provides the exporter to send data to the Collector, and opentelemetry-instrumentation packages provide auto-instrumentation for specific libraries. Second, run opentelemetry-bootstrap -a install, which scans your installed Python packages and automatically installs the matching instrumentation libraries (e.g., it detects Flask and installs opentelemetry-instrumentation-flask). Third, wrap your application startup command with opentelemetry-instrument, which initializes the SDK, activates all installed instrumentations, and runs your app. You configure the service name and exporter endpoint through environment variables like OTEL_SERVICE_NAME and OTEL_EXPORTER_OTLP_ENDPOINT.

For a Node.js application like checkout-service built with Express and MongoDB, the setup is similar but uses a different mechanism. Install @opentelemetry/api, @opentelemetry/sdk-node, @opentelemetry/auto-instrumentations-node, and @opentelemetry/exporter-trace-otlp-http. The auto-instrumentations-node package is a meta-package that bundles instrumentations for Express, HTTP, MongoDB, Redis, gRPC, and many other libraries. To activate auto-instrumentation without code changes, set the NODE_OPTIONS environment variable to --require @opentelemetry/auto-instrumentations-node/register. This loads the instrumentation before your application code, ensuring that library imports are intercepted and wrapped with tracing logic. Alternatively, you can create a tracing.js file that programmatically configures the SDK with custom settings like sampling rate and resource attributes, then require it before your app starts.

In production, environment variables are the preferred configuration method because they keep instrumentation settings out of application code. The key variables are OTEL_SERVICE_NAME (identifies the service in traces), OTEL_EXPORTER_OTLP_ENDPOINT (the Collector address), OTEL_TRACES_SAMPLER (controls sampling strategy), and OTEL_RESOURCE_ATTRIBUTES (adds metadata like deployment.environment=production). For Kubernetes deployments of order-processing-service or inventory-sync, these variables are set in the Deployment manifest or injected via ConfigMaps. The OpenTelemetry Operator for Kubernetes can even inject auto-instrumentation automatically by adding an annotation to the pod spec, eliminating the need to modify Dockerfiles or deployment scripts.

Common setup pitfalls include forgetting to install instrumentation libraries for specific frameworks (Flask spans will not appear without opentelemetry-instrumentation-flask), pointing the exporter to the wrong Collector port (gRPC uses 4317, HTTP uses 4318), and not running opentelemetry-bootstrap after adding new dependencies. Another frequent mistake is testing instrumentation without a running Collector: the SDK will silently drop telemetry if the exporter endpoint is unreachable, making it look like instrumentation is not working when the issue is actually a networking problem between user-auth-service and the Collector pod.

Think of sampling like a quality control inspector at a factory. Head-based sampling is like randomly pulling every 10th item off the conveyor belt for inspection before it is even assembled. Tail-based sampling is like letting all items go through the entire assembly line, then inspecting only the finished products that have defects or took too long to build. Both approaches reduce the volume you need to examine, but tail-based gives you much smarter filtering at the cost of temporarily storing everything.

Head-based sampling is the default and simplest approach in OpenTelemetry. The sampling decision is made at the root span of a trace, before any work is done. The SDK provides several built-in samplers: AlwaysOn (sample everything), AlwaysOff (sample nothing), TraceIdRatioBased (sample a percentage based on the trace ID hash), and ParentBased (respect the sampling decision of the parent span). The ParentBased sampler is particularly important in microservice architectures because it ensures that if the payments-api starts a trace and decides to sample it, all downstream services like order-processing-service and inventory-sync will also sample their spans for that trace. Without ParentBased, you end up with broken traces where some services recorded spans and others did not. The trace ID hash ensures deterministic sampling, meaning the same trace ID always produces the same sampling decision across all services.

Tail-based sampling requires the OpenTelemetry Collector because the decision cannot be made at the SDK level. The Collector receives all spans from all services, assembles complete traces in memory, and then applies policies to decide which traces to keep. Common policies include keeping all traces that contain an error status, keeping traces where the total duration exceeds a threshold like 2 seconds, keeping traces that touch specific services like checkout-service, or applying a probabilistic ratio to the remaining traces. The tail_sampling processor in the Collector supports combining multiple policies with AND and OR logic. For example, you might keep 100% of error traces, 100% of traces slower than 5 seconds, and 1% of everything else.

In production, the choice between head-based and tail-based sampling has significant infrastructure implications. Head-based sampling is lightweight because services discard unsampled spans immediately, reducing CPU, memory, and network overhead at the source. However, it means you lose visibility into rare errors that happen to fall in the unsampled portion. Tail-based sampling captures every interesting trace but requires the Collector to buffer all spans in memory until the trace is complete, which demands significant RAM. A typical tail-sampling Collector handling the traffic from services like payments-api, user-auth-service, and order-processing-service might need 4-8 GB of RAM. The wait_duration parameter controls how long the Collector waits for late-arriving spans before making a decision, typically 30-60 seconds. If a span arrives after the decision window, it is either dropped or sampled probabilistically.

A critical production gotcha is the interaction between head-based and tail-based sampling. If you configure a 10% head-based sampler in your SDKs and also run tail-based sampling in the Collector, you are sampling from an already-sampled stream, which means your tail-based policies only see 10% of traces. The correct setup for tail-based sampling is to use AlwaysOn in the SDK and let the Collector make all sampling decisions. Another common mistake is running a single tail-sampling Collector instance, which creates a single point of failure and limits throughput. In production, you need a load-balancing Collector tier in front of tail-sampling Collectors, using the loadbalancing exporter to route all spans from the same trace to the same tail-sampling instance. Without this, spans from the same trace land on different Collectors, and neither has the complete picture to make an accurate decision.

Think of the OpenTelemetry Collector pipeline like a water treatment plant. Water comes in from multiple sources through intake pipes (receivers), passes through filtration and purification stages (processors), and is distributed to homes and businesses through output pipes (exporters). Each stage is independent and modular. You can add new intake sources without changing the treatment stages, or add new distribution targets without modifying the intake. The Collector acts as a vendor-agnostic telemetry processing hub.

Receivers are the entry points for telemetry data. The Collector supports multiple receiver types simultaneously. The OTLP receiver accepts data via the OpenTelemetry Protocol over gRPC (port 4317) and HTTP (port 4318), and is the primary receiver for applications instrumented with OpenTelemetry SDKs. The Jaeger receiver accepts Jaeger-native formats for legacy applications. The Prometheus receiver scrapes Prometheus-format metrics endpoints, allowing the Collector to replace or augment a Prometheus server. The Kafka receiver consumes telemetry from Kafka topics for high-throughput buffered ingestion. The hostmetrics receiver collects system-level metrics like CPU, memory, and disk from the host. Each receiver can be configured independently with its own ports, TLS settings, and protocol options. The payments-api might send traces via OTLP while the legacy order-processing-service sends Jaeger-format spans.

Processors sit between receivers and exporters and form the core transformation layer. The batch processor is almost always required in production because it groups individual spans or metrics into batches before export, dramatically reducing the number of outbound network calls. The memory_limiter processor prevents the Collector from consuming unbounded memory by applying backpressure when usage exceeds a threshold. The attributes processor adds, modifies, or deletes span attributes, which is useful for injecting environment metadata or stripping sensitive data like user email addresses. The filter processor drops entire spans or metrics matching specific criteria, such as health check endpoints that generate noise. The resource processor enriches all telemetry with resource attributes like deployment environment or cluster name. Processors execute in the order they are listed in the pipeline configuration, so ordering matters. The memory_limiter should always be first to prevent OOM before other processors consume memory.

Exporters send processed telemetry to one or more backends. The OTLP exporter sends data to any OTLP-compatible backend like Grafana Tempo, Jaeger, or another Collector in a multi-tier deployment. The Prometheus exporter exposes metrics in Prometheus format on an HTTP endpoint for scraping. The logging exporter writes telemetry to stdout, useful for debugging. The loadbalancing exporter distributes traces across multiple downstream Collectors by routing spans with the same trace ID to the same instance, essential for tail-based sampling. Multiple exporters can be configured in a single pipeline, allowing you to fan out traces to both Jaeger for real-time viewing and a data lake for long-term analysis.

In production, Collector deployment follows two common patterns. The Agent pattern deploys a Collector as a DaemonSet or sidecar on every node, receiving telemetry from local applications with minimal network latency. The Gateway pattern deploys a centralized Collector cluster that receives data from agents, performs heavy processing like tail-based sampling, and exports to backends. Many organizations use both in a two-tier architecture: lightweight agents on every node forward to a gateway cluster. The Collector configuration is typically managed via ConfigMaps in Kubernetes, and the Collector supports hot-reloading when the ConfigMap changes. A common pitfall is forgetting to configure the batch processor, which results in one HTTP request per span and overwhelms both the Collector and the backend. Another gotcha is not setting memory_limiter, causing the Collector to OOM under traffic spikes from services like checkout-service during flash sales.

Think of resource attributes like the return address on an envelope. No matter what is inside the envelope (a trace, a metric, a log), the return address tells you exactly who sent it, from where, and in what context. Without a return address, you receive thousands of envelopes with no way to sort them by sender. In a microservices architecture with dozens of services like payments-api, user-auth-service, and order-processing-service, resource attributes are what make telemetry data actionable rather than an undifferentiated flood of signals.

Resource attributes are defined by the OpenTelemetry Semantic Conventions, which provide a standardized vocabulary for describing telemetry sources. The most critical attribute is service.name, which uniquely identifies the logical service. Without it, backends like Jaeger or Grafana Tempo cannot group spans into service-level views. Other essential attributes include service.version (the deployed version, critical for correlating deployments with performance changes), service.namespace (grouping related services like the payments domain), deployment.environment (production, staging, development), and service.instance.id (distinguishing between replicas of the same service). Cloud-specific attributes like cloud.provider, cloud.region, and cloud.availability_zone are standardized under the semantic conventions for infrastructure correlation.

In the OpenTelemetry SDK, resource attributes are configured when creating the TracerProvider, MeterProvider, or LoggerProvider. They are set once at startup and apply to all telemetry emitted by that provider. The SDK automatically detects some resource attributes through resource detectors. For example, the AWS resource detector populates cloud.provider, cloud.region, and cloud.account.id. The Kubernetes resource detector fills in k8s.pod.name, k8s.namespace.name, and k8s.node.name. The process resource detector captures process.pid and process.runtime.name. You can combine multiple detectors with manually specified attributes, where manual attributes take precedence over detected ones. Environment variables like OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME provide a configuration mechanism that works without code changes, making it easy to set attributes in Kubernetes manifests or Docker Compose files.

In production, resource attributes are the foundation of effective observability. When the checkout-service experiences a latency spike, you filter traces by service.name=checkout-service and deployment.environment=production. When you deploy a new version, you compare metrics between service.version=2.3.1 and service.version=2.4.0 to detect regressions. When a specific availability zone has network issues, you filter by cloud.availability_zone=us-east-1a to isolate the blast radius. The OpenTelemetry Collector can also enrich telemetry with additional resource attributes using the resource processor, adding attributes like k8s.cluster.name that the SDK might not know about. This is particularly useful in multi-cluster deployments where the same service runs in multiple Kubernetes clusters.

A common gotcha is forgetting to set service.name, which causes backends to group all telemetry under a generic name like unknown_service. Another pitfall is inconsistent naming across services. If the payments-api team uses deployment.environment=prod while the user-auth-service team uses deployment.environment=production, dashboards and queries that filter on environment will miss half the data. Establishing a naming convention and enforcing it through the Collector's resource processor or through shared SDK configuration libraries prevents this fragmentation. High-cardinality resource attributes like service.instance.id are useful for debugging but should not be used as primary grouping dimensions in metrics backends, as they create a new time series per instance.

Think of OpenTelemetry metrics instruments like different tools in a workshop. A counter is like a tally clicker that only goes up, counting events as they happen. A histogram is like a measuring tape that records each measurement and automatically sorts them into buckets for percentile analysis. A gauge is like a thermometer that reports the current temperature at any given moment. Choosing the wrong instrument is like using a thermometer to count cars passing by. It technically produces a number, but the number is meaningless for your purpose.

Counters are the most common instrument type, used for values that only increase over time, such as the total number of HTTP requests handled, bytes transferred, or errors encountered. OpenTelemetry provides two counter variants: Counter (synchronous) where you explicitly call add() in your code, and ObservableCounter (asynchronous) where you register a callback that the SDK invokes at collection time. In the payments-api, you would use a synchronous counter to count processed payments, incrementing it each time a payment completes. The counter value itself is not very useful on its own; you typically apply a rate function in your backend (like rate() in Prometheus) to compute payments per second. UpDownCounters are a variant that can both increase and decrease, useful for tracking values like active connections or items in a queue where the count fluctuates.

Histograms record the distribution of measurements, making them ideal for latency, request sizes, and other values where you care about percentiles rather than averages. When you record a value with a histogram, the SDK assigns it to a bucket based on configurable boundaries. The default bucket boundaries in OpenTelemetry are [0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000] milliseconds. For the checkout-service, you might configure custom boundaries like [10, 50, 100, 200, 500, 1000, 2000, 5000] to get better resolution in the latency range you care about. Exponential histograms are a newer feature that automatically adjust bucket boundaries based on the data, providing high accuracy without manual boundary configuration.

Gauges represent point-in-time values that can go up or down arbitrarily, like CPU utilization, memory usage, queue depth, or active sessions. OpenTelemetry primarily supports gauges through the ObservableGauge (asynchronous) instrument, where a callback function returns the current value at collection time. This design prevents the common mistake of setting a gauge value in a loop and exporting stale data. For the inventory-sync service, you might use a gauge to report the current number of items awaiting synchronization, with the callback querying the actual queue depth at each collection interval.

In production, the choice of instrument type affects how backends store and query the data. Counters are stored as cumulative values and require rate() to be useful in queries. Histograms generate multiple time series per instrument (one per bucket boundary plus sum and count), so cardinality management is critical. If your histogram has 10 buckets and you record with 5 attribute dimensions, each with 10 possible values, you generate 10 * 10^5 = 1,000,000 time series, which will overwhelm most backends. Attributes should be low-cardinality: use http.method (GET, POST, PUT, DELETE) and http.status_code (200, 400, 500), not request_id or user_id. The OpenTelemetry SDK supports Views, which let you override default aggregation settings per instrument, such as changing histogram bucket boundaries or dropping specific attributes before export.

A common gotcha is using a gauge when you need a counter. If you want to count the total number of payments processed, use a counter, not a gauge that you increment manually. Counters handle process restarts correctly because backends compute rate over the cumulative value, while a manually incremented gauge loses its value on restart. Another pitfall is not configuring histogram bucket boundaries for your specific use case. The default boundaries assume millisecond-scale latency, but if your checkout-service responds in microseconds or takes minutes for batch operations, the default buckets provide zero resolution in the ranges that matter.

Think of baggage like a shipping label attached to a package as it moves through a delivery network. Every warehouse and sorting facility along the route can read the label to make decisions: route this package via express, handle with care, deliver to the third floor. Span attributes, by contrast, are like internal notes that each warehouse writes in its own logbook. The next warehouse never sees those notes. Baggage propagates with the request across service boundaries; span attributes stay local to the service that created them.

Baggage in OpenTelemetry is part of the W3C Baggage specification and is propagated via HTTP headers (the baggage header) or gRPC metadata alongside the trace context (traceparent and tracestate headers). When the payments-api sets a baggage entry like tenant.id=acme-corp, every downstream service that handles the same request, including order-processing-service, inventory-sync, and user-auth-service, can read that value without the payments-api needing to explicitly pass it as a parameter. This is particularly powerful for cross-cutting concerns that affect multiple services but originate at the edge, such as tenant identification in multi-tenant systems, A/B test cohort assignment, feature flag values, or request priority levels.

Internally, baggage is stored in the OpenTelemetry context and injected into outgoing requests by the configured propagator. The SDK provides the W3CBaggagePropagator, which serializes baggage entries into the baggage HTTP header as a comma-separated list of key=value pairs. When a downstream service receives the request, the propagator extracts the baggage from the header and makes it available via the baggage API. Importantly, baggage entries are not automatically added to spans. This is a deliberate design decision because baggage can contain sensitive data, and automatically attaching it to every span would export that data to your tracing backend. If you want baggage values to appear as span attributes, you must explicitly copy them, either in application code or using the Collector's baggage processor.

In production, baggage is invaluable for multi-tenant observability. When the checkout-service receives a request, it sets tenant.id in baggage. Every downstream service can then use that tenant ID to add it as a span attribute, include it in log messages, or use it as a metric dimension, all without modifying the API contracts between services. Another common use case is request prioritization: the API gateway sets priority=high in baggage for premium customers, and the order-processing-service reads this to route the request to a dedicated processing queue. Feature flags are another natural fit. If the frontend assigns a user to the new-checkout-flow experiment, it sets experiment.checkout=variant-b in baggage, and every downstream service can record which variant was active, enabling end-to-end A/B test analysis across the entire request path.

The critical gotcha with baggage is that it is propagated to every downstream service, including third-party APIs and external vendors. Sensitive data like user emails, account numbers, or authentication tokens must never be placed in baggage because any service in the chain, including services you do not control, can read and log the values. The baggage header is transmitted in plaintext in HTTP requests, making it visible in network logs and proxy access logs. Another pitfall is baggage size. Each baggage entry adds bytes to every HTTP header on every request. If you add large values or too many entries, you can exceed HTTP header size limits (typically 8 KB in nginx, 16 KB in many load balancers), causing requests to fail with 431 Request Header Fields Too Large errors. Keep baggage entries small, use short keys, and limit the total number of entries to under 10.

Think of auto-instrumentation like installing a dashcam in your car. Once it is mounted, it automatically records every trip without you pressing any buttons. You still have the option to narrate important events by pressing a button (manual instrumentation), but the baseline recording of every start, stop, turn, and speed happens automatically. OpenTelemetry auto-instrumentation works the same way for HTTP and gRPC calls: install the library, and every request is automatically traced with standardized attributes.

For HTTP services, OpenTelemetry provides instrumentation libraries for virtually every popular framework. In Python, opentelemetry-instrumentation-flask instruments Flask applications, opentelemetry-instrumentation-fastapi handles FastAPI, and opentelemetry-instrumentation-django covers Django. In Java, the OpenTelemetry Java agent is a single JAR file that instruments dozens of frameworks automatically via bytecode manipulation, requiring zero code changes. In Go, instrumentation is provided through middleware for net/http, gin, echo, and other frameworks. When auto-instrumentation is active, every incoming HTTP request creates a server span with standardized semantic convention attributes: http.method (GET, POST), http.url or http.route, http.status_code, http.request_content_length, net.host.name, and net.peer.ip. Outgoing HTTP calls via libraries like requests in Python or HttpClient in Java automatically create client spans and inject the trace context (traceparent header) into the outgoing request.

For gRPC services, OpenTelemetry instruments both the client and server sides. The instrumentation registers interceptors on the gRPC channel (client) or server that create spans for each RPC call. Standard attributes include rpc.system (grpc), rpc.service (the protobuf service name like payments.PaymentService), rpc.method (the method name like ProcessPayment), rpc.grpc.status_code (OK, CANCELLED, DEADLINE_EXCEEDED), and net.peer.name. The trace context is propagated through gRPC metadata, which is functionally equivalent to HTTP headers. In the payments-api calling the user-auth-service via gRPC, the client interceptor creates a client span, injects the trace context into the gRPC metadata, and the server interceptor on user-auth-service extracts the context and creates a server span linked to the same trace.

Combining auto-instrumentation with manual instrumentation gives you the best of both worlds. Auto-instrumentation captures the infrastructure layer: every HTTP request, every gRPC call, every database query. Manual instrumentation captures business logic: processing a payment, validating inventory, applying a discount. In the checkout-service, auto-instrumentation automatically creates spans for the incoming POST /api/v1/checkout HTTP request and the outgoing gRPC call to payments-api. You then add manual spans inside the request handler for validate-cart, calculate-tax, and apply-discount, which appear as child spans under the auto-instrumented HTTP span. This gives you a complete picture: the HTTP layer shows you network latency, and the business logic spans show you where time is spent within the service.

In production, there are several important considerations. First, auto-instrumentation can generate high span volume if every health check and readiness probe creates a span. Use the Collector filter processor or SDK-level span filtering to suppress spans for paths like /healthz and /readyz. Second, the http.url attribute can contain sensitive query parameters. Use http.route (the URL template like /users/:id) instead of the full URL with parameters. Most instrumentation libraries support URL sanitization configuration. Third, gRPC streaming RPCs create spans that can remain open for minutes or hours. Configure appropriate timeouts and understand that long-lived spans consume memory in the SDK until they are exported. Fourth, the order of instrumentation initialization matters. The tracing provider must be configured before instrumentation libraries are applied, or spans will be silently dropped.

Think of span events as sticky notes placed on a timeline. If a span represents a road trip from New York to Boston, events are the sticky notes that say 'flat tire at mile 87' or 'stopped for gas at mile 150.' They mark important moments during the journey without creating a separate trip for each event. Span links, by contrast, are like cross-references between different trips. If the road trip was triggered by a phone call (a separate trace), a link connects the call to the trip, saying 'this trip happened because of that call,' even though the call and the trip are recorded separately.

Span events are lightweight annotations added to an existing span using the add_event() method. Each event has a name, a timestamp (automatically set to the current time), and optional attributes. The most common use case is exception recording. When the payments-api catches a transient error and retries, recording the exception as a span event preserves the error details (exception type, message, stack trace) within the span timeline without marking the entire span as failed. If the retry succeeds, the span status remains OK, but the event provides diagnostic context showing that a retry occurred. Other common events include cache hits and misses (event: cache.hit with attributes for the cache key), circuit breaker state changes (event: circuit_breaker.open), retry attempts (event: retry with the attempt number), and validation results. Events are lightweight because they do not create separate span objects, which keeps the trace viewer clean and reduces export overhead.

Span links create explicit relationships between spans that exist in different traces or that do not follow the standard parent-child hierarchy. The canonical example is asynchronous message processing. When the checkout-service publishes an order event to Kafka, the publish operation creates a span in trace A. When the order-processing-service consumes that message minutes later, it starts a new trace B (because the consumer may batch multiple messages or process them independently). A span link from the consumer span in trace B to the producer span in trace A creates a navigable connection between the two traces, allowing an engineer to click from the consumer span to the original producer span in the trace viewer. Links carry the full trace context (trace ID, span ID) of the linked span plus optional attributes describing the relationship.

Other important use cases for span links include fan-in operations, where a single span aggregates results from multiple parallel traces. For example, the inventory-sync service might trigger parallel stock checks across five warehouses, each in its own trace. The aggregation span links to all five warehouse traces, creating a hub-and-spoke navigation pattern. Batch processing is another fit: when a cron job processes 1000 orders from a queue, creating parent-child relationships with all 1000 original order traces would create an impossibly deep trace tree. Instead, the batch processing span links to a representative sample of the original traces. Scheduled jobs triggered by previous operations also benefit from links, connecting the scheduled job trace to the trace that created the schedule.

In production, the distinction between events and links matters for trace viewer usability. Excessive child spans create deep, cluttered trace trees that are hard to navigate. Events keep the trace compact while preserving diagnostic detail. Links keep traces independent and reasonably sized while enabling cross-trace navigation. A common gotcha is recording exceptions as separate child spans instead of events. This inflates the span count, makes the trace tree harder to read, and can trigger false alerts if error detection is based on span status rather than event analysis. Another pitfall is not capturing the producer span context at publish time. If the checkout-service publishes to Kafka without extracting the current span context and attaching it to the message headers, the consumer has nothing to link to.

Think of distributed tracing like a detective following a suspect across multiple cities. At each city, a local officer picks up the tail and writes a report (a span) that includes the case number (trace ID) and who handed off the surveillance (parent span ID). When the detective later assembles all reports by case number, they get a complete timeline of the suspect's journey across every city. Without the case number, each city's report would be an isolated document with no connection to the others.

Distributed tracing in OpenTelemetry relies on three core components working together: context creation, context propagation, and span collection. When a request enters the system at the edge service, typically an API gateway or the first microservice to receive the external request, the OpenTelemetry SDK generates a trace ID (a 128-bit random identifier) and creates the root span. This trace ID is the thread that ties together all spans across all services that participate in handling this request. The root span also gets a span ID (a 64-bit identifier) that child spans will reference as their parent. If the incoming request already carries a traceparent header (because it originated from an already-instrumented system), the SDK extracts the existing trace ID instead of generating a new one, ensuring the trace continues seamlessly.

Context propagation is the mechanism that carries the trace ID and parent span ID from one service to the next. OpenTelemetry supports multiple propagation formats, with W3C Trace Context being the default. The traceparent header carries the trace ID, parent span ID, and trace flags in a standardized format: 00-{trace_id}-{span_id}-{trace_flags}. When the checkout-service makes an HTTP call to the payments-api, the OpenTelemetry instrumentation automatically injects the traceparent header into the outgoing request. When the payments-api receives the request, its instrumentation extracts the header and creates a new span with the same trace ID and the incoming span ID as its parent. This automatic inject-and-extract cycle repeats at every service boundary, creating a connected tree of spans that represents the complete request flow from checkout-service through payments-api, user-auth-service, and inventory-sync.

The span tree is assembled by the tracing backend, not by the services themselves. Each service independently exports its spans to the OpenTelemetry Collector (or directly to a backend like Jaeger or Grafana Tempo). The backend groups all spans with the same trace ID into a single trace view. The parent-child relationships between spans are reconstructed using the parent span ID references. This decoupled design means that services do not need to know about each other's spans. The checkout-service does not wait for the payments-api to finish its spans. Each service exports independently, and the backend assembles the complete picture. The Collector acts as a central aggregation point, receiving spans from all services, batching them, and forwarding them to the backend.

In production, several challenges arise with distributed tracing at scale. Clock skew between services can make span timelines appear incorrect, with child spans starting before their parent. NTP synchronization across all hosts is essential. Missing spans create gaps in traces, which can be caused by services that are not instrumented, network failures that prevent span export, or SDK misconfiguration. The health of the tracing pipeline itself needs monitoring. Track metrics like otelcol_exporter_sent_spans and otelcol_receiver_refused_spans on the Collector to detect data loss. Service mesh proxies like Envoy in Istio can handle context propagation at the infrastructure layer, but the application must still forward incoming headers on outgoing requests. A common mistake is the checkout-service receiving a traceparent header from the client, creating its own new trace ID instead of using the propagated one, and then calling the payments-api with the new trace ID. This produces two disconnected traces instead of one continuous trace. Ensuring that all services use the same propagator configuration (W3C Trace Context) and that framework instrumentation is initialized before any requests are handled prevents this issue.

Think of the OpenTelemetry Collector like a postal system. Agent-mode Collectors are the local mailboxes on every street corner (every Kubernetes node) where residents (applications) drop off their letters (telemetry data). Gateway-mode Collectors are the central post offices that sort mail by destination, apply business rules (like filtering junk mail), and route to the final destinations (backends like Jaeger, Prometheus, or a cloud vendor). You would never have every house send mail directly to the national sorting center, and you would never have just one mailbox for the entire city.

The agent-mode Collector runs as a Kubernetes DaemonSet, meaning one instance per node. Applications on that node send telemetry (traces, metrics, logs) to localhost or the node's IP, which is fast and avoids cross-network hops. The agent handles lightweight tasks: receiving data via OTLP, adding resource attributes like node name and pod name using the k8sattributes processor, performing basic filtering to drop health-check spans, and batching data before forwarding. Because it runs on every node, it must be resource-efficient, typically limited to 200-500MB RAM and 0.5 CPU. If the agent crashes, only that node's telemetry is affected, not the entire pipeline.

The gateway-mode Collector runs as a Kubernetes Deployment with multiple replicas behind a ClusterIP or LoadBalancer service. This tier handles heavy processing: tail-based sampling decisions across the entire fleet (which requires seeing all spans of a trace in one place), complex attribute transformations, routing to multiple backends based on signal type, and rate limiting. Gateway replicas can be autoscaled with HPA based on CPU, memory, or custom metrics like the queue size of the exporter. A typical production setup uses 3-5 gateway replicas, each with 2-4GB RAM.

Scaling is the critical design concern. Agent-mode Collectors scale automatically with nodes (DaemonSet), so you only worry about per-node resource limits. Gateway-mode Collectors need careful capacity planning. The key bottleneck is usually the exporter queue: if the backend (say Tempo or Jaeger) is slow, the queue fills up, backpressure propagates to agents, and telemetry is dropped. You mitigate this with the sending_queue configuration (persistent queue backed by file storage so data survives restarts), retry_on_failure, and proper timeout settings. Monitoring the Collector itself is essential: expose its internal metrics (otelcol_exporter_queue_size, otelcol_receiver_refused_spans) to Prometheus and alert on queue saturation.

A common production pattern at companies like the ones running payments-api, order-processing-service, and checkout-service is to have separate gateway pools per signal type: one gateway Deployment for traces (which need tail-based sampling), another for metrics (which need different batching and aggregation), and a third for logs (which have different volume characteristics). This prevents a log spike from starving trace processing. Each pool has its own HPA policy tuned to its specific workload profile.

Think of tail-based sampling like a film editor reviewing complete movie takes. Head-based sampling is like randomly deciding before filming whether to keep the take or not, meaning you might discard a take that turns out to contain a brilliant performance. Tail-based sampling waits until the entire take is filmed, watches it through, and then decides whether to keep it based on its content. You keep all the takes with great performances (errors, high latency) and discard a percentage of the routine takes (successful, fast requests).

Head-based sampling, which is the default in most SDK configurations, makes the sampling decision at the root span before any processing occurs. You set a probability (say 10%) and randomly keep 1 in 10 traces. The problem is that errors and slow requests happen in less than 1% of traffic, so a 10% head-based sample might miss critical traces entirely. You could set sampling to 100% to catch everything, but that generates enormous volume and cost. Tail-based sampling solves this by deferring the decision until all spans of a trace have been collected, then applying intelligent policies.

The tail_sampling processor in the OpenTelemetry Collector works by buffering spans in memory for a configurable decision_wait period (typically 10-30 seconds). During this window, all spans arriving with the same trace ID are grouped together. After the wait period expires, the processor evaluates the complete trace against a set of policies: did any span have an ERROR status code? Did the total trace duration exceed a latency threshold? Does the trace contain spans from a specific service? Based on these policy evaluations, the trace is either kept or dropped. Composite policies allow you to combine rules, for example keeping 100% of error traces, 100% of traces over 2 seconds, 50% of traces from the payments-api service, and only 5% of everything else.

The critical architectural requirement is that all spans belonging to a single trace must arrive at the same Collector instance. If span A of trace-123 goes to Gateway-1 and span B goes to Gateway-2, neither Collector sees the complete trace and both make incorrect sampling decisions. This is solved using the loadbalancing exporter in the agent-mode Collector, which implements consistent hashing on the trace ID to route all spans of the same trace to the same gateway. The load balancing exporter discovers gateway instances via DNS or Kubernetes service endpoints and distributes traces deterministically.

Memory management is the biggest operational challenge. The Collector must buffer all spans during the decision_wait period. If your system generates 50,000 spans per second and the decision wait is 30 seconds, the Collector buffers 1.5 million spans simultaneously. Each span consumes roughly 1-3KB of memory, so you need 1.5-4.5GB of RAM just for the sampling buffer. You must tune decision_wait (shorter wait means less memory but risk of incomplete traces), num_traces (maximum concurrent traces in the buffer), and expected_new_traces_per_sec. Monitoring otelcol_processor_tail_sampling_count_traces_sampled versus otelcol_processor_tail_sampling_count_traces_dropped is essential to verify your policies are working correctly.

Think of the three observability signals as three different views of the same city. Metrics are the satellite view showing traffic density on every road (aggregate data). Traces are the GPS tracking of individual cars showing their exact route through the city (request-level detail). Logs are the dashcam recordings inside each car (event-level detail). Without correlation, you see traffic jams on the satellite but cannot find the specific cars causing them, and you have thousands of dashcam clips with no way to know which car or road they belong to. Correlation connects all three views so you can zoom from satellite to GPS to dashcam seamlessly.

Exemplars are the bridge between metrics and traces. When the payments-api records a histogram observation for request latency, the OpenTelemetry SDK can attach the current trace ID and span ID as an exemplar on that data point. In Prometheus terms, an exemplar is metadata attached to a histogram bucket that says 'this specific metric sample was produced during trace abc123.' When you see a latency spike on a Grafana dashboard, you click on the spike, Grafana reads the exemplar, extracts the trace ID, and links you directly to the trace in Tempo or Jaeger. This eliminates the manual process of searching for traces in a time window and guessing which one corresponds to the anomaly.

For metrics-to-traces correlation using exemplars, you need three things configured. First, the OpenTelemetry SDK must be configured with both a TracerProvider and a MeterProvider, and the metric instruments must have exemplar filters enabled. In newer SDK versions, the default exemplar filter is TraceBasedExemplarFilter, which automatically attaches trace context to metric recordings when a sampled trace is active. Second, the metrics backend must support exemplars: Prometheus added native exemplar support in 2.26, and Mimir supports them as well. Third, Grafana must have the exemplar data source configured to link from the Prometheus data source to the Tempo data source.

For traces-to-logs correlation, the OpenTelemetry SDK automatically injects trace_id and span_id into log records when the Logs SDK is configured alongside the Traces SDK. If you are using a logging framework like Python's logging, Java's Log4j2, or Go's slog, the OpenTelemetry instrumentation bridges inject trace context into log messages. When you view a trace in Grafana Tempo, you can click 'Logs for this span' and it queries Loki with the trace_id filter, showing exactly the log lines emitted during that span's execution. This works because both the trace and the log carry the same W3C Trace Context identifiers.

In a real production scenario at a company running the order-processing-service, checkout-service, and user-auth-service, the workflow looks like this: an SRE sees a p99 latency spike on the order-processing-service Grafana dashboard. They click the spike, see the exemplar, and jump to a specific trace in Tempo showing that the order-processing-service waited 8 seconds for the inventory-sync service. They click on the inventory-sync span, open its logs in Loki, and see a database connection pool exhaustion error. The entire investigation took 60 seconds instead of 20 minutes of manual log searching. This connected experience is what makes OpenTelemetry's unified data model fundamentally superior to siloed monitoring tools.

Think of telemetry protocols like languages for shipping packages. Jaeger's protocol is like a specialized courier that only delivers letters (traces) using a specific packaging format (Thrift). Zipkin's protocol is another courier for letters but uses different packaging (JSON). OTLP is like a universal shipping system that delivers letters (traces), packages (metrics), and large crates (logs) using the same standardized container format. If you send a richly labeled package via the Jaeger courier, some labels might fall off because Jaeger's format does not support them. OTLP preserves every label because the format was designed for it.

OTLP is defined as a Protocol Buffers schema with three sub-protocols: one for traces, one for metrics, and one for logs. It supports two transport mechanisms: gRPC (on port 4317 by default) and HTTP/protobuf (on port 4318). The gRPC transport provides streaming, bidirectional communication, and built-in backpressure handling. The HTTP transport is simpler and works better through proxies, load balancers, and firewalls that may not support HTTP/2. Both transports serialize data identically using Protocol Buffers, which is significantly more compact than JSON (typically 3-5x smaller on the wire) and faster to serialize and deserialize.

The key advantage of OTLP over Jaeger and Zipkin protocols is semantic fidelity. OpenTelemetry's data model includes rich concepts like span events (structured logs attached to spans), span links (connecting causally related traces that are not parent-child), resource attributes (describing the entity producing telemetry), and instrumentation scope (identifying which library generated the data). When you export via OTLP, all of these concepts are preserved exactly. When you export via the Jaeger exporter, span events get converted to Jaeger logs (losing some structure), span links may be partially supported, and instrumentation scope is dropped. When you export via the Zipkin exporter, the data loss is even greater because Zipkin's model is simpler: no native span events, no span links, and annotations replace structured attributes.

For teams running services like payments-api, user-auth-service, and checkout-service on the OpenTelemetry SDK, the recommendation is clear: always use OTLP as the primary export protocol. Configure SDKs to send OTLP to the OpenTelemetry Collector, then let the Collector export to whatever backend you use. If your backend is Jaeger, the Collector's Jaeger exporter handles the translation. This means you never compromise your SDK-side data model for a backend's protocol limitations, and switching backends requires only a Collector configuration change, not an application code change.

Performance-wise, OTLP with gRPC transport is the fastest option for high-throughput environments. In benchmarks, OTLP/gRPC handles 2-3x more spans per second than the Jaeger Thrift compact protocol on the same hardware, primarily due to Protocol Buffers' efficient binary serialization versus Thrift's overhead. OTLP/HTTP is slightly slower than gRPC but still faster than Zipkin's JSON format. The batch processing built into the OpenTelemetry SDK's BatchSpanProcessor amortizes connection overhead across thousands of spans, making OTLP even more efficient in practice. For the order-processing-service handling 10,000 requests per second, the difference between OTLP and Zipkin JSON can mean the difference between 2% and 8% CPU overhead for telemetry export.

Think of telemetry cost control like managing a library's catalog system. If every book checkout creates a unique catalog card with the borrower's full biography, shoe size, and breakfast preference, the card catalog becomes unmanageable. Instead, the catalog card records the book title, checkout date, and a reference number (trace ID) that links to the borrower's full file stored separately. In OpenTelemetry terms, metrics are the catalog cards (low cardinality, aggregate data) and traces are the full borrower files (high cardinality, detailed data). The mistake teams make is putting the full biography on every catalog card.

Cardinality is the number of unique combinations of attribute values on a metric. A metric like http_request_duration with attributes method (4 values), status_code (5 values), and endpoint (20 values) produces 4 times 5 times 20 equals 400 time series, which is manageable. Add user_id with 1 million unique users, and suddenly you have 400 million time series. At roughly 2 bytes per sample and 15-second scrape intervals, that is 3.2 GB per hour just for one metric. Backends like Prometheus, Mimir, or Datadog charge proportionally to series count, so a single high-cardinality attribute can multiply your observability bill by 1000x.

The first defense is at the SDK level using OpenTelemetry's Views API. A View lets you customize how a metric instrument's measurements are aggregated and which attributes are retained. For the payments-api service, you might define a View that keeps only method, status_code, and route on the HTTP request duration histogram, explicitly dropping user_id, session_id, and request_id. These high-cardinality identifiers remain on spans (where each span is a single event, not an aggregated time series), so you can still find individual user requests via traces. Views are powerful because they operate at the SDK level before data ever leaves the application, reducing both network bandwidth and backend cost.

The second defense is at the Collector level using processors. The attributes processor can delete, hash, or truncate specific attributes from spans and metrics. The filter processor can drop entire metric series matching a pattern. The transform processor can use OTTL (OpenTelemetry Transformation Language) to conditionally modify attributes, for example hashing user_id into 100 buckets to reduce cardinality while preserving some analytical value. For the order-processing-service generating 50,000 spans per second, the Collector can use the probabilistic sampler to reduce volume while the attributes processor strips attributes like db.statement (which often contains unique query parameters).

The third defense is architectural: use the right signal for the right purpose. Metrics should answer aggregate questions (what is the p99 latency of checkout-service?) with low-cardinality attributes. Traces should answer specific questions (why was this particular request slow?) with high-cardinality attributes. Logs should capture detailed event data. When a developer wants to track per-user latency, guide them to record user_id as a span attribute (which has trace-level cardinality) and use exemplars to link from the aggregate metric to specific traces. This pattern gives you both the aggregate view and the per-user detail without the cost explosion.

Monitoring your telemetry pipeline's cost is itself an observability challenge. Track otelcol_processor_dropped_metric_points, otelcol_exporter_sent_spans, and backend-specific ingestion metrics. Set up alerts when cardinality exceeds thresholds, and implement a governance process where new instrumentation must be reviewed for cardinality impact before deployment, much like code review for the inventory-sync service would catch a developer adding order_line_item_sku as a metric attribute.

Think of span processors like customs checkpoints at an airport. Every piece of luggage (span) passes through the checkpoint before boarding the plane (exporter). The default SimpleSpanProcessor is like a checkpoint that waves everything through immediately. The BatchSpanProcessor is like a checkpoint that groups luggage onto pallets for efficient loading. A custom span processor is like adding your own checkpoint that inspects each piece of luggage: stamping it with a destination tag, removing prohibited items (sensitive data), routing oversized luggage to a special handler, or rejecting luggage entirely based on specific criteria.

The SpanProcessor interface in OpenTelemetry has four methods. on_start is called when a span begins, giving you a mutable reference to add or modify attributes before any application code runs within that span's context. on_end is called when a span finishes, with the completed span including its duration, status, events, and all attributes. shutdown is called once during application teardown for cleanup. force_flush ensures all pending spans are exported. Most custom logic lives in on_start and on_end. You can chain multiple processors together, and each span passes through every processor in order.

A common production use case at companies running the payments-api and checkout-service is a PII redaction processor. The on_end method inspects every span attribute and event attribute, replacing values matching patterns like credit card numbers, Social Security numbers, or email addresses with redacted placeholders. This ensures that even if a developer accidentally logs sensitive data as a span attribute during checkout-service instrumentation, it never reaches the tracing backend. Another use case is a business-context enrichment processor: the on_start method reads a thread-local or context variable containing the current customer tier (premium, standard, free) and adds it as a span attribute, enabling the SRE team to filter traces by customer tier without requiring every service to manually add the attribute.

For the order-processing-service, a routing processor can examine the service.name and span status to decide which exporter receives the span. Error spans go to a high-retention exporter (keeping them for 30 days), while successful spans go to a low-retention exporter (keeping them for 7 days). This is implemented by wrapping multiple exporters and calling the appropriate one's export method from within the processor's on_end. This pattern is sometimes called a fan-out processor and gives you fine-grained cost control.

Performance is critical because custom processors run in the hot path of every request. The on_start method is called synchronously within the request thread, so it must be fast, ideally under 1 microsecond. Heavy operations like database lookups or HTTP calls must never happen in on_start. The on_end method in a BatchSpanProcessor runs asynchronously in a background thread, so it has slightly more latitude, but should still avoid blocking I/O. If your custom processor needs external data, cache it locally and refresh periodically. Always benchmark your processor under realistic load: a processor adding 50 microseconds per span at 10,000 requests per second adds 500 milliseconds of cumulative overhead per second across the fleet.

Testing custom processors requires the OpenTelemetry SDK's InMemorySpanExporter, which captures exported spans in a list for assertion. You create a TracerProvider with your custom processor chained with the InMemorySpanExporter, generate test spans, and verify that attributes were added, redacted, or filtered as expected. This is essential for the PII redaction processor where missing a pattern means a compliance violation for the user-auth-service.

Think of the OpenTelemetry Operator like a building superintendent who manages the heating system and installs smart thermometers in every apartment. Without the superintendent, every tenant (development team) would need to buy, install, and maintain their own thermometer (SDK instrumentation) and connect it to the building's heating system (Collector). With the superintendent, tenants just put a sign on their door saying 'please install a thermometer' (add an annotation), and the superintendent handles everything: installs the right thermometer model, wires it to the central system, and manages the heating infrastructure.

The OpenTelemetry Operator is installed via Helm and runs as a controller in the cluster. It watches for two custom resources: OpenTelemetryCollector and Instrumentation. The OpenTelemetryCollector CRD lets you declare Collector deployments in Kubernetes-native YAML instead of manually managing Deployments, DaemonSets, ConfigMaps, and Services. You specify the mode (deployment, daemonset, statefulset, or sidecar), the Collector configuration, resource limits, replicas, and autoscaling parameters. The Operator creates and manages all underlying Kubernetes objects, handles configuration updates with rolling restarts, and supports version upgrades.

The Instrumentation CRD is the most powerful feature. It defines auto-instrumentation configuration for a specific language: which OTLP endpoint to send to, which propagators to use (W3C Trace Context, B3), resource attributes to add, and sampler configuration. Once created, you enable instrumentation for any pod by adding an annotation like instrumentation.opentelemetry.io/inject-java: "true" to the pod spec. The Operator's mutating webhook intercepts pod creation and injects an init container that downloads the appropriate instrumentation agent (like the Java JAVAAGENT JAR), modifies environment variables to activate it, and mounts the agent into the application container. The payments-api service written in Java gets full distributed tracing, metric collection, and log correlation without changing a single line of application code.

For a production Kubernetes environment running the checkout-service, order-processing-service, user-auth-service, and inventory-sync, the typical setup involves three Operator-managed components. First, an OpenTelemetryCollector in daemonset mode for agent-level collection on every node, configured with the k8sattributes processor for Kubernetes metadata enrichment. Second, an OpenTelemetryCollector in deployment mode with 3 replicas for the gateway tier, configured with tail-based sampling and multi-backend export. Third, multiple Instrumentation resources, one per language (Java, Python, Node.js), each configured with the appropriate SDK settings. Teams onboard by adding a single annotation to their Deployment spec, and within minutes their service appears in the tracing backend.

The sidecar mode is worth special mention. When you set mode: sidecar on an OpenTelemetryCollector resource and annotate a pod with sidecar.opentelemetry.io/inject: "true", the Operator injects a Collector container directly into the application pod. This gives each pod its own dedicated Collector, which is useful for multi-tenant clusters where teams want isolated telemetry pipelines, or for services with extremely high telemetry volume that would overwhelm a shared DaemonSet agent. The trade-off is higher resource consumption since every pod runs an additional container. For the inventory-sync service that generates 100,000 spans per minute during peak sync operations, a sidecar Collector prevents it from starving other services of telemetry bandwidth on the node.

Operational considerations include RBAC configuration (the Operator needs permissions to mutate pods and create resources), certificate management for webhook TLS, and careful resource limit tuning on both Collectors and injected init containers. Upgrades should be tested in a staging cluster first, as Operator version changes can affect how instrumentation agents are injected.

Think of distributed tracing like a relay race where runners pass a baton. The baton is the trace context (trace ID and parent span ID). If any runner drops the baton or fails to pass it, the chain breaks. A broken trace in your tracing backend looks like disconnected fragments: the checkout-service shows a root span, and the payments-api shows a separate root span, when they should be parent-child. The trace ID is different in each, meaning context was lost at the HTTP call between them. Troubleshooting means finding where the baton was dropped.

The most common cause of broken traces is propagator mismatch. OpenTelemetry supports multiple propagation formats: W3C Trace Context (traceparent header), B3 (X-B3-TraceId headers, used by Zipkin), Jaeger (uber-trace-id header), and AWS X-Ray (X-Amzn-Trace-Id header). If the checkout-service is configured with W3C propagators and sends a traceparent header, but the payments-api is configured with B3 propagators and looks for X-B3-TraceId, the payments-api will not find any incoming context and will start a new trace. The fix is ensuring every service uses the same propagator configuration. The environment variable OTEL_PROPAGATORS=tracecontext,baggage should be set consistently across all services. In the Collector, the propagator configuration is separate from SDK propagation and only matters if the Collector is acting as a proxy.

The second common cause is middleware or framework interceptors that do not propagate context. If the order-processing-service makes an HTTP call to the inventory-sync service using a custom HTTP client wrapper that does not inject trace context headers, the chain breaks at that call. OpenTelemetry's auto-instrumentation libraries handle this for standard HTTP clients (requests, urllib3, axios, HttpClient), but custom wrappers, message queue producers, or async task dispatchers often need manual context injection. You can verify by inspecting the outgoing HTTP headers: if traceparent is missing, the instrumentation library is not wrapping the HTTP client, or the span context is not active in the current thread or async context.

The third cause is sampling. If the checkout-service uses a head-based sampler at 10% and the payments-api uses a separate head-based sampler at 10%, the probability that both services sample the same trace is only 1% (10% times 10%). The fix is to use a parent-based sampler: the root service makes the sampling decision, and all downstream services respect the parent's decision by reading the sampled flag from the traceparent header. Set OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1 on every service.

For systematic troubleshooting, follow a five-step process. Step one: enable debug logging on the SDK by setting OTEL_LOG_LEVEL=debug, which logs every span creation, context injection, and export attempt. Step two: add the debug exporter to the Collector pipeline, which prints every received span to stdout, letting you verify spans actually arrive at the Collector. Step three: check the Collector's internal metrics: otelcol_receiver_accepted_spans versus otelcol_receiver_refused_spans tells you if spans are being rejected, and otelcol_exporter_send_failed_spans tells you if export is failing. Step four: manually inspect HTTP headers between services using curl or tcpdump to verify traceparent headers are present and correctly formatted. Step five: compare trace IDs across services in your backend: if two services show different trace IDs for what should be the same request, the break is at the call between them.

Async context loss is a subtle variant that affects Node.js and Python applications heavily. In Node.js, if the user-auth-service uses callbacks instead of async/await, the OpenTelemetry context (stored in AsyncLocalStorage) may be lost when the callback executes in a different async context. In Python, using thread pools or asyncio.run_in_executor can lose the context unless you explicitly propagate it. The symptom is spans that exist but are orphaned, having no parent despite being generated within what should be a child operation. The fix involves using context.with_current_context wrappers around callbacks and executor submissions.