How do you implement distributed tracing across microservices with OpenTelemetry?

Think of distributed tracing like a detective following a suspect across multiple cities. At each city, a local officer picks up the tail and writes a report (a span) that includes the case number (trace ID) and who handed off the surveillance (parent span ID). When the detective later assembles all reports by case number, they get a complete timeline of the suspect's journey across every city. Without the case number, each city's report would be an isolated document with no connection to the others.

Distributed tracing in OpenTelemetry relies on three core components working together: context creation, context propagation, and span collection. When a request enters the system at the edge service, typically an API gateway or the first microservice to receive the external request, the OpenTelemetry SDK generates a trace ID (a 128-bit random identifier) and creates the root span. This trace ID is the thread that ties together all spans across all services that participate in handling this request. The root span also gets a span ID (a 64-bit identifier) that child spans will reference as their parent. If the incoming request already carries a traceparent header (because it originated from an already-instrumented system), the SDK extracts the existing trace ID instead of generating a new one, ensuring the trace continues seamlessly.

Context propagation is the mechanism that carries the trace ID and parent span ID from one service to the next. OpenTelemetry supports multiple propagation formats, with W3C Trace Context being the default. The traceparent header carries the trace ID, parent span ID, and trace flags in a standardized format: 00-{trace_id}-{span_id}-{trace_flags}. When the checkout-service makes an HTTP call to the payments-api, the OpenTelemetry instrumentation automatically injects the traceparent header into the outgoing request. When the payments-api receives the request, its instrumentation extracts the header and creates a new span with the same trace ID and the incoming span ID as its parent. This automatic inject-and-extract cycle repeats at every service boundary, creating a connected tree of spans that represents the complete request flow from checkout-service through payments-api, user-auth-service, and inventory-sync.

The span tree is assembled by the tracing backend, not by the services themselves. Each service independently exports its spans to the OpenTelemetry Collector (or directly to a backend like Jaeger or Grafana Tempo). The backend groups all spans with the same trace ID into a single trace view. The parent-child relationships between spans are reconstructed using the parent span ID references. This decoupled design means that services do not need to know about each other's spans. The checkout-service does not wait for the payments-api to finish its spans. Each service exports independently, and the backend assembles the complete picture. The Collector acts as a central aggregation point, receiving spans from all services, batching them, and forwarding them to the backend.

In production, several challenges arise with distributed tracing at scale. Clock skew between services can make span timelines appear incorrect, with child spans starting before their parent. NTP synchronization across all hosts is essential. Missing spans create gaps in traces, which can be caused by services that are not instrumented, network failures that prevent span export, or SDK misconfiguration. The health of the tracing pipeline itself needs monitoring. Track metrics like otelcol_exporter_sent_spans and otelcol_receiver_refused_spans on the Collector to detect data loss. Service mesh proxies like Envoy in Istio can handle context propagation at the infrastructure layer, but the application must still forward incoming headers on outgoing requests. A common mistake is the checkout-service receiving a traceparent header from the client, creating its own new trace ID instead of using the propagated one, and then calling the payments-api with the new trace ID. This produces two disconnected traces instead of one continuous trace. Ensuring that all services use the same propagator configuration (W3C Trace Context) and that framework instrumentation is initialized before any requests are handled prevents this issue.