How do you troubleshoot missing spans, broken traces, and context propagation failures in OpenTelemetry?

Think of distributed tracing like a relay race where runners pass a baton. The baton is the trace context (trace ID and parent span ID). If any runner drops the baton or fails to pass it, the chain breaks. A broken trace in your tracing backend looks like disconnected fragments: the checkout-service shows a root span, and the payments-api shows a separate root span, when they should be parent-child. The trace ID is different in each, meaning context was lost at the HTTP call between them. Troubleshooting means finding where the baton was dropped.

The most common cause of broken traces is propagator mismatch. OpenTelemetry supports multiple propagation formats: W3C Trace Context (traceparent header), B3 (X-B3-TraceId headers, used by Zipkin), Jaeger (uber-trace-id header), and AWS X-Ray (X-Amzn-Trace-Id header). If the checkout-service is configured with W3C propagators and sends a traceparent header, but the payments-api is configured with B3 propagators and looks for X-B3-TraceId, the payments-api will not find any incoming context and will start a new trace. The fix is ensuring every service uses the same propagator configuration. The environment variable OTEL_PROPAGATORS=tracecontext,baggage should be set consistently across all services. In the Collector, the propagator configuration is separate from SDK propagation and only matters if the Collector is acting as a proxy.

The second common cause is middleware or framework interceptors that do not propagate context. If the order-processing-service makes an HTTP call to the inventory-sync service using a custom HTTP client wrapper that does not inject trace context headers, the chain breaks at that call. OpenTelemetry's auto-instrumentation libraries handle this for standard HTTP clients (requests, urllib3, axios, HttpClient), but custom wrappers, message queue producers, or async task dispatchers often need manual context injection. You can verify by inspecting the outgoing HTTP headers: if traceparent is missing, the instrumentation library is not wrapping the HTTP client, or the span context is not active in the current thread or async context.

The third cause is sampling. If the checkout-service uses a head-based sampler at 10% and the payments-api uses a separate head-based sampler at 10%, the probability that both services sample the same trace is only 1% (10% times 10%). The fix is to use a parent-based sampler: the root service makes the sampling decision, and all downstream services respect the parent's decision by reading the sampled flag from the traceparent header. Set OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1 on every service.

For systematic troubleshooting, follow a five-step process. Step one: enable debug logging on the SDK by setting OTEL_LOG_LEVEL=debug, which logs every span creation, context injection, and export attempt. Step two: add the debug exporter to the Collector pipeline, which prints every received span to stdout, letting you verify spans actually arrive at the Collector. Step three: check the Collector's internal metrics: otelcol_receiver_accepted_spans versus otelcol_receiver_refused_spans tells you if spans are being rejected, and otelcol_exporter_send_failed_spans tells you if export is failing. Step four: manually inspect HTTP headers between services using curl or tcpdump to verify traceparent headers are present and correctly formatted. Step five: compare trace IDs across services in your backend: if two services show different trace IDs for what should be the same request, the break is at the call between them.

Async context loss is a subtle variant that affects Node.js and Python applications heavily. In Node.js, if the user-auth-service uses callbacks instead of async/await, the OpenTelemetry context (stored in AsyncLocalStorage) may be lost when the callback executes in a different async context. In Python, using thread pools or asyncio.run_in_executor can lose the context unless you explicitly propagate it. The symptom is spans that exist but are orphaned, having no parent despite being generated within what should be a child operation. The fix involves using context.with_current_context wrappers around callbacks and executor submissions.