How do you implement distributed tracing across microservices to diagnose latency in a payments flow?

Think of distributed tracing like tracking a package through FedEx's system. When you ship a package, it gets a tracking number (trace ID). At every facility it passes through — pickup, sorting hub, regional center, delivery truck, doorstep — a timestamp is recorded with how long it spent at each stage (spans). If the package arrives 3 days late, you look at the tracking history and see it sat at the Memphis sorting hub for 2 days because of weather. Without tracking, you'd just know it was late but have no idea where the delay occurred. Distributed tracing does the same thing for requests flowing through microservices.

In a banking payments flow, a single API request might touch 5-8 services: the API gateway receives the request, payments-api validates the input and checks idempotency, fraud-detector runs ML scoring against the transaction, settlements-processor debits the source account, transaction-router selects the payment network (Visa, SWIFT, ACH), and a notification service sends the confirmation. Without distributed tracing, when a customer reports a slow payment, you'd have to manually correlate timestamps across 8 different service log files to find the bottleneck. With tracing, you pull up the trace ID and immediately see that fraud-detector took 2.3 seconds because its ML model was loading cold, while every other service completed in under 50ms.

OpenTelemetry provides auto-instrumentation and manual instrumentation options. Auto-instrumentation uses language-specific agents (Java agent JAR, Python sitecustomize, Node.js --require) that automatically capture spans for common frameworks — HTTP clients/servers, database drivers, gRPC, message queue producers/consumers. This gives you 80% of the visibility with zero code changes. Manual instrumentation adds custom spans for business-critical operations: you wrap your payment validation logic, your fraud scoring call, and your settlement execution in explicit spans with domain-specific attributes like payment.amount, payment.currency, merchant.id, and risk.score. These attributes become searchable in your trace backend, letting you query for 'all traces where payment.amount > 10000 and risk.score > 0.8'.

Trace context propagation is what makes distributed tracing work across service boundaries. The W3C Trace Context standard defines the traceparent HTTP header format: version-traceId-parentSpanId-traceFlags. When payments-api calls fraud-detector, the OTel SDK automatically injects the traceparent header into the outgoing HTTP request. Fraud-detector's OTel SDK extracts this header and creates a child span linked to the same trace. This works across HTTP, gRPC (via metadata), and message queues (Kafka headers, RabbitMQ properties). For message queues, the trace context is attached to the message when produced and extracted when consumed, creating a trace that spans asynchronous processing — critical for event-driven banking architectures.

The trace backend (Tempo or Jaeger) stores and indexes the completed spans. Tempo is designed for high-throughput, cost-effective storage — it stores traces in object storage (S3/GCS) without indexing span contents, relying on trace ID lookups and Grafana's integration for search. Jaeger provides richer querying with tag-based search but requires more storage resources. Both support tail-based sampling, where the decision to keep or drop a trace is made after all spans complete — this ensures you always keep error traces and slow traces while sampling routine traffic to control storage costs.

The gotcha that undermines tracing implementations: incomplete propagation. If even one service in the chain doesn't propagate trace context, the trace breaks into disconnected fragments. This commonly happens with legacy services that strip unknown HTTP headers, message queue consumers that don't read trace headers, or services behind load balancers configured to remove non-standard headers. Audit every service boundary and ensure context propagation is verified end-to-end before declaring your tracing implementation complete.