advancedobservability

How do you correlate traces, metrics, and logs using exemplars and trace context in OpenTelemetry?

⚡

Quick Answer

Exemplars attach trace IDs to specific metric data points, allowing you to jump from a spike on a metric graph directly to the exact trace that caused it. Trace context propagation (W3C Trace Context headers) links logs to traces by injecting trace_id and span_id into log records. Together, these create a unified observability experience where you can navigate between signals seamlessly.

Detailed Answer

Think of the three observability signals as three different views of the same city. Metrics are the satellite view showing traffic density on every road (aggregate data). Traces are the GPS tracking of individual cars showing their exact route through the city (request-level detail). Logs are the dashcam recordings inside each car (event-level detail). Without correlation, you see traffic jams on the satellite but cannot find the specific cars causing them, and you have thousands of dashcam clips with no way to know which car or road they belong to. Correlation connects all three views so you can zoom from satellite to GPS to dashcam seamlessly.

Exemplars are the bridge between metrics and traces. When the payments-api records a histogram observation for request latency, the OpenTelemetry SDK can attach the current trace ID and span ID as an exemplar on that data point. In Prometheus terms, an exemplar is metadata attached to a histogram bucket that says 'this specific metric sample was produced during trace abc123.' When you see a latency spike on a Grafana dashboard, you click on the spike, Grafana reads the exemplar, extracts the trace ID, and links you directly to the trace in Tempo or Jaeger. This eliminates the manual process of searching for traces in a time window and guessing which one corresponds to the anomaly.

For metrics-to-traces correlation using exemplars, you need three things configured. First, the OpenTelemetry SDK must be configured with both a TracerProvider and a MeterProvider, and the metric instruments must have exemplar filters enabled. In newer SDK versions, the default exemplar filter is TraceBasedExemplarFilter, which automatically attaches trace context to metric recordings when a sampled trace is active. Second, the metrics backend must support exemplars: Prometheus added native exemplar support in 2.26, and Mimir supports them as well. Third, Grafana must have the exemplar data source configured to link from the Prometheus data source to the Tempo data source.

For traces-to-logs correlation, the OpenTelemetry SDK automatically injects trace_id and span_id into log records when the Logs SDK is configured alongside the Traces SDK. If you are using a logging framework like Python's logging, Java's Log4j2, or Go's slog, the OpenTelemetry instrumentation bridges inject trace context into log messages. When you view a trace in Grafana Tempo, you can click 'Logs for this span' and it queries Loki with the trace_id filter, showing exactly the log lines emitted during that span's execution. This works because both the trace and the log carry the same W3C Trace Context identifiers.

In a real production scenario at a company running the order-processing-service, checkout-service, and user-auth-service, the workflow looks like this: an SRE sees a p99 latency spike on the order-processing-service Grafana dashboard. They click the spike, see the exemplar, and jump to a specific trace in Tempo showing that the order-processing-service waited 8 seconds for the inventory-sync service. They click on the inventory-sync span, open its logs in Loki, and see a database connection pool exhaustion error. The entire investigation took 60 seconds instead of 20 minutes of manual log searching. This connected experience is what makes OpenTelemetry's unified data model fundamentally superior to siloed monitoring tools.

# Python: Configure OpenTelemetry with exemplars and log correlation # pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp from opentelemetry import trace, metrics # Import trace and metrics APIs from opentelemetry.sdk.trace import TracerProvider # SDK tracer provider from opentelemetry.sdk.metrics import MeterProvider # SDK meter provider from opentelemetry.sdk.trace.export import BatchSpanProcessor # Batch span export from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader # Periodic metric export from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter # OTLP trace exporter from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter # OTLP metric exporter from opentelemetry.sdk.metrics.view import ExplicitBucketHistogramAggregation # Histogram config import logging # Python standard logging # Set up tracing with OTLP export trace_provider = TracerProvider() # Create tracer provider trace_provider.add_span_processor( # Add batch processor BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317")) # Export to Collector ) trace.set_tracer_provider(trace_provider) # Register globally # Set up metrics with exemplar support metric_reader = PeriodicExportingMetricReader( # Create metric reader OTLPMetricExporter(endpoint="otel-collector:4317"), # Export to Collector export_interval_millis=10000 # Export every 10 seconds ) meter_provider = MeterProvider(metric_readers=[metric_reader]) # Create meter provider metrics.set_meter_provider(meter_provider) # Register globally # Create instruments tracer = trace.get_tracer("order-processing-service") # Named tracer meter = metrics.get_meter("order-processing-service") # Named meter latency_histogram = meter.create_histogram( # Histogram for latency name="order.processing.duration", # Metric name unit="ms", # Unit in milliseconds description="Time to process an order" # Human-readable description ) # Log correlation: inject trace context into logs logging.basicConfig( # Configure logging format format='%(asctime)s %(levelname)s [trace_id=%(otelTraceID)s span_id=%(otelSpanID)s] %(message)s' ) # Include trace/span IDs in every log line logger = logging.getLogger(__name__) # Get logger instance # Application code: trace, metric, and log are auto-correlated with tracer.start_as_current_span("process-order") as span: # Start a traced span logger.info("Processing order #12345") # Log auto-includes trace_id duration_ms = 245.7 # Measured processing time latency_histogram.record(duration_ms) # Exemplar auto-attached with trace_id span.set_attribute("order.id", "12345") # Add business context to span # Grafana data source config for exemplar linking: # Prometheus data source → Internal link → Tempo # URL: ${__value.raw} (extracts trace_id from exemplar) # Query: ${__value.raw} (Tempo trace lookup)

How do you correlate traces, metrics, and logs using exemplars and trace context in OpenTelemetry?

More OpenTelemetry questions