How do you correlate metrics, logs, and traces using OpenTelemetry and Grafana?

Think of investigating a crime scene. Fingerprints (metrics) tell you something happened. Witness statements (logs) describe what happened. Security camera footage (traces) shows exactly who went where and when. Each clue is useful alone, but correlating them — the fingerprint matches the person on camera who matches the witness description — solves the case. Observability correlation works the same way.

OpenTelemetry is the CNCF standard for collecting and exporting telemetry data. A single OpenTelemetry SDK instrumented in your application generates metrics, logs, and traces with shared context. The key correlation mechanism is the trace ID — a unique identifier generated when a request enters the system. This trace ID is automatically injected into every log line, every metric exemplar, and every trace span generated during that request's lifecycle across all microservices.

The Grafana stack provides the visualization and correlation layer. Prometheus stores metrics (request rate, error rate, latency histograms). Loki stores logs (structured log lines with trace IDs as labels). Tempo stores traces (distributed spans showing the full request journey). Grafana connects to all three and provides cross-signal navigation: click an anomalous point on a latency chart, see the exemplar trace ID, jump to the full trace in Tempo, then click any span to see the logs for that service during that request. This trace-to-metric-to-log correlation turns a 30-minute investigation into a 3-minute one.

At production scale, teams must manage cardinality and cost. Metrics should use bounded label values (HTTP method, status code category) not unbounded ones (user ID, request ID). Logs should be structured JSON with consistent fields across services. Traces should use head-based or tail-based sampling to reduce storage — capturing 100% of traces is prohibitively expensive at high throughput, but capturing 100% of error traces and 1% of success traces provides good coverage.

The non-obvious gotcha is that correlation only works when all three signals share the same trace ID format and propagation context. If one service uses W3C TraceContext and another uses B3 propagation, the trace breaks. Teams must standardize on one propagation format across all services and ensure the OpenTelemetry SDK is configured consistently. Also, log-to-trace correlation requires the trace ID to be a searchable field in Loki — adding it as a label enables fast lookups but increases cardinality, while using structured metadata keeps cardinality low but requires full-text search.