What is Grafana Tempo and how does it integrate with Prometheus metrics and Loki logs for full observability?
Quick Answer
Tempo is a distributed tracing backend that stores traces in object storage. It integrates via exemplars (metrics→traces), trace-to-logs (traces→Loki), and service graphs (traces→metrics).
Detailed Answer
Grafana Tempo
- Open-source, high-scale distributed tracing backend - Stores traces in object storage (S3/GCS) — no local disk needed - Only requires trace ID for lookup — no indexing of span attributes (similar to Loki's approach for logs) - Supports OpenTelemetry, Jaeger, and Zipkin protocols
Integration Points
1. Metrics → Traces (Exemplars): - Prometheus stores exemplar data: a specific trace_id attached to a metric sample - In Grafana, clicking a data point on a metrics panel shows the exemplar - Click the exemplar → jumps directly to the trace in Tempo - Use case: High latency spike in p99 → click → see the exact slow trace
2. Traces → Logs (Trace-to-Logs): - Configure Tempo data source with Loki as the linked log source - Click a trace span → 'View Logs' → Loki query filtered by trace_id, service, and time range - Shows the exact log entries for that request
3. Traces → Metrics (Service Graph): - Tempo generates service graph metrics from traces automatically - Visualizes request rate, error rate, and latency between services - Stored as Prometheus metrics for dashboarding
The result: A fully correlated observability stack where you can seamlessly navigate between metrics, traces, and logs without context-switching tools.
Code Example
# Tempo configuration (tempo.yaml)
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
storage:
trace:
backend: s3
s3:
bucket: tempo-traces
endpoint: s3.amazonaws.com
metrics_generator:
storage:
path: /var/tempo/wal
traces_storage:
path: /var/tempo/traces
processor:
service_graphs:
dimensions: ["service"] # Generate service graph metrics
# Grafana data source provisioning with correlations
apiVersion: 1
datasources:
- name: Tempo
type: tempo
url: http://tempo:3200
jsonData:
tracesToLogs:
datasourceUid: loki
tags: ['service.name']
filterByTraceID: true
tracesToMetrics:
datasourceUid: prometheus
tags: [{ key: 'service.name', value: 'service' }]
serviceMap:
datasourceUid: prometheusInterview Tip
The three-way correlation (metrics↔traces↔logs) is what makes the Grafana stack compelling. Walk through a debugging scenario: metric alert → exemplar → trace → logs. This shows you use observability tools holistically.