What are the three pillars of observability in OpenTelemetry (traces, metrics, logs)?

Imagine diagnosing a car problem. Traces are like the GPS route showing where the car went and how long each leg took. Metrics are the dashboard gauges showing speed, RPM, and fuel level over time. Logs are the mechanic's notes recording specific events like 'engine light turned on at 45,000 miles.' You need all three together to understand what happened and why.

Traces in OpenTelemetry represent the end-to-end journey of a request as it flows through a distributed system. When a user clicks 'Place Order' on checkout-service, that request might flow to order-processing-service, then to inventory-sync to check stock, then to payments-api to charge the card. A trace captures this entire journey as a tree of spans, where each span represents a unit of work in a specific service. The trace gives you latency breakdowns showing that order-processing-service took 50ms, inventory-sync took 120ms, and payments-api took 300ms. Without traces, debugging a slow request across 15 microservices becomes a guessing game of searching through logs on each service independently.

Metrics are numerical values that measure system behavior over time. OpenTelemetry supports three primary metric instruments: counters (monotonically increasing values like total_requests or total_errors), gauges (values that go up and down like active_connections or queue_depth), and histograms (distributions of values like request_duration or response_size). Metrics are lightweight and designed for aggregation. Instead of recording every single request, you record summaries: the average latency, the 99th percentile response time, the error rate per minute. In a production environment, metrics from payments-api might show that the p99 latency increased from 200ms to 800ms over the last hour, which triggers an alert. Metrics tell you that something is wrong and help you identify the affected service.

Logs are timestamped text records of discrete events. A log entry might say '2024-03-15T10:23:45Z ERROR payments-api: Failed to charge card ending in 4242, timeout after 30s.' OpenTelemetry's logging support correlates logs with traces by embedding trace IDs and span IDs into log entries. This correlation is the key differentiator: when user-auth-service logs an authentication failure, you can click the trace ID and see the full request path that led to that failure. Without correlation, logs from checkout-service, order-processing-service, and payments-api exist in separate silos, and connecting them requires manual timestamp matching and guesswork.

The power of OpenTelemetry comes from unifying all three pillars under a single framework with shared context. A trace ID generated when a request enters checkout-service propagates through every downstream service. Metrics recorded during that request carry the same service metadata. Logs emitted during processing embed the trace ID. This means you can start with a metric alert (p99 latency spike on inventory-sync), drill into traces to find slow requests, and then examine the logs for those specific traces to understand the root cause. This workflow, moving seamlessly from metrics to traces to logs, is what observability practitioners call the 'three pillars' working together.