Explain Datadog's three pillars of observability and how they correlate. How do you troubleshoot a production issue using metrics, traces, and logs together?
Quick Answer
Metrics identify WHAT is wrong (latency spike), traces show WHERE in the request path the issue is, and logs reveal WHY (specific error message). Datadog correlates them via tags and trace IDs.
Detailed Answer
Three Pillars in Datadog
1. Metrics (Infrastructure/APM): Time-series data showing system health — CPU, memory, request rate, error rate, latency percentiles. Metrics are the first signal: dashboards and monitors alert you to problems.
2. Traces (APM): Distributed traces showing the full path of a request across microservices. Each trace contains spans with timing, service name, resource, and error info. Traces tell you WHICH service and WHICH endpoint is slow.
3. Logs: Detailed text records with context — error messages, stack traces, request payloads. Logs explain the root cause after metrics and traces narrow the scope.
Correlation Workflow
1. Monitor fires: p99 latency > 500ms for order-service 2. Click through to APM → Service Map shows order-service → payment-service is slow 3. View traces: sort by duration, see payment-service spans taking 2s (normally 50ms) 4. Click span → 'View Related Logs' → see logs with trace_id showing Connection pool exhausted 5. Root cause: payment-service DB connection pool at max (20), needs increase
Key Integration
- Unified tagging: service:order-service, env:production, version:2.1.0 - Trace-to-log correlation via dd.trace_id injected into log entries - Metrics from traces: Datadog auto-generates RED metrics (Rate, Errors, Duration) from APM data
Code Example
# Datadog Agent configuration (datadog.yaml)
apm_config:
enabled: true
env: production
logs_enabled: true
# Python APM instrumentation with log correlation
import logging
from ddtrace import tracer, patch_all
patch_all()
logging.basicConfig(
format='%(asctime)s %(levelname)s [dd.trace_id=%(dd.trace_id)s] %(message)s'
)
logger = logging.getLogger(__name__)
@tracer.wrap(service='order-service', resource='create_order')
def create_order(data):
logger.info('Creating order', extra={'order_id': data['id']})
# trace_id is automatically injected into logs
# Datadog monitor (terraform)
resource "datadog_monitor" "latency" {
name = "High p99 latency - order-service"
type = "metric alert"
query = "avg(last_5m):p99:trace.http.request{service:order-service} > 0.5"
message = "p99 latency > 500ms @pagerduty-oncall"
}Interview Tip
Demonstrate the correlation workflow: metrics → traces → logs. This shows you use observability tools to solve problems, not just set them up. Mention unified tagging as the glue that connects all three pillars.