How does the Datadog Agent work? What is DogStatsD and when would you use custom metrics vs APM metrics?
Quick Answer
The Datadog Agent runs on each host, collects system metrics, receives APM traces, and tails logs. DogStatsD is a UDP/UDS listener for custom application metrics. Use custom metrics for business KPIs, APM metrics for request-level performance.
Detailed Answer
Agent Architecture
- Core Agent: Collects host-level metrics (CPU, memory, disk, network) and runs integration checks (Redis, PostgreSQL, etc.) - Trace Agent: Receives APM traces from instrumented applications via localhost:8126 - Log Agent: Tails log files or receives logs via TCP/UDP, processes and forwards to Datadog - Process Agent: Collects live process and container information - DogStatsD: UDP/UDS server (port 8125) that receives custom metrics from applications
DogStatsD Custom Metrics
Applications send custom metrics via lightweight UDP packets — no blocking, fire-and-forget. Types: counters, gauges, histograms, distributions, sets.
When to use each
- APM metrics (automatic): Request rate, error rate, latency — generated from traces. No code changes needed beyond initial instrumentation. - Custom metrics (DogStatsD): Business metrics (orders_placed, revenue, cart_abandonment), infrastructure metrics not covered by integrations, SLI measurements.
Cost consideration: Custom metrics are billed per unique time series. High-cardinality tags (user_id, request_id) on custom metrics can explode costs. Use Distributions instead of Histograms for server-side aggregation to reduce cardinality.
Code Example
# Send custom metrics via DogStatsD (Python)
from datadog import statsd
# Counter - track event occurrences
statsd.increment('orders.placed', tags=['env:production', 'region:us-east'])
# Gauge - track current value
statsd.gauge('queue.depth', 142, tags=['queue:order-processing'])
# Distribution - track latency with percentiles
statsd.distribution('payment.processing_time', 0.250, tags=['provider:stripe'])
# Histogram - client-side aggregation
statsd.histogram('api.response_size', 1024, tags=['endpoint:/v2/orders'])
# DogStatsD config in Kubernetes (Helm values)
datadog:
dogstatsd:
useHostPort: true
port: 8125
nonLocalTraffic: trueInterview Tip
Show you understand the cost implications of custom metrics. Mention tag cardinality as the #1 cost driver. Distribution metrics (server-side aggregation) vs Histograms (client-side) is a key distinction.