How do you handle high-cardinality attributes and control telemetry costs in OpenTelemetry?

Think of telemetry cost control like managing a library's catalog system. If every book checkout creates a unique catalog card with the borrower's full biography, shoe size, and breakfast preference, the card catalog becomes unmanageable. Instead, the catalog card records the book title, checkout date, and a reference number (trace ID) that links to the borrower's full file stored separately. In OpenTelemetry terms, metrics are the catalog cards (low cardinality, aggregate data) and traces are the full borrower files (high cardinality, detailed data). The mistake teams make is putting the full biography on every catalog card.

Cardinality is the number of unique combinations of attribute values on a metric. A metric like http_request_duration with attributes method (4 values), status_code (5 values), and endpoint (20 values) produces 4 times 5 times 20 equals 400 time series, which is manageable. Add user_id with 1 million unique users, and suddenly you have 400 million time series. At roughly 2 bytes per sample and 15-second scrape intervals, that is 3.2 GB per hour just for one metric. Backends like Prometheus, Mimir, or Datadog charge proportionally to series count, so a single high-cardinality attribute can multiply your observability bill by 1000x.

The first defense is at the SDK level using OpenTelemetry's Views API. A View lets you customize how a metric instrument's measurements are aggregated and which attributes are retained. For the payments-api service, you might define a View that keeps only method, status_code, and route on the HTTP request duration histogram, explicitly dropping user_id, session_id, and request_id. These high-cardinality identifiers remain on spans (where each span is a single event, not an aggregated time series), so you can still find individual user requests via traces. Views are powerful because they operate at the SDK level before data ever leaves the application, reducing both network bandwidth and backend cost.

The second defense is at the Collector level using processors. The attributes processor can delete, hash, or truncate specific attributes from spans and metrics. The filter processor can drop entire metric series matching a pattern. The transform processor can use OTTL (OpenTelemetry Transformation Language) to conditionally modify attributes, for example hashing user_id into 100 buckets to reduce cardinality while preserving some analytical value. For the order-processing-service generating 50,000 spans per second, the Collector can use the probabilistic sampler to reduce volume while the attributes processor strips attributes like db.statement (which often contains unique query parameters).

The third defense is architectural: use the right signal for the right purpose. Metrics should answer aggregate questions (what is the p99 latency of checkout-service?) with low-cardinality attributes. Traces should answer specific questions (why was this particular request slow?) with high-cardinality attributes. Logs should capture detailed event data. When a developer wants to track per-user latency, guide them to record user_id as a span attribute (which has trace-level cardinality) and use exemplars to link from the aggregate metric to specific traces. This pattern gives you both the aggregate view and the per-user detail without the cost explosion.

Monitoring your telemetry pipeline's cost is itself an observability challenge. Track otelcol_processor_dropped_metric_points, otelcol_exporter_sent_spans, and backend-specific ingestion metrics. Set up alerts when cardinality exceeds thresholds, and implement a governance process where new instrumentation must be reviewed for cardinality impact before deployment, much like code review for the inventory-sync service would catch a developer adding order_line_item_sku as a metric attribute.