You're tasked with designing an observability platform for a 500-microservice architecture. How do you handle metrics cardinality explosion and keep costs manageable?
Quick Answer
Control cardinality at the source with label guidelines, use recording rules for pre-aggregation, implement tiered storage (hot/warm/cold), and set per-team cardinality budgets.
Detailed Answer
The Cardinality Problem
Each unique combination of metric name + label values creates a time series. A metric with labels {service, endpoint, status, method, instance} across 500 services × 50 endpoints × 5 statuses × 4 methods × 20 instances = 1 billion series. Prometheus can handle ~10M active series before performance degrades.
Prevention (Most Important)
1. Label guidelines: Ban unbounded labels (user_id, request_id, IP addresses). Publish allowed label values. 2. Metric naming conventions: Require teams to register new metrics via PR review 3. Client-side aggregation: Aggregate in-process before exposing to scrape. Don't export per-request metrics. 4. Histogram buckets: Use default buckets unless justified. Each bucket is a separate series.
Aggregation & Storage
1. Recording rules: Pre-aggregate high-cardinality metrics into lower-cardinality summaries 2. Relabeling: Drop unnecessary labels at scrape time using metric_relabel_configs 3. Tiered storage: Thanos/Cortex/Mimir with: - Hot: 2 hours in-memory (Prometheus) - Warm: 30 days in block storage with compaction - Cold: 1 year in object storage (S3) with downsampling (5m, 1h)
Cost Control
- Per-team cardinality budgets enforced by admission webhook or Mimir tenant limits - Prom-label-proxy to enforce label matchers (teams can only query their own metrics) - Monitor cardinality itself: count({__name__=~".+"}) by (job) - Set per-tenant series limits in Mimir: max_global_series_per_user
Architecture
Prometheus (per-cluster) → Remote write → Mimir (centralized, multi-tenant) → Grafana With Grafana Loki for logs and Tempo for traces, correlated via exemplars and trace IDs.
Code Example
# Recording rule to reduce cardinality
groups:
- name: aggregated-metrics
interval: 30s
rules:
- record: service:http_requests:rate5m
expr: sum(rate(http_requests_total[5m])) by (service, status)
- record: service:http_latency:p99
expr: histogram_quantile(0.99, sum(rate(http_duration_bucket[5m])) by (le, service))
# Drop high-cardinality labels at scrape
scrape_configs:
- job_name: my-service
metric_relabel_configs:
- source_labels: [__name__]
regex: 'http_request_duration_seconds_bucket'
action: drop # Use recording rule instead
- regex: 'request_id|trace_id'
action: labeldropInterview Tip
Cardinality management is the #1 operational challenge of metrics at scale. Lead with prevention (label guidelines, metric registration) before talking about technical solutions. Mention specific numbers (10M series limit, storage tiers) to show you've operated this at scale.