intermediatemetrics

How do you create custom metrics using the OpenTelemetry Metrics API (counters, histograms, gauges)?

⚡

Quick Answer

OpenTelemetry provides three primary metric instrument types: Counters (monotonically increasing values like request counts), Histograms (distribution of measurements like latency), and Gauges (point-in-time values like queue depth). You create instruments via a Meter instance, record measurements with attributes, and the SDK periodically exports aggregated data to backends.

Detailed Answer

Think of OpenTelemetry metrics instruments like different tools in a workshop. A counter is like a tally clicker that only goes up, counting events as they happen. A histogram is like a measuring tape that records each measurement and automatically sorts them into buckets for percentile analysis. A gauge is like a thermometer that reports the current temperature at any given moment. Choosing the wrong instrument is like using a thermometer to count cars passing by. It technically produces a number, but the number is meaningless for your purpose.

Counters are the most common instrument type, used for values that only increase over time, such as the total number of HTTP requests handled, bytes transferred, or errors encountered. OpenTelemetry provides two counter variants: Counter (synchronous) where you explicitly call add() in your code, and ObservableCounter (asynchronous) where you register a callback that the SDK invokes at collection time. In the payments-api, you would use a synchronous counter to count processed payments, incrementing it each time a payment completes. The counter value itself is not very useful on its own; you typically apply a rate function in your backend (like rate() in Prometheus) to compute payments per second. UpDownCounters are a variant that can both increase and decrease, useful for tracking values like active connections or items in a queue where the count fluctuates.

Histograms record the distribution of measurements, making them ideal for latency, request sizes, and other values where you care about percentiles rather than averages. When you record a value with a histogram, the SDK assigns it to a bucket based on configurable boundaries. The default bucket boundaries in OpenTelemetry are [0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000] milliseconds. For the checkout-service, you might configure custom boundaries like [10, 50, 100, 200, 500, 1000, 2000, 5000] to get better resolution in the latency range you care about. Exponential histograms are a newer feature that automatically adjust bucket boundaries based on the data, providing high accuracy without manual boundary configuration.

Gauges represent point-in-time values that can go up or down arbitrarily, like CPU utilization, memory usage, queue depth, or active sessions. OpenTelemetry primarily supports gauges through the ObservableGauge (asynchronous) instrument, where a callback function returns the current value at collection time. This design prevents the common mistake of setting a gauge value in a loop and exporting stale data. For the inventory-sync service, you might use a gauge to report the current number of items awaiting synchronization, with the callback querying the actual queue depth at each collection interval.

In production, the choice of instrument type affects how backends store and query the data. Counters are stored as cumulative values and require rate() to be useful in queries. Histograms generate multiple time series per instrument (one per bucket boundary plus sum and count), so cardinality management is critical. If your histogram has 10 buckets and you record with 5 attribute dimensions, each with 10 possible values, you generate 10 * 10^5 = 1,000,000 time series, which will overwhelm most backends. Attributes should be low-cardinality: use http.method (GET, POST, PUT, DELETE) and http.status_code (200, 400, 500), not request_id or user_id. The OpenTelemetry SDK supports Views, which let you override default aggregation settings per instrument, such as changing histogram bucket boundaries or dropping specific attributes before export.

A common gotcha is using a gauge when you need a counter. If you want to count the total number of payments processed, use a counter, not a gauge that you increment manually. Counters handle process restarts correctly because backends compute rate over the cumulative value, while a manually incremented gauge loses its value on restart. Another pitfall is not configuring histogram bucket boundaries for your specific use case. The default boundaries assume millisecond-scale latency, but if your checkout-service responds in microseconds or takes minutes for batch operations, the default buckets provide zero resolution in the ranges that matter.

# Create custom metrics using OpenTelemetry Python SDK from opentelemetry import metrics # Import the SDK metrics module from opentelemetry.sdk.metrics import MeterProvider # Import the OTLP metrics exporter from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter # Import periodic reader for scheduled export from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader # Import View for customizing aggregation from opentelemetry.sdk.metrics.view import View, ExplicitBucketHistogramAggregation # Define custom histogram bucket boundaries for latency in milliseconds latency_view = View( instrument_name="checkout.request.duration", # Match this specific instrument aggregation=ExplicitBucketHistogramAggregation( # Override default buckets boundaries=[10, 50, 100, 200, 500, 1000, 2000, 5000] # Boundaries in ms ), ) # Configure the OTLP exporter to send metrics to the Collector exporter = OTLPMetricExporter(endpoint="otel-collector:4317", insecure=True) # Create a periodic reader that exports every 30 seconds reader = PeriodicExportingMetricReader(exporter, export_interval_millis=30000) # Initialize the MeterProvider with custom views provider = MeterProvider(metric_readers=[reader], views=[latency_view]) # Set as the global meter provider metrics.set_meter_provider(provider) # Get a Meter instance for the checkout-service meter = metrics.get_meter("checkout-service", version="2.4.0") # --- COUNTER: Track total number of processed orders --- order_counter = meter.create_counter( name="checkout.orders.total", # Metric name description="Total number of orders processed", # Human-readable description unit="orders", # Unit of measurement ) # Record a successful order with attributes order_counter.add(1, { # Increment by 1 "payment.method": "credit_card", # Low-cardinality attribute "order.status": "completed", # Order outcome "currency": "USD", # Currency type }) # --- HISTOGRAM: Track request latency distribution --- latency_histogram = meter.create_histogram( name="checkout.request.duration", # Metric name (matches View above) description="Request duration for checkout operations",# Description unit="ms", # Milliseconds ) # Record a latency measurement with attributes latency_histogram.record(235.7, { # Record 235.7ms latency "http.method": "POST", # HTTP method "http.route": "/api/v1/checkout", # API route (not full URL) "http.status_code": 200, # Response status code }) # --- UP-DOWN COUNTER: Track items in processing queue --- queue_counter = meter.create_up_down_counter( name="checkout.queue.depth", # Metric name description="Number of items in the checkout queue", # Description unit="items", # Unit ) # Increment when item is added to queue queue_counter.add(1, {"queue.name": "payment-processing"}) # Item enqueued # Decrement when item is processed queue_counter.add(-1, {"queue.name": "payment-processing"}) # Item dequeued # --- OBSERVABLE GAUGE: Report current inventory levels --- def get_inventory_level(options): # Callback function # Query actual inventory database for current count current_level = 4250 # Simulated database query options.observe(current_level, { # Report the current value "warehouse": "us-east-1", # Warehouse location "product.category": "electronics", # Product category }) # Register the gauge with its callback function meter.create_observable_gauge( name="inventory.stock.level", # Metric name description="Current stock level in warehouse", # Description unit="items", # Unit callbacks=[get_inventory_level], # Callback list )

How do you create custom metrics using the OpenTelemetry Metrics API (counters, histograms, gauges)?

More OpenTelemetry questions