What is the difference between automatic and manual instrumentation?

Think of automatic instrumentation like a dashcam that records everything your car does without you pressing any buttons. Manual instrumentation is like a passenger taking notes about specific landmarks and events along the route. The dashcam captures the broad picture, but the passenger's notes add context and meaning that a camera cannot capture on its own.

Automatic instrumentation (also called auto-instrumentation or zero-code instrumentation) works by hooking into popular libraries and frameworks to capture telemetry without modifying application code. In Python, the opentelemetry-instrument command wraps your application and automatically instruments libraries like Flask, Django, requests, SQLAlchemy, and psycopg2. In Java, a Java agent JAR is attached to the JVM at startup, intercepting method calls in frameworks like Spring Boot, gRPC, and JDBC. In Node.js, the @opentelemetry/auto-instrumentations-node package registers instrumentation hooks before your application loads its dependencies. The result is that a payments-api built with Flask and PostgreSQL immediately gets spans for every HTTP request and every database query without a single line of instrumentation code.

Manual instrumentation gives developers explicit control over what telemetry is generated. Using the OpenTelemetry API, a developer in order-processing-service can create a span named 'validate_inventory' that wraps the specific business logic for checking stock levels against inventory-sync. They can add custom attributes like order.item_count=5 and order.total=249.99, record events like 'backorder_triggered' when stock is insufficient, and set the span status to ERROR with a descriptive message when validation fails. None of this business-specific context can be captured by automatic instrumentation because the auto-instrumenter only sees HTTP calls and database queries, not the business meaning behind them.

In production, the standard approach is to layer both techniques. Automatic instrumentation provides the baseline: every HTTP request to checkout-service generates a span, every database query in user-auth-service is traced, and every outgoing HTTP call to payments-api creates a client span. Manual instrumentation then enriches this baseline with business context. A developer adds a custom span around the fraud detection logic in payments-api, sets attributes for the risk score and decision, and records events for each fraud rule evaluated. The auto-generated spans provide the structural skeleton of the trace, while manual spans fill in the business logic details that matter for debugging.

There are important tradeoffs to understand. Automatic instrumentation requires minimal effort but produces generic telemetry. Every Flask endpoint gets a span named 'POST /api/payments,' but without manual enrichment, you cannot tell whether the payment was for a subscription renewal or a one-time purchase. Manual instrumentation captures exactly what you need but requires developer discipline and ongoing maintenance. If a team adds a new payment provider to payments-api but forgets to add manual spans, that code path becomes a blind spot. Platform teams address this by establishing instrumentation guidelines and reviewing spans as part of the code review process, treating observability as a first-class concern alongside testing.