OpenTelemetry: Zero to Hero

The vendor-neutral standard for observability — one set of APIs, SDKs, and a Collector to produce traces, metrics, and logs and ship them anywhere (Prometheus, Grafana, Jaeger, any backend). You'll instrument an app, run the Collector, understand the pipeline, propagate context across services, and adopt production patterns. Hands-on throughout.

The journey:

1. What OpenTelemetry is and why it matters
2. The three signals — traces, metrics, logs
3. Instrument an app
4. The Collector — receive, process, export
5. Context propagation across services
6. Auto-instrumentation
7. Semantic conventions and sampling
8. Wiring to backends
9. Production practices

---

Prerequisites

OpenTelemetry (OTel) is a CNCF project and the successor to OpenTracing/OpenCensus. Confirm Docker: docker run --rm hello-world.

---

Part 1 — What OpenTelemetry is and why it matters

Before OTel, every observability vendor had its own agent and SDK — instrument once per vendor, and switching meant re-instrumenting everything. OpenTelemetry standardizes the generation of telemetry: a common API/SDK to create signals, a common wire protocol (OTLP), and a Collector to receive, process, and export to any backend.

The payoff: instrument your code once, and route the data to Prometheus, Grafana, Jaeger, Datadog, or anything else by changing Collector config — no vendor lock-in. OTel owns the producer side; your backend stays pluggable.

---

Part 2 — The three signals — traces, metrics, logs

OTel unifies the three pillars of observability:

The breakthrough is correlation: a trace has a trace_id that OTel propagates into logs and links to metrics, so you jump from "latency spiked" (metric) to the exact slow request (trace) to its error line (log). A trace is a tree of spans; each span is one operation with a start/end, attributes, and a parent.

---

Part 3 — Instrument an app

Use the SDK to create spans around meaningful operations (Python shown; every language mirrors it):

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://collector:4317")))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

def handle_order(order_id):
    with tracer.start_as_current_span("handle_order") as span:
        span.set_attribute("order.id", order_id)      # attributes = queryable context
        charge_card(order_id)                          # nested spans link automatically

def charge_card(order_id):
    with tracer.start_as_current_span("charge_card"):
        ...

The SDK batches spans and exports them over OTLP to the Collector (:4317 gRPC / :4318 HTTP). start_as_current_span nests spans into a trace automatically. Add attributes (order.id) so you can filter and group later.

---

Part 4 — The Collector — receive, process, export

The OpenTelemetry Collector is a standalone service that decouples your apps from backends. Its config is a pipeline of receivers → processors → exporters:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  batch: {}                       # batch before export (efficiency)
  memory_limiter: { check_interval: 1s, limit_mib: 512 }

exporters:
  prometheus: { endpoint: 0.0.0.0:8889 }   # metrics scraped by Prometheus
  otlp/jaeger: { endpoint: jaeger:4317, tls: { insecure: true } }

service:
  pipelines:
    traces:  { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlp/jaeger] }
    metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheus] }

Apps send to the Collector; the Collector fans out to backends. This means you re-route or add a backend by editing Collector config, never your app. Processors also transform (drop noisy spans, add resource attributes, redact PII, tail-sample).

---

Part 5 — Context propagation across services

A trace is only useful if it spans services. OTel propagates context — the trace_id and span_id — across process boundaries via HTTP/gRPC headers (the W3C traceparent header):

Service A (span) --HTTP w/ traceparent--> Service B (child span) --> Service C
         └──────────────── one trace, three services ────────────────┘

Instrumentation libraries inject traceparent on outgoing requests and extract it on incoming ones automatically, so B's span becomes a child of A's. The result is a single end-to-end trace across your whole request path — the thing that makes distributed systems debuggable. Propagation "just works" with auto-instrumentation for standard HTTP/gRPC clients.

---

Part 6 — Auto-instrumentation

You don't have to hand-write spans. Auto-instrumentation wraps common libraries (web frameworks, HTTP clients, DB drivers) to emit spans with zero code changes:

# Python: install and run under the agent
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317 \
OTEL_SERVICE_NAME=api \
opentelemetry-instrument python app.py

On Kubernetes, the OpenTelemetry Operator can inject auto-instrumentation into pods via an annotation — no image changes at all. Start with auto-instrumentation for breadth (every HTTP/DB call traced), then add manual spans around your key business operations for depth.

---

Part 7 — Semantic conventions and sampling

Semantic conventions are OTel's standard attribute names (http.request.method, db.system, service.name) so backends understand your data regardless of language. Use them — custom names lose you built-in dashboards and correlation.

Sampling controls cost. Tracing every request at scale is expensive:

• Head sampling — decide at the start (e.g. keep 10%). Simple, but may miss the rare error.
• Tail sampling (in the Collector) — decide after the trace completes, so you can keep all errors and slow traces and drop boring fast ones. Far better signal-to-cost.

processors:
  tail_sampling:
    policies:
      - name: errors, type: status_code, status_code: { status_codes: [ERROR] }
      - name: slow,   type: latency,     latency: { threshold_ms: 500 }

---

Part 8 — Wiring to backends

OTel is producer-only — you choose where data lands:

• Traces → Jaeger, Tempo, or a vendor (via OTLP).
• Metrics → Prometheus (scrape the Collector's prometheus exporter) → Grafana.
• Logs → Loki, OpenSearch, or a vendor.

Because everything speaks OTLP into the Collector, adding Grafana Tempo for traces or switching vendors is a config change. This is the whole promise: instrument once, choose (and change) backends freely.

---

Part 9 — Production practices

• Run the Collector as a gateway (and/or a per-node agent) — apps talk to it, not to backends directly.
• Use OTLP + the Collector, not direct-to-vendor exporters, to stay portable.
• Follow semantic conventions for attribute names — free dashboards and correlation.
• Tail-sample to keep errors/slow traces and control cost.
• Set service.name and resource attributes (env, version, k8s metadata) on everything.
• Batch + memory_limiter processors so the Collector is efficient and safe under load.
• Redact PII in a processor before export.
• Correlate — propagate trace_id into logs so the three signals link.

---

Common mistakes to avoid

• Exporting straight to a vendor from apps. You lose portability and central control — send to the Collector.
• Custom attribute names. Break built-in dashboards; use semantic conventions.
• Sampling away your errors. Head-only sampling drops the rare failure you needed — tail-sample.
• No service.name. Telemetry is unattributable; always set it.
• Tracing everything at 100% at scale. Huge cost; sample deliberately.
• Ignoring the Collector's memory limits. Without memory_limiter/batch it can OOM under bursts.
• Not propagating context. Per-service spans that don't join into one trace defeat the purpose.

---

Practice: try it yourself

1. Instrument. Add manual spans around two operations in an app and export via OTLP to a local Collector.
2. Run the Collector. Configure an OTLP receiver, batch processor, and a Jaeger/Tempo exporter; view the trace.
3. Auto-instrument. Run the same app under auto-instrumentation and confirm HTTP/DB spans appear with no code changes.
4. Correlate. Log a line inside a span and confirm the trace_id matches the trace.
5. Tail-sample. Add a tail-sampling processor that keeps only errors and slow traces; generate both and verify.

Self-check:

• What does OTel standardize, and what stays pluggable?
• How do traces, metrics, and logs correlate?
• What are the three stages of a Collector pipeline?
• Head vs tail sampling — which keeps your errors?

---

Where to go next

• Store & visualize — the Prometheus (metrics) and Grafana (dashboards) tutorials; the Collector feeds them.
• Generate a Collector config with the OTel Collector Config Generator.
• Practice with observability interview questions in the question bank.

You now have the loop: instrument → OTLP to the Collector → process → export anywhere → propagate context → sample smartly. That's vendor-neutral observability done right.