How do you build custom OpenTelemetry SDK processors and span processors?

Think of span processors like customs checkpoints at an airport. Every piece of luggage (span) passes through the checkpoint before boarding the plane (exporter). The default SimpleSpanProcessor is like a checkpoint that waves everything through immediately. The BatchSpanProcessor is like a checkpoint that groups luggage onto pallets for efficient loading. A custom span processor is like adding your own checkpoint that inspects each piece of luggage: stamping it with a destination tag, removing prohibited items (sensitive data), routing oversized luggage to a special handler, or rejecting luggage entirely based on specific criteria.

The SpanProcessor interface in OpenTelemetry has four methods. on_start is called when a span begins, giving you a mutable reference to add or modify attributes before any application code runs within that span's context. on_end is called when a span finishes, with the completed span including its duration, status, events, and all attributes. shutdown is called once during application teardown for cleanup. force_flush ensures all pending spans are exported. Most custom logic lives in on_start and on_end. You can chain multiple processors together, and each span passes through every processor in order.

A common production use case at companies running the payments-api and checkout-service is a PII redaction processor. The on_end method inspects every span attribute and event attribute, replacing values matching patterns like credit card numbers, Social Security numbers, or email addresses with redacted placeholders. This ensures that even if a developer accidentally logs sensitive data as a span attribute during checkout-service instrumentation, it never reaches the tracing backend. Another use case is a business-context enrichment processor: the on_start method reads a thread-local or context variable containing the current customer tier (premium, standard, free) and adds it as a span attribute, enabling the SRE team to filter traces by customer tier without requiring every service to manually add the attribute.

For the order-processing-service, a routing processor can examine the service.name and span status to decide which exporter receives the span. Error spans go to a high-retention exporter (keeping them for 30 days), while successful spans go to a low-retention exporter (keeping them for 7 days). This is implemented by wrapping multiple exporters and calling the appropriate one's export method from within the processor's on_end. This pattern is sometimes called a fan-out processor and gives you fine-grained cost control.

Performance is critical because custom processors run in the hot path of every request. The on_start method is called synchronously within the request thread, so it must be fast, ideally under 1 microsecond. Heavy operations like database lookups or HTTP calls must never happen in on_start. The on_end method in a BatchSpanProcessor runs asynchronously in a background thread, so it has slightly more latitude, but should still avoid blocking I/O. If your custom processor needs external data, cache it locally and refresh periodically. Always benchmark your processor under realistic load: a processor adding 50 microseconds per span at 10,000 requests per second adds 500 milliseconds of cumulative overhead per second across the fleet.

Testing custom processors requires the OpenTelemetry SDK's InMemorySpanExporter, which captures exported spans in a list for assertion. You create a TracerProvider with your custom processor chained with the InMemorySpanExporter, generate test spans, and verify that attributes were added, redacted, or filtered as expected. This is essential for the PII redaction processor where missing a pattern means a compliance violation for the user-auth-service.