What are span events and span links, and when would you use each?

Think of span events as sticky notes placed on a timeline. If a span represents a road trip from New York to Boston, events are the sticky notes that say 'flat tire at mile 87' or 'stopped for gas at mile 150.' They mark important moments during the journey without creating a separate trip for each event. Span links, by contrast, are like cross-references between different trips. If the road trip was triggered by a phone call (a separate trace), a link connects the call to the trip, saying 'this trip happened because of that call,' even though the call and the trip are recorded separately.

Span events are lightweight annotations added to an existing span using the add_event() method. Each event has a name, a timestamp (automatically set to the current time), and optional attributes. The most common use case is exception recording. When the payments-api catches a transient error and retries, recording the exception as a span event preserves the error details (exception type, message, stack trace) within the span timeline without marking the entire span as failed. If the retry succeeds, the span status remains OK, but the event provides diagnostic context showing that a retry occurred. Other common events include cache hits and misses (event: cache.hit with attributes for the cache key), circuit breaker state changes (event: circuit_breaker.open), retry attempts (event: retry with the attempt number), and validation results. Events are lightweight because they do not create separate span objects, which keeps the trace viewer clean and reduces export overhead.

Span links create explicit relationships between spans that exist in different traces or that do not follow the standard parent-child hierarchy. The canonical example is asynchronous message processing. When the checkout-service publishes an order event to Kafka, the publish operation creates a span in trace A. When the order-processing-service consumes that message minutes later, it starts a new trace B (because the consumer may batch multiple messages or process them independently). A span link from the consumer span in trace B to the producer span in trace A creates a navigable connection between the two traces, allowing an engineer to click from the consumer span to the original producer span in the trace viewer. Links carry the full trace context (trace ID, span ID) of the linked span plus optional attributes describing the relationship.

Other important use cases for span links include fan-in operations, where a single span aggregates results from multiple parallel traces. For example, the inventory-sync service might trigger parallel stock checks across five warehouses, each in its own trace. The aggregation span links to all five warehouse traces, creating a hub-and-spoke navigation pattern. Batch processing is another fit: when a cron job processes 1000 orders from a queue, creating parent-child relationships with all 1000 original order traces would create an impossibly deep trace tree. Instead, the batch processing span links to a representative sample of the original traces. Scheduled jobs triggered by previous operations also benefit from links, connecting the scheduled job trace to the trace that created the schedule.

In production, the distinction between events and links matters for trace viewer usability. Excessive child spans create deep, cluttered trace trees that are hard to navigate. Events keep the trace compact while preserving diagnostic detail. Links keep traces independent and reasonably sized while enabling cross-trace navigation. A common gotcha is recording exceptions as separate child spans instead of events. This inflates the span count, makes the trace tree harder to read, and can trigger false alerts if error detection is based on span status rather than event analysis. Another pitfall is not capturing the producer span context at publish time. If the checkout-service publishes to Kafka without extracting the current span context and attaching it to the message headers, the consumer has nothing to link to.