How does Kafka enable event-driven microservices, and how do you guarantee message ordering?

Think of a newspaper printing operation. Writers (producers) submit articles to different sections (topics) — sports, finance, weather. Each section has multiple columns (partitions). Articles within the same column are printed in order, but articles across different columns may interleave. If you want all articles about the same team in order, you assign them to the same column using the team name as a key.

Kafka enables event-driven architecture by acting as a durable, distributed message log between microservices. Instead of services calling each other directly via HTTP (tight coupling), services publish events to Kafka topics and other services consume them independently. A payments service publishes payment-completed events, and the notification service, analytics service, and fraud detection service each consume those events at their own pace without knowing about each other.

Internally, each Kafka topic is divided into partitions. Producers send messages with an optional key, and Kafka hashes the key to determine the partition. All messages with the same key go to the same partition, and within a partition, messages are strictly ordered by offset. Consumer groups assign partitions to consumers, so each partition is read by exactly one consumer in a group. For exactly-once semantics, Kafka provides idempotent producers (deduplicate retries using producer ID and sequence number) and transactional writes (atomic writes across multiple partitions).

At production scale, partition key design is the most important decision. For a payments system, using the account ID as the partition key ensures all events for one account are ordered — payment initiated, payment authorized, payment completed. If you use random partitioning for throughput, you lose ordering guarantees. Consumer group rebalancing during scaling or failures can cause brief processing pauses, so teams should use cooperative sticky assignor to minimize disruption.

The non-obvious gotcha is that ordering only works within a partition, not across partitions. If you need global ordering across all events, you must use a single partition — but that limits throughput to one consumer. Most systems design for per-entity ordering (all events for account-123 in order) rather than global ordering, which is sufficient for almost all business requirements.