advancedGCP

Your team needs to implement a real-time event streaming platform on GCP for processing clickstream data at 500K events/second. Compare Pub/Sub with managed Apache Kafka (Confluent on GCP or self-managed on GKE) for this use case. What are the ordering, exactly-once, and replay trade-offs?

⚡

Quick Answer

Pub/Sub offers serverless scaling and at-least-once delivery with ordering keys, but limited replay (7-day retention). Kafka provides true partition-level ordering, exactly-once semantics, longer retention, and consumer group flexibility but requires cluster management. Use Pub/Sub for cloud-native event routing; Kafka for stream processing requiring replay and ordering guarantees.

Detailed Answer

Pub/Sub Architecture and Guarantees

Pub/Sub is a fully managed, serverless messaging service. It automatically scales to handle millions of messages per second without provisioning. Messages are stored across multiple zones and replicated for durability. Pub/Sub provides at-least-once delivery by default, with exactly-once delivery available via Dataflow (which handles deduplication). Ordering is supported within ordering keys—messages with the same key are delivered in publish order to a single subscriber, but there's no global ordering across keys.

Kafka's Strengths for Stream Processing

Kafka provides partition-level ordering guarantees, configurable retention (days to infinite with tiered storage), consumer group-based parallel processing, and exactly-once semantics with idempotent producers and transactional APIs. For clickstream analytics requiring sessionization (grouping events by user session), Kafka Streams or ksqlDB can perform windowed aggregations directly on the broker cluster. Kafka's log-based architecture means any consumer can replay from any offset, enabling reprocessing of historical data when business logic changes.

Performance and Cost at 500K events/sec

Pub/Sub pricing is $40/TiB for message delivery plus $0.20/GiB/month for retained messages. At 500K events/sec with 1KB average message size, that's ~43 TiB/day or ~$1,720/day in delivery costs alone. Kafka on GKE with 10 n2-highmem-8 brokers (with Persistent Disk) costs ~$4,000/month but handles the throughput. Confluent Cloud on GCP charges by CKU ($1,800/CKU/month) with 1-2 CKUs sufficient for this load. At high volumes, Kafka is significantly cheaper than Pub/Sub.

When to Choose Each

Pub/Sub excels as event routing infrastructure: fan-out to multiple subscribers, push delivery to Cloud Run/Cloud Functions, integration with Dataflow for ETL, and zero operational overhead. Kafka excels as a stream processing platform: complex event processing, exactly-once transformations, multi-datacenter replication (MirrorMaker 2), and long-term event sourcing. Many production architectures use both: Pub/Sub for ingest and routing, then bridge to Kafka for stream processing.

Operational Considerations

Pub/Sub requires zero cluster management—Google handles scaling, replication, and failover. Kafka (even managed) requires partition planning, broker sizing, ISR monitoring, consumer lag tracking, and rolling upgrade management. A platform team of 2-3 engineers typically manages a production Kafka cluster, while Pub/Sub can be managed by the application team directly.

# Pub/Sub setup for clickstream ingestion gcloud pubsub topics create clickstream-events \ --message-retention-duration=7d gcloud pubsub subscriptions create clickstream-analytics \ --topic=clickstream-events \ --ack-deadline=60 \ --enable-exactly-once-delivery \ --enable-message-ordering \ --max-delivery-attempts=5 \ --dead-letter-topic=clickstream-dlq # Publish with ordering key (user_id for session ordering) from google.cloud import pubsub_v1 from google.protobuf import timestamp_pb2 import json, time publisher = pubsub_v1.PublisherClient() topic_path = publisher.topic_path('myproject', 'clickstream-events') # Enable ordering on publisher publisher_options = pubsub_v1.types.PublisherOptions( enable_message_ordering=True, ) publisher = pubsub_v1.PublisherClient(publisher_options=publisher_options) event = { "user_id": "user-12345", "event_type": "page_view", "url": "/products/widget-a", "timestamp": time.time() } future = publisher.publish( topic_path, json.dumps(event).encode('utf-8'), ordering_key="user-12345" # Ensures per-user ordering ) # --- Kafka alternative on GKE --- # Helm install Strimzi Kafka operator helm repo add strimzi https://strimzi.io/charts/ helm install strimzi strimzi/strimzi-kafka-operator -n kafka # Kafka cluster manifest apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: clickstream-cluster namespace: kafka spec: kafka: replicas: 6 version: 3.7.0 config: num.partitions: 64 default.replication.factor: 3 min.insync.replicas: 2 log.retention.hours: 168 message.max.bytes: 1048576 storage: type: persistent-claim size: 500Gi class: premium-rwo resources: requests: { memory: 16Gi, cpu: 4 } limits: { memory: 24Gi, cpu: 8 } zookeeper: replicas: 3 storage: type: persistent-claim size: 50Gi # Topic with 64 partitions for parallelism kubectl apply -f - <<EOF apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaTopic metadata: name: clickstream-events labels: strimzi.io/cluster: clickstream-cluster spec: partitions: 64 replicas: 3 config: retention.ms: 604800000 cleanup.policy: delete compression.type: lz4 EOF

Your team needs to implement a real-time event streaming platform on GCP for processing clickstream data at 500K events/second. Compare Pub/Sub with managed Apache Kafka (Confluent on GCP or self-managed on GKE) for this use case. What are the ordering, exactly-once, and replay trade-offs?

More GCP questions