How would you implement a zero-downtime migration from one Kubernetes cluster to another while maintaining stateful workloads?
Quick Answer
Use a phased approach: replicate data stores first, gradually shift traffic via weighted DNS or service mesh, validate with shadow traffic, then cut over with rollback capability.
Detailed Answer
Migration Strategy (4 Phases)
Phase 1: Foundation (Week 1-2) - Deploy identical infrastructure on target cluster using GitOps (ArgoCD) - Set up cross-cluster networking (VPC peering, transit gateway, or service mesh federation) - Establish data replication for stateful workloads (database replicas, PV snapshots)
Phase 2: Parallel Run (Week 3-4) - Deploy applications to target cluster alongside source - Route shadow/mirror traffic to target for validation (Istio traffic mirroring) - Compare responses between clusters for correctness - Monitor latency, error rates, resource usage on target
Phase 3: Gradual Traffic Shift (Week 5-6) - Use weighted DNS (Route53 weighted routing) or service mesh to shift traffic: 5% → 25% → 50% → 100% - At each step: validate SLIs, error budgets, latency percentiles - Stateful workload cut-over: promote read-replica to primary, update connection strings
Phase 4: Cleanup (Week 7) - Keep source cluster in standby for 1 week as rollback - Decommission source cluster after validation period
Stateful Workload Specifics
- Databases: Use logical replication (PostgreSQL) or change data capture (Debezium) for real-time sync - PersistentVolumes: CSI volume snapshots + cross-region restore, or Velero for backup/restore - StatefulSets: Cannot simply move — must re-create with same ordinal identity and reattach data - Kafka/MQ: Use MirrorMaker 2 for topic replication across clusters
Code Example
# Istio traffic mirroring for validation
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
spec:
hosts:
- my-service
http:
- route:
- destination:
host: my-service-source
weight: 95
- destination:
host: my-service-target
weight: 5
mirror:
host: my-service-target
mirrorPercentage:
value: 100.0Interview Tip
This is a staff+ system design question. Emphasize the phased approach with rollback at every stage. Mention specific tools (Velero, Istio mirroring, weighted DNS) and the distinction between stateless and stateful migration strategies.