A Kubernetes cluster experiences a cascading failure where one microservice's pod crashes cause a chain reaction bringing down 15 dependent services. Describe the failure propagation mechanism and what architectural patterns prevent this.
Quick Answer
Cascading failures propagate through retry storms, connection pool exhaustion, and health check failures. Prevent with circuit breakers, bulkhead patterns, pod disruption budgets, priority classes, and graceful degradation with fallback responses.
Detailed Answer
Cascading Failure Anatomy
Service A depends on Service B. When Service B pods crash (OOM, bug, resource exhaustion), Service A's requests to B start timing out. Service A's threads/goroutines pile up waiting for B, exhausting A's connection pool and memory. Service A's health checks fail because it can't respond within timeout. Kubernetes kills A's pods (liveness probe failure). Services C-O that depend on A experience the same pattern. Within minutes, the entire dependency graph collapses.
Failure Amplifiers in Kubernetes
1. Retry storms: Each client retries 3x with exponential backoff, but 100 clients x 3 retries = 300 requests hitting an already-overwhelmed service 2. Liveness probe misconfiguration: Liveness probes that check downstream dependencies cause healthy pods to be killed when a dependency is down 3. Resource limits too tight: Under load, CPU throttling causes latency spikes, which cause more retries, which cause more CPU usage 4. HPA feedback loop: HPA scales up pods to handle retries, but new pods also retry, amplifying the storm 5. No pod priority: When resources are scarce, critical control plane pods (CoreDNS, ingress) get evicted
Prevention Architecture
- Circuit breakers: Istio DestinationRule with outlierDetection, or application-level (Resilience4j, Hystrix). Trip after 5 consecutive 5xx errors, wait 30s before half-open - Bulkhead pattern: Separate thread pools/connection pools per downstream dependency. If Service B is slow, only the B-calling thread pool is exhausted - Pod Priority and Preemption: Set PriorityClass for critical services (system-critical > platform > application > batch) - Proper liveness vs readiness: Liveness checks should ONLY verify the process is alive. Readiness checks can verify dependencies. Never check downstream health in liveness probes - Rate limiting and load shedding: Return HTTP 503 with Retry-After header when at capacity rather than accepting and queuing indefinitely