Production Issues

Symptom

Security patch window stalled at 38% complete. Cluster autoscaler reported nodes could not be drained because multiple pods violated disruption budgets.

Error Message

Cannot evict pod as it would violate the pod's disruption budget.

Root Cause

The PDBs were copied from a template that required 100% availability for services with only two replicas. During a normal upgrade this looked conservative, but during an emergency it prevented voluntary evictions. The team had confused desired safety with impossible safety: Kubernetes could not remove any pod without violating the declared rule.

Diagnosis Steps

1. 1.kubectl get pdb -A # Find budgets with zero allowed disruptions
2. 2.kubectl describe pdb checkout-api -n payments # Inspect currentHealthy and desiredHealthy
3. 3.kubectl get deploy checkout-api -n payments -o yaml # Check replica count and rollout settings

Solution

Temporarily scale the service to add capacity, wait for readiness, then drain nodes one at a time. Do not delete the PDB first unless the service owner accepts the risk; that removes the protection entirely instead of making it satisfiable.

Commands

kubectl scale deploy/checkout-api -n payments --replicas=4

kubectl wait --for=condition=available deploy/checkout-api -n payments --timeout=180s

kubectl drain ip-10-42-8-15 --ignore-daemonsets --delete-emptydir-data

Symptom

PagerDuty alert: 'payments-api pod restarts > 10 in 5m' at 02:14 UTC. Grafana showed memory usage hitting 512Mi limit, OOMKill events every 90 seconds, and HPA scaling from 4 to maxReplicas 20 within 3 minutes. Node CPU utilization spiked to 94% across all three nodes as 20 pods competed for CPU during startup. p99 latency jumped from 180ms to 12,400ms. Customer-facing error rate hit 38%.

Error Message

Last State: Terminated, Reason: OOMKilled, Exit Code: 137

Root Cause

The payments-api memory limit was set to 512Mi based on average usage, but a new feature released that afternoon added an in-memory cache for frequently accessed payment methods. Under load, this cache grew to 480Mi, leaving only 32Mi for the JVM heap overhead, garbage collection, and thread stacks. When a traffic spike hit at 02:10 UTC, the cache grew past the limit and the kernel OOM killer terminated the container. Kubernetes restarted the pod, which immediately received traffic from the Service endpoint before the cache warmed, causing it to allocate memory rapidly and get OOMKilled again within 90 seconds. The HPA saw CPU utilization spike during restart churn and scaled from 4 to 20 replicas, each of which also got OOMKilled. Now 20 pods were restarting simultaneously, consuming massive CPU during JVM startup. The three nodes hit 94% CPU utilization, which caused the existing healthy pods from other services to get CPU-throttled. The cascading effect spread to checkout-worker and inventory-service on the same nodes.

Diagnosis Steps

1. 1.kubectl get pods -n payments -o wide --sort-by='.status.containerStatuses[0].restartCount' | head -20
2. 2.kubectl describe pod payments-api-7d4f8b6c9-x2k4m -n payments | grep -A 5 'Last State'
3. 3.kubectl top pods -n payments --sort-by=memory
4. 4.kubectl get hpa payments-api-hpa -n payments -o yaml | grep -A 10 'status:'
5. 5.kubectl get events -n payments --sort-by='.lastTimestamp' | grep -E 'OOMKill|FailedScheduling|Unhealthy'
6. 6.kubectl top nodes

Solution

First, stop the cascade by setting HPA maxReplicas to the current healthy count. Run kubectl patch hpa payments-api-hpa -n payments -p '{"spec":{"maxReplicas":6}}' to prevent further scale-out. Then increase the memory limit to 1Gi to accommodate the new cache. Apply with kubectl set resources deployment/payments-api -n payments --limits=memory=1Gi --requests=memory=1Gi. Wait for pods to stabilize with kubectl rollout status deployment/payments-api -n payments. Once all pods are Running and Ready, gradually restore HPA maxReplicas. For the long-term fix, add a bounded LRU eviction policy to the in-memory cache with a maxSize of 256Mi. Profile memory under load using async-profiler or jcmd to establish the real memory ceiling. Set the memory limit to the p99 usage plus 20% headroom.

Commands

kubectl patch hpa payments-api-hpa -n payments -p '{"spec":{"maxReplicas":6}}'

kubectl set resources deployment/payments-api -n payments --limits=memory=1Gi --requests=memory=1Gi

kubectl rollout status deployment/payments-api -n payments --timeout=180s

kubectl get pods -n payments -l app=payments-api -o jsonpath='{range .items[*]}{.metadata.name} {.status.phase} {.status.containerStatuses[0].restartCount}{"\n"}{end}'

Symptom

Deployment rollout stuck at 50%. kubectl rollout status shows 'Waiting for deployment user-auth-service rollout to finish: 2 of 4 updated replicas are available.' PagerDuty alert: 'user-auth-service CrashLoopBackOff count > 5 in namespace identity.' New pods cycling through Running for 30s then Terminated with exit code 143 (SIGTERM from kubelet).

Error Message

Liveness probe failed: HTTP probe failed with statuscode: 503
Back-off restarting failed container user-auth-service in pod user-auth-service-5f8d9c7b4-qm3kn_identity

Root Cause

The user-auth-service is a Spring Boot Java application that takes 55-70 seconds to fully initialize. During startup, it runs Flyway database migrations, establishes connection pools to PostgreSQL and Redis, loads feature flag configurations from a remote service, and warms JIT compilation paths. The Deployment had a liveness probe with initialDelaySeconds: 30 and no startup probe. At second 31, the liveness probe sent its first HTTP GET to /health, which returned 503 because Spring Boot's actuator was not yet ready. After 3 consecutive failures (failureThreshold: 3, periodSeconds: 10), kubelet killed the container at second 61. The container restarted and the same sequence repeated, creating CrashLoopBackOff. The rolling update strategy (maxUnavailable: 25%) meant two old pods stayed running while two new pods kept crash-looping. The rollout never completed. Making it worse, the team had recently added a new Flyway migration that added 8 seconds to startup time, pushing total initialization from 48 seconds to 63 seconds, past the liveness probe's 60-second detection window. The issue did not appear in staging because the staging database was smaller and migrations ran in 2 seconds.

Diagnosis Steps

1. 1.kubectl rollout status deployment/user-auth-service -n identity
2. 2.kubectl get pods -n identity -l app=user-auth-service -o wide
3. 3.kubectl describe pod user-auth-service-5f8d9c7b4-qm3kn -n identity | grep -A 20 'Events:'
4. 4.kubectl logs user-auth-service-5f8d9c7b4-qm3kn -n identity --previous | tail -50
5. 5.kubectl get deployment user-auth-service -n identity -o jsonpath='{.spec.template.spec.containers[0].livenessProbe}'

Solution

Add a startup probe to the Deployment spec that gives the container enough time to initialize. Set startupProbe with httpGet on /health/started, periodSeconds: 10, and failureThreshold: 30, giving 300 seconds total. The liveness probe only activates after the startup probe succeeds. Apply the change with kubectl edit or a Helm values update. To unblock the current rollout immediately, temporarily increase initialDelaySeconds on the liveness probe to 120 while you add the startup probe properly. After applying, verify with kubectl rollout status and check that new pods reach Ready state. For the long-term fix, also optimize startup by running Flyway migrations in a separate init container, lazy-loading feature flags, and using Spring Boot's lazy initialization for non-critical beans.

Commands

kubectl rollout undo deployment/user-auth-service -n identity

kubectl patch deployment user-auth-service -n identity --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/startupProbe","value":{"httpGet":{"path":"/health/started","port":8080},"periodSeconds":10,"failureThreshold":30}}]'

kubectl rollout status deployment/user-auth-service -n identity --timeout=300s

kubectl get pods -n identity -l app=user-auth-service -o jsonpath='{range .items[*]}{.metadata.name} {.status.phase} {.status.containerStatuses[0].ready}{"\n"}{end}'

Symptom

Multiple teams reporting intermittent 5xx errors across unrelated services starting at 14:22 UTC. Datadog alert: 'DNS resolution failure rate > 5% cluster-wide.' Services returning errors like 'connection refused' to downstream dependencies. Not all requests fail, creating a confusing pattern where 30% of requests succeed and 70% timeout or return 503. CoreDNS pods showing 2-4 restarts in the last hour.

Error Message

dial tcp: lookup payments-db.payments.svc.cluster.local on 10.96.0.10:53: read udp 10.244.3.15:43210->10.96.0.10:53: i/o timeout
[ERROR] plugin/errors: 2 payments-db.payments.svc.cluster.local. A: read udp 10.96.0.10:53: i/o timeout

Root Cause

The CoreDNS deployment was running with the default resource limits of 170Mi memory inherited from the kubeadm default configuration. Over time, the cluster grew from 200 to 800 pods, each generating DNS queries for service discovery, external API calls, and health checks. CoreDNS maintains an in-memory cache of DNS responses, and with 800 pods each making 20-50 queries per minute, the cache plus goroutine overhead exceeded 170Mi. The kernel OOM killer terminated the CoreDNS container. Kubernetes restarted it, but during the 5-8 seconds of restart, all DNS queries from every pod in the cluster failed or timed out. Since most services resolve downstream dependencies via DNS for every request (database connections, API calls, cache lookups), the DNS outage propagated as connection errors across the entire service mesh. With only 2 CoreDNS replicas and both on the same node (no pod anti-affinity), a single node memory pressure event could kill both simultaneously. The intermittent pattern occurred because CoreDNS restarted quickly but the query queue during restart caused timeouts that propagated as retries, creating a thundering herd on the freshly restarted CoreDNS pod.

Diagnosis Steps

1. 1.kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
2. 2.kubectl top pods -n kube-system -l k8s-app=kube-dns
3. 3.kubectl describe pod coredns-5d78c9869d-abc12 -n kube-system | grep -A 5 'Last State'
4. 4.kubectl logs coredns-5d78c9869d-abc12 -n kube-system --previous | grep -i 'error\|oom\|timeout' | tail -30
5. 5.kubectl get events -n kube-system --sort-by='.lastTimestamp' | grep -i coredns
6. 6.kubectl exec -it debug-pod -- nslookup kubernetes.default.svc.cluster.local 10.96.0.10

Solution

Immediately increase CoreDNS memory limits and replica count. Patch the CoreDNS deployment to set memory requests to 256Mi and limits to 512Mi. Scale from 2 to 4 replicas to handle the current pod count. Add pod anti-affinity to spread CoreDNS pods across nodes so a single node failure cannot take out all DNS. Enable the CoreDNS autopath plugin to reduce the number of DNS queries per lookup from 5 (search domain iterations) to 1. Add NodeLocal DNSCache as a DaemonSet to cache DNS responses on each node, reducing load on CoreDNS by 80% and providing resilience during CoreDNS restarts. Configure the cluster-proportional-autoscaler to scale CoreDNS replicas based on node count.

Commands

kubectl set resources deployment/coredns -n kube-system --requests=memory=256Mi,cpu=200m --limits=memory=512Mi,cpu=500m

kubectl scale deployment/coredns -n kube-system --replicas=4

kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

kubectl exec -it debug-pod -- nslookup payments-db.payments.svc.cluster.local 10.96.0.10

Symptom

All kubectl apply and helm upgrade commands hang for 30 seconds then fail. No new pods are being created across any namespace. CI/CD pipelines timing out on deployment steps. Existing running pods are unaffected. PagerDuty alert: 'Deployment rollout failure rate 100% across all namespaces for 12 minutes.'

Error Message

Error from server (InternalError): error when creating "deployment.yaml": Internal error occurred: failed calling webhook "validate.security-scanner.internal": failed to call webhook: Post "https://security-scanner-webhook.security.svc:443/validate?timeout=10s": x509: certificate has expired or is not yet valid: current time 2026-01-15T03:14:00Z is after 2026-01-14T23:59:59Z

Root Cause

The cluster had a ValidatingWebhookConfiguration for a third-party security scanning tool (security-scanner) that intercepted all CREATE and UPDATE operations on Deployments, StatefulSets, and DaemonSets. The webhook's TLS certificate was issued with a 1-year validity period by the team that installed the tool 12 months ago. The certificate expired at midnight UTC on January 15th. When the API server attempted to send admission review requests to the webhook, the TLS handshake failed because the certificate was expired. The webhook was configured with failurePolicy: Fail, which meant that any failure to reach the webhook resulted in the admission request being rejected. This was a deliberate security choice to prevent bypassing the scanner, but it created a single point of failure for all deployments cluster-wide. The certificate renewal was not automated because the tool was installed manually via kubectl apply rather than through a Helm chart with cert-manager integration. No monitoring existed for certificate expiration on webhook endpoints. The issue was particularly insidious because it only affected mutating and creating operations, not reads, so kubectl get commands worked fine and the cluster appeared healthy.

Diagnosis Steps

1. 1.kubectl get validatingwebhookconfigurations -o name
2. 2.kubectl get validatingwebhookconfiguration security-scanner-webhook -o yaml | grep -A 5 'failurePolicy'
3. 3.kubectl get secret security-scanner-tls -n security -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates
4. 4.kubectl logs -n security deployment/security-scanner-webhook --tail=50
5. 5.kubectl get events --all-namespaces --sort-by='.lastTimestamp' | grep -i 'webhook\|certificate\|tls'

Solution

For immediate relief, change the webhook failurePolicy from Fail to Ignore to unblock deployments. Run kubectl edit validatingwebhookconfiguration security-scanner-webhook and change failurePolicy: Fail to failurePolicy: Ignore. This restores deployment capability but disables security scanning. Then renew the certificate: generate a new TLS cert using cert-manager or openssl, update the Secret, and restart the webhook deployment. Once the webhook is healthy, change failurePolicy back to Fail. For the permanent fix, install cert-manager and create a Certificate resource that auto-renews the webhook TLS certificate 30 days before expiry. Add a Prometheus alert on certmanager_certificate_expiration_timestamp_seconds to catch any renewal failures.

Commands

kubectl patch validatingwebhookconfiguration security-scanner-webhook --type='json' -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"}]'

kubectl get secret security-scanner-tls -n security -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

kubectl rollout restart deployment/security-scanner-webhook -n security

kubectl apply -f deployment.yaml

Symptom

Prometheus alert: 'KubeNodeDiskPressure firing on node ip-10-42-8-22 for 8 minutes.' Pods being evicted from the node with status 'Evicted' and reason 'The node was low on resource: ephemeral-storage.' Payments-api pods evicted first despite being tier-1, causing transaction failures. kubectl describe node shows 'DiskPressure: True'.

Error Message

Status: Failed, Reason: Evicted, Message: The node was low on resource: ephemeral-storage. Container checkout-worker was using 18476032Ki, which exceeds its request of 0.

Root Cause

The checkout-worker deployment had a verbose debug logging level that was accidentally promoted to production three weeks earlier in a config merge. Each pod was writing 200MB of structured JSON logs per hour to stdout, which the container runtime (containerd) captured to /var/log/containers/ on the node filesystem. With 6 checkout-worker pods on the node, total log output was 1.2GB per hour. The kubelet's containerLogMaxSize was set to the default 10Mi per file with containerLogMaxFiles: 5, giving 50Mi per container. However, the actual rotation was not keeping up because the high write rate filled the 10Mi file faster than logrotate could compress and rotate. Over three weeks, accumulated logs consumed 38GB of the node's 50GB root volume. When the filesystem hit 85% usage, kubelet set the DiskPressure condition and began evicting pods. Critically, the eviction order is based on pod QoS class and resource usage, not business priority. The payments-api pods were Burstable QoS and got evicted before the BestEffort dev-tool pods because the eviction algorithm considers pods exceeding their ephemeral-storage requests, and payments-api had no ephemeral-storage request set.

Diagnosis Steps

1. 1.kubectl describe node ip-10-42-8-22 | grep -A 10 'Conditions:'
2. 2.kubectl get pods --all-namespaces --field-selector spec.nodeName=ip-10-42-8-22,status.phase=Failed -o wide
3. 3.ssh ip-10-42-8-22 'df -h / && du -sh /var/log/containers/* | sort -rh | head -20'
4. 4.ssh ip-10-42-8-22 'du -sh /var/lib/containerd/io.containerd.grpc.v1.cri/containers/*/io.containerd.grpc.v1.cri/sandboxes/*/log*'
5. 5.kubectl logs checkout-worker-7f8d9a3b2-k4m2n -n orders --tail=5 | wc -c

Solution

First, reclaim disk space by identifying and cleaning the largest log files. SSH to the node and truncate the oversized log files with truncate -s 0 on the specific container log files. Do not delete them as containerd still holds file descriptors. Then fix the root cause by setting the checkout-worker log level back to INFO via ConfigMap update. Set ephemeral-storage requests and limits on all Deployments so the kubelet can enforce per-pod storage quotas and evict the right pods. Configure containerLogMaxSize: 50Mi and containerLogMaxFiles: 3 in the kubelet configuration to cap log retention at 150Mi per container. For long-term prevention, deploy a log shipping agent like Fluent Bit that forwards to a central log store and deletes local files after successful delivery.

Commands

ssh ip-10-42-8-22 'sudo find /var/log/containers -name "checkout-worker*" -size +100M -exec truncate -s 0 {} \;'

kubectl set env deployment/checkout-worker -n orders LOG_LEVEL=info

kubectl set resources deployment/checkout-worker -n orders --requests=ephemeral-storage=100Mi --limits=ephemeral-storage=500Mi

kubectl get pods --all-namespaces --field-selector spec.nodeName=ip-10-42-8-22 -o wide

Symptom

Grafana dashboard showing inventory-service replica count oscillating between 3 and 12 every 4-5 minutes in a sawtooth pattern. HPA events alternating between 'New size: 12; reason: cpu resource utilization above target' and 'New size: 3; reason: All metrics below target' in rapid succession. Customer complaints about intermittent slowness coinciding with scale-down phases.

Root Cause

The HPA was configured to scale on a custom Prometheus metric (http_requests_per_second) exposed through the Prometheus adapter. Prometheus was scraping the inventory-service pods every 60 seconds, and the Prometheus adapter was configured with a metricsRelistInterval of 30 seconds. The HPA evaluation period was the default 15 seconds. This created a timing mismatch: the HPA evaluated metrics every 15 seconds, but the underlying data was only refreshed every 60-90 seconds due to the scrape plus adapter pipeline lag. When traffic spiked, the stale metrics showed low values for one or two HPA cycles, then suddenly jumped to the actual high value, triggering aggressive scale-out to 12 replicas. The new replicas diluted the per-pod metric. On the next Prometheus scrape, the metric dropped well below the target because 12 pods were now sharing the load. The HPA then scaled down aggressively. But the scale-down removed capacity, traffic concentrated on fewer pods, the metric spiked again on the next scrape, and the cycle repeated. The default HPA stabilizationWindowSeconds for scale-down was only 300 seconds, which was not long enough to smooth out the oscillation caused by the 60-second metric lag.

Diagnosis Steps

1. 1.kubectl get hpa inventory-service-hpa -n catalog -o yaml
2. 2.kubectl describe hpa inventory-service-hpa -n catalog | grep -A 20 'Events:'
3. 3.kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1/namespaces/catalog/pods/*/http_requests_per_second' | jq .
4. 4.kubectl get pods -n catalog -l app=inventory-service --watch
5. 5.curl -s http://prometheus:9090/api/v1/query?query=scrape_duration_seconds{job="inventory-service"} | jq '.data.result[].value[1]'

Solution

Fix the HPA behavior configuration to add stabilization windows that account for metric lag. Set behavior.scaleDown.stabilizationWindowSeconds to 600 seconds to prevent rapid scale-down. Add a scaleDown policy that limits removal to 1 pod per 120 seconds, giving metrics time to reflect the impact of each scale-down step. For scale-up, set stabilizationWindowSeconds to 60 and allow scaling up by 100% to handle genuine traffic spikes quickly. Reduce the Prometheus scrape interval for the inventory-service to 15 seconds if cardinality permits. Configure the Prometheus adapter metricsRelistInterval to 15 seconds to match. Consider using a rate query over a 2-minute window instead of an instant value to smooth metric fluctuations.

Commands

kubectl patch hpa inventory-service-hpa -n catalog --type=merge -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":600,"policies":[{"type":"Pods","value":1,"periodSeconds":120}]},"scaleUp":{"stabilizationWindowSeconds":60,"policies":[{"type":"Percent","value":100,"periodSeconds":60}]}}}}'

kubectl describe hpa inventory-service-hpa -n catalog

kubectl get pods -n catalog -l app=inventory-service -w

Symptom

During a Kubernetes version upgrade from 1.29 to 1.30, all kubectl commands started returning 'Error from server: etcdserver: request timed out' or 'Error from server: etcdserver: leader changed'. Helm deployments across all namespaces failing. ArgoCD showing 'ComparisonError' on all applications. Prometheus alert: 'etcd_server_leader_changes_seen_total increased by 47 in 5 minutes.' API server request latency p99 exceeding 30 seconds.

Error Message

Error from server: etcdserver: request timed out
etcdserver: leader changed

Root Cause

The cluster ran a 3-node etcd cluster with the control plane upgrade proceeding one node at a time. During the upgrade of the second etcd member, the process involved stopping etcd, replacing the binary, and restarting. This briefly reduced the cluster to 2 members, which maintained quorum. However, the upgraded etcd node came back with a slightly different set of performance characteristics due to the new version's changed default values for heartbeat-interval (from 100ms to 150ms on the new binary) and election-timeout (from 1000ms to 1500ms). The mixed-version cluster had inconsistent timing expectations. The older leader sent heartbeats at 100ms intervals, but the new member expected them at 150ms intervals and occasionally interpreted normal network jitter as a missed heartbeat, triggering an election. Each election caused a brief write pause of 1-3 seconds. With the third node also upgrading, the cluster went through a window where all three members had different timing configurations, triggering 47 leader elections in 5 minutes. Each election paused writes, and the API server, which uses etcd as its sole datastore, returned timeout errors for any request involving writes or consistent reads. The situation was amplified by ArgoCD's reconciliation loop, which generated hundreds of read requests per second during the instability, further loading the struggling etcd cluster.

Diagnosis Steps

1. 1.etcdctl endpoint status --cluster -w table
2. 2.etcdctl endpoint health --cluster -w table
3. 3.kubectl get --raw /metrics | grep etcd_server_leader_changes_seen_total
4. 4.kubectl logs -n kube-system etcd-master-1 --tail=100 | grep -E 'leader|election|heartbeat|timeout'
5. 5.kubectl get --raw /metrics | grep 'etcd_disk_wal_fsync_duration_seconds|etcd_disk_backend_commit_duration_seconds'
6. 6.journalctl -u etcd --since '30 minutes ago' | grep -E 'elected|lost|changed'

Solution

First, pause all non-essential controllers that generate API server load. Scale down ArgoCD application-controller to 0 replicas temporarily with kubectl scale deployment argocd-application-controller -n argocd --replicas=0. Then ensure all etcd members are running the same version by completing the upgrade of the remaining node. If the storm continues after version alignment, explicitly set consistent heartbeat and election timeout values across all members by updating the etcd manifest in /etc/kubernetes/manifests/etcd.yaml on each control plane node. Set --heartbeat-interval=100 and --election-timeout=1000 consistently. Monitor etcd_server_leader_changes_seen_total until it stabilizes. Once etcd is stable, scale ArgoCD back up. Verify cluster health with etcdctl endpoint health --cluster.

Commands

kubectl scale deployment argocd-application-controller -n argocd --replicas=0

etcdctl endpoint status --cluster -w table

etcdctl endpoint health --cluster -w table

kubectl scale deployment argocd-application-controller -n argocd --replicas=1

Symptom

After migrating notification-service from the shared namespace to a dedicated notifications namespace, the service returns 504 Gateway Timeout on all requests. Health checks pass (pod is Running and Ready) but no traffic reaches the application. Other services in the old namespace report connection timeout errors when calling notification-service. PagerDuty alert: 'notification-service error rate 100% for 8 minutes.'

Error Message

upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111
curl: (28) Failed to connect to notification-service.notifications.svc.cluster.local port 8080 after 5001 ms: Connection timed out

Root Cause

The notifications namespace was provisioned with the team's standard default-deny NetworkPolicy that blocks all ingress and egress traffic. The team created allow policies for the notification-service to accept traffic from the ingress controller and to reach the email-gateway on port 587. However, the ingress allow policy referenced the source pods using a namespaceSelector with matchLabels: namespace: shared, which was the label on the old shared namespace. The notification-service's upstream callers (order-processing-service in the orders namespace and user-auth-service in the identity namespace) were not included in any allow policy because the original NetworkPolicies only permitted traffic within the shared namespace. After migration, the notification-service could receive traffic from ingress but not from internal service-to-service callers. The readiness probe passed because it was an HTTP check to /health that did not depend on receiving actual application traffic. The service appeared healthy in Kubernetes but was unreachable from its consumers. The team spent 20 minutes checking DNS, Service endpoints, and pod logs before realizing the issue was at the network layer. The pod logs showed zero incoming requests, which was the critical clue that traffic was being dropped before reaching the application.

Diagnosis Steps

1. 1.kubectl get networkpolicy -n notifications -o yaml
2. 2.kubectl describe networkpolicy allow-ingress-to-notification -n notifications
3. 3.kubectl exec -it order-processing-7d4f8-abc12 -n orders -- curl -v --connect-timeout 5 http://notification-service.notifications.svc.cluster.local:8080/health
4. 4.kubectl exec -it debug-pod -n notifications -- tcpdump -i eth0 port 8080 -c 10
5. 5.kubectl get pods -n notifications -l app=notification-service -o wide
6. 6.kubectl get endpoints notification-service -n notifications

Solution

Update the NetworkPolicy in the notifications namespace to allow ingress from all namespaces that contain caller services. Create specific allow policies for each caller namespace using namespaceSelector with the correct labels. First, verify which namespaces need access by checking the service dependency map. Then apply new NetworkPolicies that permit ingress from the orders namespace (for order-processing-service) and the identity namespace (for user-auth-service) on port 8080. Test each policy by running curl from a pod in the caller namespace. For the permanent fix, adopt a label-based approach where caller services are labeled with a role like notification-consumer and the NetworkPolicy selects on that label regardless of namespace, making it resilient to future namespace migrations.

Commands

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-internal-callers
  namespace: notifications
spec:
  podSelector:
    matchLabels:
      app: notification-service
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: orders
      podSelector:
        matchLabels:
          app: order-processing-service
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: identity
      podSelector:
        matchLabels:
          app: user-auth-service
    ports:
    - protocol: TCP
      port: 8080
EOF

kubectl exec -it order-processing-7d4f8-abc12 -n orders -- curl -s -o /dev/null -w '%{http_code}' http://notification-service.notifications.svc.cluster.local:8080/health

kubectl get networkpolicy -n notifications

Symptom

P95 latency on /v1/charges jumped from 180ms to 4.2s; HPA scaled payments-api from 12 to 48 pods but error rate climbed to 23%; Grafana alert KubePodOOMKilled fired across 31 pods in 6 minutes; downstream order-service began timing out.

Error Message

OOMKilled: container payments-api exceeded memory limit; exit code 137; last state: Terminated reason OOMKilled

Root Cause

The payments-api deployment was sized for baseline traffic with a 512Mi memory limit and JVM heap set to -Xmx384m, leaving insufficient headroom for GC spikes and off-heap buffers during authorization batching. When Black Friday traffic arrived, concurrent Stripe webhook bursts caused the in-memory deduplication cache (Caffeine, unbounded per design doc) to grow from ~40MB to over 900MB per pod within three minutes. HPA reacted correctly on CPU but ignored memory because no memory metric was configured, so new pods inherited the same undersized limits and were OOMKilled within 90 seconds of becoming Ready. Each OOMKill triggered Kubernetes to restart the container, which replayed queued webhooks from Redis, amplifying memory pressure in surviving pods. The cascade was worsened by readiness probes passing while the JVM was still warming, sending traffic to pods that had not yet stabilized heap usage. Node memory pressure was not the issue—per-pod cgroup limits were the bottleneck—but the rapid churn caused kube-proxy conntrack table pressure on two worker nodes, adding secondary latency. No PodDisruptionBudget existed, so rolling updates during the incident compounded restarts. Root cause is a combination of unbounded in-process cache, memory limits set 18 months ago for a smaller payload schema, and HPA scoped to CPU-only metrics.

Diagnosis Steps

1. 1.Filter Prometheus for container_memory_working_set_bytes and kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} on namespace payments-prod, correlate timestamps with traffic ingress rate from nginx-ingress.
2. 2.Run kubectl describe pod -n payments-prod -l app=payments-api and inspect Last State / Limits for memory; check JVM heap dumps in /tmp if ephemeral storage allows.
3. 3.Review HPA spec (kubectl get hpa payments-api -o yaml) for metrics—confirm absence of memory utilization—and compare pod count vs. nginx upstream healthy backends.
4. 4.Inspect application logs for webhook replay loops and cache size metrics; validate whether readiness probe fires before JVM warm-up completes.

Solution

Immediately raise memory limits to 2Gi and requests to 1Gi via emergency patch, and cap the Caffeine cache with maximumSize=50000 and expireAfterWrite=5m deployed through a hotfix image tag. Scale HPA maxReplicas temporarily to 64 but add a second metric on memory utilization at 75% average to prevent blind CPU-only scaling. Drain webhook replay by pausing the Stripe webhook consumer Deployment for 10 minutes while pods stabilize, then resume with rate limiting at 200 rps per pod. For JVM tuning, set -Xmx1280m -XX:MaxMetaspaceSize=256m and enable G1GC with -XX:+UseStringDeduplication. After stabilization, load-test at 2x peak with memory profiling (async-profiler) to validate headroom. Longer term, move deduplication to Redis with TTL and shrink per-pod state. Document new SLO-based sizing: limit = 1.5x p99 working set from 30-minute load test. Roll out via canary (10% traffic) before full promotion. Post-incident, add alert KubePodOOMKilled with runbook link and page on-call if more than 3 OOMKills in 5 minutes in payments-prod.

Commands

kubectl top pods -n payments-prod -l app=payments-api --containers

kubectl describe pod -n payments-prod -l app=payments-api | grep -A5 'Last State'

kubectl patch deployment payments-api -n payments-prod -p '{"spec":{"template":{"spec":{"containers":[{"name":"payments-api","resources":{"limits":{"memory":"2Gi"},"requests":{"memory":"1Gi"}}}]}}}}'

kubectl get hpa payments-api -n payments-prod -o yaml

kubectl logs -n payments-prod -l app=payments-api --previous --tail=200 | grep -i oom

Symptom

kubectl drain hung for 38 minutes on ip-10-0-44-12; alert NodeDrainBlocked fired; cluster-autoscaler could not terminate underutilized node group; 14 pods remained Pending with event 'cannot evict due to PodDisruptionBudget restrictions'.

Error Message

error when evicting pods: cannot evict pod "auth-service-7f9c8-abcde" - disruption budget auth-service-pdb expects 3 healthy pods but only 2 available

Root Cause

Platform team initiated a scheduled EKS minor version upgrade requiring sequential drain of the m5.xlarge node group. The auth-service PodDisruptionBudget was configured with minAvailable: 3 while the Deployment replica count was also 3, meaning zero voluntary disruptions were permitted unless an extra replica existed. During the drain, one auth pod was already CrashLooping due to an unrelated bad ConfigMap mount, leaving only 2 healthy pods—below the PDB floor. Kubernetes correctly refused eviction, blocking the drain and stalling the entire node rotation. Compounding the issue, cluster-autoscaler had scaled the node group down overnight to exactly three nodes per AZ for cost savings, leaving no spare capacity to schedule surplus replicas during disruption. The PDB had been copied from a template intended for five-replica services without adjusting minAvailable relative to replica count. Terraform module drift meant staging had maxUnavailable: 1 but production still had minAvailable: 3 on a three-replica Deployment. SRE runbooks assumed PDBs used maxUnavailable semantics. No pre-drain validation job existed to simulate evictions. The incident extended the maintenance window, delayed security patches on three nodes, and caused Pending pods on other workloads when the blocked node retained tainted workloads that could not reschedule.

Diagnosis Steps

1. 1.Run kubectl get pdb -A and compare minAvailable/maxUnavailable against Deployment .spec.replicas for each protected workload.
2. 2.Inspect kubectl describe node ip-10-0-44-12 for cordon/drain conditions and kubectl get events --field-selector involvedObject.name=ip-10-0-44-12.
3. 3.Identify unhealthy pods lowering available count: kubectl get pods -n identity -l app=auth-service -o wide and check READY column.
4. 4.Simulate eviction safely: kubectl drain --dry-run=server --ignore-daemonsets --delete-emptydir-data ip-10-0-44-12.

Solution

First fix the unhealthy auth-service pod by correcting the ConfigMap volume key so three replicas become Ready—this alone may unblock drain. If maintenance urgency is higher, temporarily patch PDB to maxUnavailable: 1 (or minAvailable: 2) with change ticket CHG-8842, documenting rollback. Scale auth-service to 4 replicas before drain so minAvailable: 3 allows one eviction. Resume drain with kubectl drain ip-10-0-44-12 --ignore-daemonsets --delete-emptydir-data --grace-period=120. After node group upgrade completes, revert PDB to policy matching SLO: for R=3 use maxUnavailable: 1 not minAvailable: 3. Add pre-drain CI check in cluster-lifecycle pipeline validating PDB ≤ replicas−1 for minAvailable configs. Update Terraform module default to maxUnavailable: 1 with override guard requiring minAvailable < replicas. Train on-call that PDB blocking is often a symptom of unhealthy pods—always check Ready count before forcing PDB patch.

Commands

kubectl get pdb -n identity -o custom-columns=NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,ALLOWED:.status.disruptionsAllowed

kubectl get deploy auth-service -n identity

kubectl scale deploy auth-service -n identity --replicas=4

kubectl patch pdb auth-service-pdb -n identity -p '{"spec":{"maxUnavailable":1,"minAvailable":null}}'

kubectl drain ip-10-0-44-12 --ignore-daemonsets --delete-emptydir-data --grace-period=120

Symptom

postgres-primary-0 remained Pending 22 minutes after failover attempt; alert PersistentVolumeClaimStuckTerminating and VolumeMultiAttachError fired; read/write IO on analytics-dashboard dropped to zero; Velero backup job failed with volume mount timeout.

Error Message

Multi-Attach error for volume "pvc-a8f3c2e1-4b9d-4e2a-9c1c-0d4e5f6a7b8c" Volume is already used by pod(s) postgres-primary-0, postgres-primary-0

Root Cause

The stateful Postgres cluster used a ReadWriteOnce EBS volume (gp3, 500Gi) in us-east-1a. A manual kubectl delete pod during incident response left the old pod object in Terminating because the kubelet on a failed worker node (NotReady for 11 minutes) could not release the volume attachment. Simultaneously, the StatefulSet controller created a replacement pod that scheduled onto a healthy node in the same AZ, attempting to attach the same volume while the stale attachment record still pointed to the dead node's EC2 instance ID. AWS EBS multi-attach is disabled for RWO, so the second attach failed with MultiAttachError. The PVC entered Terminating because an operator ran kubectl delete pvc during confusion—finalizers kubernetes.io/pvc-protection and external-provisioner remained, but the volume attachment CRD on the dead node blocked cleanup. EBS CSI driver version 1.19 had a known race when NodePublishSecretRef was unset and force-detach was not invoked within SLA. No automatic volume attachment monitoring existed. The database SLO breach occurred because primary could not start, replicas were read-only, and connection pool exhaustion propagated to 14 microservices using JDBC URLs pointing at the primary Service.

Diagnosis Steps

1. 1.kubectl get pvc,pv,va -n data-platform | grep postgres-primary; note Phase Terminating and VolumeAttachment nodeName.
2. 2.Check pod events: kubectl describe pod postgres-primary-0 -n data-platform for FailedAttachVolume and MultiAttachError messages.
3. 3.Verify node health: kubectl get node <stale-node> -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'.
4. 4.In AWS console or CLI, describe-volume Attachments for volume vol-xxx and identify stale instance attachment.

Solution

Do not delete PVCs during failover—restore from snapshot if data loss risk exists. Cordon the NotReady node and confirm pod is gone: kubectl delete pod postgres-primary-0 -n data-platform --force --grace-period=0 if kubelet is unresponsive. Remove finalizers only after backup validation: kubectl patch pvc data-postgres-primary-0 -n data-platform -p '{"metadata":{"finalizers":null}}' if stuck Terminating without active consumers. Force-detach stale EBS volume via AWS CLI aws ec2 detach-volume --volume-id vol-xxx --force, then delete VolumeAttachment: kubectl delete volumeattachment <name>. Recreate pod or wait for StatefulSet to retry attach. Upgrade EBS CSI driver to 1.28+ with attachment lease improvements. Implement runbook using kubectl wait and VolumeAttachment status before pod delete. For HA, migrate to RDS or enable Postgres operator with synchronous replica promotion instead of RWO pod reschedule. Post-incident, disable manual pvc delete in prod RBAC except break-glass role.

Commands

kubectl get pvc -n data-platform

kubectl describe pod postgres-primary-0 -n data-platform

kubectl get volumeattachment | grep pvc-a8f3c2e1

kubectl delete pod postgres-primary-0 -n data-platform --force --grace-period=0

aws ec2 describe-volumes --volume-ids vol-0abc123def456789

aws ec2 detach-volume --volume-id vol-0abc123def456789 --force

Symptom

Intermittent 'i/o timeout' and 'no such host' errors across 40+ Deployments; alert CoreDNSHighLatency p99 > 800ms; kube-dns autoscaler lagging; new pod startup time increased from 12s to 95s; external-dns sync delays.

Error Message

dial tcp: lookup api.internal.svc.cluster.local on 10.96.0.10:53: read udp 10.244.88.12:45678->10.96.0.10:53: i/o timeout

Root Cause

CoreDNS ran as a two-replica Deployment on overloaded t3.medium nodes shared with application workloads. A misconfigured Java service in namespace batch-jobs opened 500 parallel JDBC connections each performing reverse DNS lookups on every pool refresh (every 30s), generating ~16,000 PTR and A record queries per second cluster-wide. CoreDNS cache TTL for negative responses was 30s, amplifying repeated NXDOMAIN lookups from a typoed internal hostname batch-etl.svc.cluster.local in CronJob manifests. The cluster used NodeLocal DNSCache daemonset but it was disabled in prod after a prior upgrade left hostNetwork conflicts undocumented. CPU throttling on CoreDNS pods (limit 100m, actual need ~600m under spike) caused request drops. Prometheus metrics showed forward_requests_total exceeding 45k/min with plugin errors in log: [ERROR] plugin/errors: 2 quota.prod.svc.cluster.local. A/AAAA: read tcp 10.96.0.10:53->10.244.1.5:xxxxx: i/o timeout. HPA for CoreDNS did not exist—only static replicas=2 from 2022 baseline for 80-node cluster now at 240 nodes. The saturation presented as application failures rather than DNS pod CrashLoop, delaying root-cause identification.

Diagnosis Steps

1. 1.kubectl top pod -n kube-system -l k8s-app=kube-dns; check CPU throttling via kubectl describe pod coredns-xxx.
2. 2.Port-forward CoreDNS metrics :9153 and query coredns_dns_request_duration_seconds_bucket p99 in Prometheus.
3. 3.Enable CoreDNS log plugin temporarily or inspect logs for high-cardinality qnames; identify top query sources with dnsperf or packet capture on node.
4. 4.Audit CronJobs and services for malformed DNS names and excessive lookup patterns in application connection pools.

Solution

Emergency scale CoreDNS to 8 replicas and raise CPU limits to 500m: kubectl -n kube-system scale deployment coredns --replicas=8; patch resource limits via kustomize overlay. Re-enable NodeLocal DNSCache v1.22 with proper anti-affinity on kube-system. Fix batch-etl CronJob SERVICE_HOST env to batch-etl.batch-jobs.svc.cluster.local. Disable reverse DNS in JDBC URL (?dnsSkipReverseLookup=true or use IP). Deploy CoreDNS HPA on metrics-server custom metrics from coredns_dns_requests_total rate. Increase cache block for cluster.local to 300s where safe. Roll out topology spread constraints so CoreDNS pods land on dedicated system node pool (t3.large, taint system=true). After mitigation, run dnsperf from multiple namespaces to validate p99 < 50ms. Document noisy-neighbor query budgets per namespace via NetworkPolicy logging optional. Long-term: split internal zones, consider external authoritative DNS for large service meshes.

Commands

kubectl top pod -n kube-system -l k8s-app=kube-dns

kubectl -n kube-system scale deployment coredns --replicas=8

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100

kubectl run dns-test --image=busybox:1.36 --rm -it -- nslookup kubernetes.default.svc.cluster.local

dig @10.96.0.10 api.internal.svc.cluster.local +stats

Symptom

Browser and mobile clients showed ERR_CERT_DATE_INVALID; alert SSLCertificateExpiringSoon missed due to wrong label selector; nginx-ingress logs flooded with SSL_do_handshake failed; conversion rate dropped 67% in 15 minutes; StatusCake external check failed.

Error Message

2026-06-27T08:14:22Z [error] 28#28: *1048573 SSL_do_handshake() failed (SSL: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: certificate has expired) while SSL handshaking, client: 203.0.113.44

Root Cause

Production checkout traffic terminates at nginx-ingress-controller v1.9.4 using a cert-manager Certificate resource checkout-tls referencing Let's Encrypt ACME HTTP-01 solver. Six weeks earlier, a platform migration moved the storefront Ingress from ingressClassName nginx-public to nginx-internal for WAF integration, but the HTTP-01 challenge Ingress still pointed at nginx-public. cert-manager could not complete renewal; Certificate status showed Ready=True while the stored Secret checkout-tls-secret retained an expired cert because status was not reconciled after manual Secret restore from backup during DR drill. The monitoring rule SSLCertificateExpiringSoon scoped to secret labels cert-manager.io/certificate-name but the restored Secret lacked those labels, so Prometheus ssl_exporter never scraped expiry. Additionally, cert-manager v1.12 renewal window (default 30 days before expiry) failed silently—Challenge resources remained pending with reason 'wrong challenge type' after ingress class change. No alert fired on nginx SSL handshake error rate. Mobile apps using certificate pinning rejected connections entirely. The expired cert was the default tls entry on the Ingress spec, affecting checkout.example.com and www.checkout.example.com SANs. Root cause chain: DR restore without labels + ingress class drift breaking ACME + absent handshake error alerting.

Diagnosis Steps

1. 1.kubectl get certificate,challenge,order -n storefront | grep checkout; describe Certificate checkout-tls for NotReady conditions.
2. 2.openssl s_client -connect checkout.example.com:443 -servername checkout.example.com 2>/dev/null | openssl x509 -noout -dates
3. 3.Compare Secret checkout-tls-secret tls.crt expiry vs Certificate .status.notAfter.
4. 4.Check nginx-ingress logs for SSL_do_handshake failures and verify ingressClassName on challenge solver Ingress.

Solution

Immediate mitigation: import emergency cert from DigiCert wildcard *.example.com (valid 90 days) into Secret checkout-tls-secret via kubectl create secret tls --dry-run=client -o yaml | kubectl apply -f -. Reload nginx: kubectl rollout restart deployment ingress-nginx-controller -n ingress-nginx. Fix cert-manager: patch Certificate to use DNS-01 with Route53 solver eliminating HTTP-01 ingress class dependency. Reconcile labels on Secret: cert-manager.io/certificate-name=checkout-tls. Delete stale Challenge/Order objects and force renewal: cmctl renew checkout-tls -n storefront. Update Prometheus rules to alert on ssl_cert_not_after{secret=~".*tls.*"} < 14 days regardless of labels, and nginx ingress controller error rate ssl_handshake_failed. Run cert-manager upgrade to v1.14. Document DR restore checklist requiring cert-manager reconcile annotation cert-manager.io/issue-temporary-certificate. Post-fix, validate with ssllabs scan and mobile app smoke test.

Commands

kubectl get certificate -n storefront

kubectl describe certificate checkout-tls -n storefront

echo | openssl s_client -connect checkout.example.com:443 -servername checkout.example.com 2>/dev/null | openssl x509 -noout -dates

kubectl get secret checkout-tls-secret -n storefront -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

cmctl renew checkout-tls -n storefront

kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=50 | grep SSL

Symptom

api-gateway Deployment stuck at 20/20 replicas for 90 minutes; alert HPAMaxReplicasReached fired; CPU sustained 89-94%; queue depth in Kafka consumer lag 2.1M; 503 responses 18% from ingress; no new pods created despite high utilization.

Error Message

Warning FailedScale: horizontalpodautoscaler/api-gateway-hpa failed to scale up: proposed replica count 21 would exceed max replicas limit of 20

Root Cause

The api-gateway HPA was created during initial launch with maxReplicas: 20 when peak traffic was 8k RPS. Product launch and mobile app v3 shifted traffic to 34k RPS sustained. HPA targeted CPU utilization 70% with requests.cpu=500m and limits.cpu=1000m; at 20 pods aggregate CPU could not drop below threshold because each pod spent 40% time in GC and JWT validation crypto (RSA4096 per request without cache). maxReplicas acted as a hard ceiling—Kubernetes HPA controller logs proposed 21 replicas and rejected scale. Cluster had ample capacity: 40 idle cores across three node pools, but scheduling was never attempted. No Vertical Pod Autoscaler recommendation was applied; actual per-pod need was ~1.8 cores under load. Kafka ingress path shared the same Deployment, so consumer lag grew when HTTP threads starved. Cost optimization quarter had lowered maxReplicas from 30 to 20 without capacity review. No custom metrics (requests per second from Prometheus adapter) were wired—CPU-only HPA is a poor proxy for IO-bound gateway.work. Alert HPAMaxReplicasReached existed but routed to low-priority Slack until pager threshold exceeded 30 minutes—this incident hit 90 minutes before page.

Diagnosis Steps

1. 1.kubectl get hpa api-gateway-hpa -n edge -o yaml; note maxReplicas and currentMetrics.
2. 2.kubectl describe hpa api-gateway-hpa -n edge for FailedScale events and proposed vs. accepted replica counts.
3. 3.Check cluster capacity: kubectl describe nodes | grep -A5 'Allocated resources' and confirm no ResourceQuota blocking.
4. 4.Profile pod CPU vs. request: kubectl top pods -n edge -l app=api-gateway; compare to ingress RPS from Prometheus.

Solution

Immediate: patch maxReplicas to 50 and raise CPU limit to 2000m with request 1000m: kubectl patch hpa api-gateway-hpa -n edge -p '{"spec":{"maxReplicas":50}}'. Deploy JWT validation cache (60s TTL for non-revoked tokens) hotfix to cut CPU 35%. Scale manually to 35 while HPA catches up: kubectl scale deploy api-gateway -n edge --replicas=35. Add Prometheus adapter metric nginx_ingress_controller_requests_rate for RPS-based scaling target 8000 per pod. Restore maxReplicas to 60 after load test sign-off. Enable VPA in recommendation mode and apply right-sizing next maintenance. Fix alert routing: HPAMaxReplicasReached pages after 10 minutes in edge namespace. Document capacity model linking RPS to replicas with headroom 1.4x. Post-incident review with finance on cost vs. maxReplicas policy requiring SRE approval for decreases.

Commands

kubectl get hpa api-gateway-hpa -n edge

kubectl describe hpa api-gateway-hpa -n edge

kubectl patch hpa api-gateway-hpa -n edge -p '{"spec":{"maxReplicas":50}}'

kubectl scale deploy api-gateway -n edge --replicas=35

kubectl top pods -n edge -l app=api-gateway

Symptom

recommendation-engine pods restarted 847 times in 4 hours; alert KubePodCrashLooping fired; readiness flapped; ML model load takes 90-120s but liveness failed at 30s; downstream homepage feed returned empty shelves.

Error Message

Liveness probe failed: HTTP probe failed with statuscode: 503; container recommendation-engine failed liveness probe, will be restarted; Back-off restarting failed container

Root Cause

A platform-wide Helm chart bump standardized liveness probes to httpGet path /healthz port 8080 initialDelaySeconds: 10 periodSeconds: 10 failureThreshold: 3 for all Java and Python services. recommendation-engine loads a 2.3GB ONNX model from S3 init container plus warms embedding cache from Redis—startup routinely exceeds 100 seconds on m5.large nodes with EBS gp3 throughput capped at 125 MB/s. The liveness probe hit /healthz which returns 503 until modelLoadComplete flag is true, correctly reflecting not-ready state but incorrectly used for liveness. Kubernetes killed the container at ~40s (10 + 3×10), restarting before model load finished, creating CrashLoopBackOff with exponential backoff up to 5 minutes. Readiness probe shared the same endpoint but had initialDelaySeconds: 15—also insufficient. No startupProbe existed. The Deployment rolled out during business hours via Argo CD auto-sync without service-specific override. Previous chart allowed startupProbe with failureThreshold: 30. Metrics showed container_start_time_seconds never exceeding 45s before SIGKILL. GPU node pool was not involved—CPU inference only. Incident impact: empty recommendations on homepage, 12% session bounce rate increase.

Diagnosis Steps

1. 1.kubectl describe pod -n ml-serving -l app=recommendation-engine | grep -A10 Liveness
2. 2.kubectl logs -n ml-serving -l app=recommendation-engine --previous | tail -50 for model load progress vs. kill timestamp
3. 3.Compare Deployment probe spec to documented startup SLA (120s) in service README.
4. 4.Check Argo CD diff for Helm chart version bump that removed startupProbe block.

Solution

Patch Deployment immediately with startupProbe: httpGet /healthz:8080, failureThreshold: 30, periodSeconds: 10 (300s budget), and liveness initialDelaySeconds: 120 only after startup succeeds. Alternatively split /healthz (liveness: always 200 if process up) from /ready (model loaded). Roll out: kubectl patch or Argo CD sync with service-specific values.yaml override. Kill CrashLoop pods to reset backoff: kubectl delete pods -n ml-serving -l app=recommendation-engine. Revert platform chart default to require startupProbe for any init >60s documented in service catalog. Add alert on restart count rate >5/hour per pod with label ml-serving. CI contract test: deploy to staging with artificial 90s init sleep, assert pod becomes Ready without restart. Document probe guidelines in platform standards: never reuse readiness endpoint semantics on liveness for batch-loaded services.

Commands

kubectl get pods -n ml-serving -l app=recommendation-engine

kubectl describe pod -n ml-serving -l app=recommendation-engine | grep -A15 'Liveness\|Startup\|Readiness'

kubectl logs -n ml-serving -l app=recommendation-engine --previous --tail=100

kubectl patch deployment recommendation-engine -n ml-serving --type=strategic -p '{"spec":{"template":{"spec":{"containers":[{"name":"recommendation-engine","startupProbe":{"httpGet":{"path":"/healthz","port":8080},"failureThreshold":30,"periodSeconds":10}}]}}}}'

Symptom

Seven worker nodes flipped NotReady within 8 minutes; alert NodeNotReadyCritical fired; 412 pods Evicted with reason NodeNotReady; new pods Pending with FailedScheduling insufficient cpu; AWS EC2 status checks failed on affected instances.

Error Message

The node status is NotReady; pod eviction manager evicting pod "inventory-worker-6d4f9-xyz" due to node NotReady; 0/14 nodes are available: 7 node(s) had untolerated taint {node.kubernetes.io/unreachable: }

Root Cause

A botched CNI plugin upgrade (aws-node DaemonSet v1.16.0 → v1.18.2) rolled out simultaneously on all nodes via maxUnavailable: 100% override in a hotfix PR merged Friday 17:00 UTC. The new aws-node version introduced a race on AL2023 kernels where iptables-nft rules were programmed before ENI secondary IPs attached, causing kubelet to fail NodeReady heartbeat to API server while the EC2 instance remained running. Nodes received node.kubernetes.io/unreachable taint after 300s default pod eviction timeout. kube-controller-manager evicted workloads aggressively—412 pods—many with local cache state lost. Replacement pods could not schedule because remaining healthy nodes lacked CPU after absorbing evicted workloads; cluster-autoscaler hit max node group size 14. Some evicted pods had PVCs pinned to AZs of dead nodes. The incident resembled application failure but metrics showed apiserver node lease updates stopped while EC2 SystemStatus passed. Root cause: unsafe DaemonSet rollout strategy plus missing PDB coverage on stateless workers causing thundering herd reschedule. No canary node pool existed for CNI changes.

Diagnosis Steps

1. 1.kubectl get nodes -o wide; correlate NotReady timestamps with aws-node DaemonSet rollout revision.
2. 2.Inspect kubelet logs on affected node via SSM: journalctl -u kubelet | grep -i cni
3. 3.kubectl get events -A --field-selector reason=Evicted | head -20
4. 4.Check aws-node pod logs on NotReady node: kubectl logs -n kube-system aws-node-xxxxx -c aws-node

Solution

Rollback aws-node DaemonSet to v1.16.0: kubectl rollout undo daemonset aws-node -n kube-system. Uncordon healthy nodes after CNI stabilizes. For stuck NotReady nodes, terminate EC2 instances and let ASG replace—do not manual kubelet restart on all at once. Temporarily raise cluster-autoscaler max nodes +5 for burst capacity. Restore evicted workloads by deleting Evicted pod objects and letting controllers recreate; prioritize tier-1 namespaces via sync-wave in Argo CD. Tune eviction-hard only after stability—not during incident. Post-fix: enforce maxUnavailable: 1 or 10% on all kube-system DaemonSets; canary one node pool before prod-wide CNI rollout. Add alert on node Ready condition transition rate >3 in 5 minutes. Document break-glass: pause DaemonSet rollout on first NotReady.

Commands

kubectl get nodes

kubectl rollout history daemonset/aws-node -n kube-system

kubectl rollout undo daemonset/aws-node -n kube-system

kubectl get pods -A --field-selector status.phase=Failed | grep Evicted

kubectl describe node <notready-node> | grep -A20 Conditions

Symptom

Stripe webhook delivery failures 94%; alert ExternalWebhookFailureRate critical; application logs show UnknownHostException; dig from debug pod returns NXDOMAIN for hooks.payments.internal; only production affected.

Error Message

java.net.UnknownHostException: hooks.payments.internal: Name or service not known; Caused by: io.netty.resolver.dns.DnsNameResolverTimeoutException: query timed out after 5000 milliseconds; NXDOMAIN for hooks.payments.internal.cluster.local

Root Cause

Platform team migrated webhook ingress from in-cluster nginx to external Apigee gateway. A Kubernetes ExternalName Service hooks-gateway in namespace payments-prod was created pointing to externalName: apigee-gateway.example.com but application ConfigMap retained WEBHOOK_CALLBACK_HOST=hooks.payments.internal without the cluster.local suffix. CoreDNS search list appended .cluster.local, resolving hooks.payments.internal.cluster.local—which matched no Service or ExternalName because the Service name was hooks-gateway not hooks.payments.internal. Shorter name hooks.payments.internal failed upstream forward to VPC DNS (Route53 private zone) where the record existed as CNAME to Apigee, but Java resolver tried cluster search first, got NXDOMAIN from CoreDNS with negative caching 30s, and never reached Route53 for some resolver paths. Staging worked because developers used FQDN hooks.payments.internal.prod.example.com in local env files. The misconfiguration shipped via Kustomize overlay typo: configMapGenerator literal WEBHOOK_CALLBACK_HOST copied from deprecated internal DNS convention doc. No external-dns annotation involved—pure in-cluster resolution failure. Stripe retried webhooks for 72 hours but DLQ filled after 6 hours.

Diagnosis Steps

1. 1.kubectl run dnsdebug --image=busybox:1.36 --rm -it -- nslookup hooks.payments.internal and nslookup hooks.payments.internal.cluster.local from payments-prod.
2. 2.kubectl get svc -n payments-prod hooks-gateway -o yaml; verify ExternalName target.
3. 3.Compare ConfigMap WEBHOOK_CALLBACK_HOST in prod vs staging: kubectl get cm payment-config -n payments-prod -o yaml.
4. 4.Check CoreDNS logs for NXDOMAIN on hooks.payments.internal.cluster.local.

Solution

Immediate fix: patch ConfigMap to FQDN apigee-gateway.example.com or in-cluster Service DNS hooks-gateway.payments-prod.svc.cluster.local and rollout restart payment-webhook Deployment. Alternative: create Service alias hooks.payments.internal as ExternalName to apigee-gateway.example.com matching application expectation. Flush negative cache by restarting CoreDNS pods after config fix (cache TTL 30s otherwise sufficient). Replay failed webhooks from Stripe dashboard Events API for last 6 hours. Update Kustomize with validation webhook rejecting WEBHOOK_* values not ending in .svc.cluster.local or verified external FQDN. Add synthetic check CronJob performing DNS lookup on all ConfigMap-declared hostnames every 5 minutes. Document DNS naming standard: applications must use Kubernetes FQDN or explicit external FQDN, never bare internal pseudo-domains.

Commands

kubectl run dnsdebug --image=busybox:1.36 --rm -it -n payments-prod -- nslookup hooks.payments.internal

kubectl get svc -n payments-prod

kubectl get cm payment-config -n payments-prod -o yaml | grep WEBHOOK

kubectl patch cm payment-config -n payments-prod -p '{"data":{"WEBHOOK_CALLBACK_HOST":"hooks-gateway.payments-prod.svc.cluster.local"}}'

kubectl rollout restart deploy payment-webhook -n payments-prod

Symptom

Grafana and Prometheus unreachable; alert NodeDiskPressure fired on 5 nodes; kubelet evicted pods with reason DiskPressure; container logs missing; nodes reported ImageFS and NodeFS usage above eviction thresholds; on-call flying blind during unrelated deploy.

Error Message

EvictionManager: attempting to reclaim nodefs: pods ranked for eviction; Evicted pod: prometheus-k8s-0 in namespace monitoring; status: Failed, reason: Evicted, message: The node was low on resource: ephemeral-storage

Root Cause

Worker nodes used 100Gi gp3 root volumes with default kubelet eviction-hard thresholds (nodefs.available<10%, imagefs.available<15%). Over six months, container image layers accumulated—47 versions of deploy-agent sidecar never pruned because imageGCHighThresholdPercent was default and CI/CD deployed 30 times daily. Prometheus in monitoring namespace used emptyDir for WAL on nodes without dedicated data volumes due to cost shortcut; WAL grew to 38Gi during cardinality incident week prior. Logrotate on node /var/log/pods was disabled by a Chef recipe bug leaving 12Gi of rotated json logs uncompressed. When deployment pushed 2.4GB container images simultaneously to five nodes, imagefs.available dropped below 15% triggering kubelet DiskPressure taint node.kubernetes.io/disk-pressure. Eviction manager ranked pods by QoS and usage—BestEffort log collectors first, then Burstable prometheus-k8s-0. Observability stack collapse masked the original deploy failure symptoms. Cluster had no centralized logging yet—node disk metrics existed in node_exporter but alert DiskPressure fired only at hard threshold not soft. No ephemeral-storage requests on Prometheus pods allowed scheduling onto saturated nodes.

Diagnosis Steps

1. 1.kubectl describe node <pressure-node> | grep -A5 'Conditions' | grep DiskPressure
2. 2.SSH/SSM to node: df -h /var/lib/kubelet /var/lib/containerd; du -sh /var/log/pods/* | sort -h | tail
3. 3.kubectl get pods -A --field-selector spec.nodeName=<node> -o json | jq '.items[] | select(.status.reason=="Evicted")'
4. 4.Review kubelet config for eviction-hard thresholds and imageGC settings.

Solution

Emergency: kubectl cordon affected nodes; run crictl rmi --prune on each via DaemonSet job; truncate oversized logs in /var/log/pods with logrotate fix. Uncordon after nodefs.available >20%. Move Prometheus WAL to PVC with 100Gi request/limit ephemeral-storage. Patch kubelet config (KubeletConfiguration) eviction-soft with grace periods and raise imageGCLowThresholdPercent to 60. Deploy weekly image prune CronJob on nodes. Scale node root volumes to 200Gi via launch template v43. Restore monitoring: delete Evicted prometheus pods, StatefulSet recreates; verify TSDB integrity. Add alert at nodefs.available <25% warning. Require ephemeral-storage requests for any pod writing >1Gi. Re-enable logrotate via fixed Chef cookbook. During incident, use cloudwatch agent on nodes as backup metrics path.

Commands

kubectl describe node | grep -A3 DiskPressure

kubectl get pods -A --field-selector status.reason=Evicted

kubectl debug node/<node> -it --image=busybox -- chroot /host df -h /var/lib/kubelet

crictl rmi --prune

kubectl patch statefulset prometheus-k8s -n monitoring --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/resources/requests/ephemeral-storage","value":"10Gi"}]'

Symptom

After changing clusterDomain, internal DNS resolution fails. Pods cannot reach services. Existing certificates with old domain SANs fail TLS validation.

Error Message

could not resolve host: kubernetes.default.svc.company.internal

Root Cause

clusterDomain is embedded in kubelet config, CoreDNS configmap, kube-apiserver service-account-issuer, pod DNS configs, service account tokens, and TLS certificates. Changing only some components leaves others pointing to the old domain.

Diagnosis Steps

1. 1.Check kubelet config: systemctl cat kubelet | grep clusterDomain
2. 2.Check CoreDNS: kubectl get cm coredns -n kube-system -o yaml
3. 3.Check apiserver flags: kubectl get pod kube-apiserver -n kube-system -o yaml | grep service-account-issuer
4. 4.Check pod DNS: kubectl exec pod -- cat /etc/resolv.conf

Solution

Changing clusterDomain requires updating kubelet on all nodes (drain+restart), CoreDNS configmap, kube-apiserver flags, regenerating service account tokens, re-issuing TLS certificates. Effectively a full cluster rolling restart. Avoid if possible — set correctly at cluster creation.

Commands

kubectl get cm coredns -n kube-system -o yaml

systemctl cat kubelet | grep clusterDomain

Symptom

Namespace deletion hangs indefinitely. Namespace conditions show NamespaceDeletionDiscoveryFailure with stale GroupVersion discovery error.

Error Message

Discovery failed for some groups: unable to retrieve the complete list of server APIs: cluster.core.oam.dev/v1alpha1: stale GroupVersion discovery

Root Cause

An APIService was registered for cluster.core.oam.dev/v1alpha1 but the backing controller was removed. Namespace deletion requires discovering all resource types to run finalizers. When API discovery fails, the namespace controller blocks deletion.

Diagnosis Steps

1. 1.Check namespace conditions: kubectl get ns stuck-ns -o yaml
2. 2.List API services: kubectl get apiservices | grep -v Available
3. 3.Check for orphaned CRDs: kubectl get crd | grep oam
4. 4.Check backing service exists: kubectl get svc -A | grep oam

Solution

Remove the stale APIService: kubectl delete apiservice v1alpha1.cluster.core.oam.dev. If still stuck, remove finalizers from remaining resources. Last resort: patch namespace to remove its finalizer.

Commands

kubectl get apiservices | grep -v Available

kubectl delete apiservice v1alpha1.cluster.core.oam.dev

kubectl patch ns stuck-ns -p "{\"metadata\":{\"finalizers\":[]}}" --type=merge

Symptom

During the May 29-30, 2026 Azure West US 2 incident, AKS was listed among affected services. Customer workloads in the region could experience unavailable nodes, failed volume operations, delayed service management, missing telemetry, and degraded application availability.

Error Message

NodeNotReady, pod scheduling failures, persistent volume attach or mount failures, Azure API management errors

Root Cause

The underlying Azure regional incident affected compute, networking, storage, and service management paths after power disturbances caused cooling lockouts and infrastructure shutdowns. Even if the Kubernetes control plane was reachable, worker nodes and persistent storage in the affected regional failure domain could not reliably serve workloads.

Diagnosis Steps

1. 1.Check Azure Service Health before assuming a Kubernetes-only fault.
2. 2.Run `kubectl get nodes -o wide` to see node readiness and zone placement.
3. 3.Inspect pods stuck in Pending, ContainerCreating, or Terminating states.
4. 4.Check events for volume attach, cloud provider, and node heartbeat errors.
5. 5.Review whether critical services have cross-zone and cross-region replicas.

Solution

Freeze non-essential deployments, fail over customer traffic to a healthy region if the application supports it, and prioritize stateless service recovery before stateful workloads. For stateful services, verify storage consistency and application-level replication before forcing recovery.

Commands

kubectl get nodes -o wide

kubectl get pods -A --field-selector=status.phase!=Running

kubectl get events -A --sort-by=.lastTimestamp

az aks show --resource-group <rg> --name <cluster>

az resource list --location westus2

Symptom

11:04 PM — a scheduled rolling deploy of payments-api began replacing pods across the EKS cluster. Within 6 minutes, 40 of 60 new pods were stuck in ImagePullBackOff while the 20 old pods kept running normally, since Kubernetes only pulls a fresh image when a pod is actually rescheduled. On-call was paged for 'deployment stuck, rollout not progressing.'

Error Message

Failed to pull image "123456789012.dkr.ecr.us-east-1.amazonaws.com/payments-api:2.7": rpc error: code = Unknown desc = failed to pull and unpack image: failed to resolve reference: unexpected status from HEAD request: 401 Unauthorized

Root Cause

Earlier that day, an unrelated Terraform PR refactored the EKS node group module to consolidate several duplicate IAM policies into a single shared managed policy, intending to reduce policy sprawl across 5 node groups. The refactor correctly preserved EC2 and CNI permissions but the engineer copied the policy list from an older node group that predated a security-hardening pass which had added ecr:GetAuthorizationToken and ecr:BatchGetImage as explicit statements rather than relying on the AmazonEC2ContainerRegistryReadOnly managed policy. Because the older node group being copied from had that policy attached separately, its absence from the new consolidated policy wasn't caught in code review — the terraform plan showed the new policy being created and attached, which looked correct, without surfacing that a previously separate policy attachment was being silently detached in the same apply. New nodes launched after the change had a node IAM role that could authenticate to AWS generally (EC2 describe calls worked fine) but was denied specifically for ECR image pulls, and since ImagePullBackOff had already occurred sporadically before from transient network blips, on-call initially chased NAT gateway health for 20 minutes before checking IAM.

Diagnosis Steps

1. 1.Run `kubectl describe pod <pod> | grep -A5 Events` to see the raw pull error rather than just the ImagePullBackOff state name.
2. 2.Confirm it's IAM and not networking by manually running `aws ecr get-login-password --region us-east-1` from a debug pod scheduled on an affected node.
3. 3.Diff the node group's attached IAM policies against a known-good node group: `aws iam list-attached-role-policies --role-name <new-role> ` vs `--role-name <old-role>`.
4. 4.Correlate the timestamp of the first failing pull against recent Terraform applies via `terraform state list | grep node_group` and the CI pipeline's deploy history.

Solution

Re-attach the AmazonEC2ContainerRegistryReadOnly managed policy (or the equivalent explicit ecr:GetAuthorizationToken / ecr:BatchGetImage / ecr:GetDownloadUrlForLayer statements) to the affected node group's IAM role via a targeted Terraform apply, rather than a manual console fix, so the correction is captured in code and doesn't regress on the next apply. Do not restart or delete the already-failing pods first — new pods scheduled on the newly-fixed node role will pull successfully once the policy propagates (IAM changes typically take under a minute to be effective for new token requests), and Kubernetes will naturally retry the existing failing pods on its backoff schedule without manual intervention. Roll new nodes only if the node group's instance profile needs to pick up the corrected role, since existing running nodes already have the instance profile attached and just need the underlying role's policy to update.

Commands

aws iam attach-role-policy --role-name eks-payments-nodegroup-role --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

terraform plan -target=module.eks_node_group.aws_iam_role_policy_attachment.ecr_readonly

terraform apply -target=module.eks_node_group.aws_iam_role_policy_attachment.ecr_readonly

kubectl get pods -A --field-selector=status.phase=Pending -o wide

Symptom

2:03 PM — during a scheduled flash-sale promotion, checkout-api's p99 latency climbed from 180ms to 4.2s over 12 minutes, and the ALB began returning 504s at a rate of roughly 3% of total requests. Grafana showed CPU utilization per pod pegged near 100%, and the on-call engineer's first assumption was that HPA had failed to trigger at all.

Root Cause

HPA was in fact functioning correctly and had already scaled checkout-api from its baseline of 6 replicas up to 20 replicas within the first 4 minutes of the traffic spike — but 20 was the hardcoded maxReplicas value set 8 months earlier when the service handled roughly a third of current baseline traffic, and nobody had revisited it since, because normal day-to-day load never came close to that ceiling. There was no alert configured for 'HPA currentReplicas equals maxReplicas while target utilization is still exceeded,' so from the dashboards available to on-call, a fully-scaled, fully-saturated fleet at its configured limit looked identical to a fleet that had simply failed to scale — both show sustained high CPU with no further replica growth. The marketing team's flash-sale traffic projection had been shared with the data team for CDN capacity planning but was never cross-referenced against Kubernetes HPA ceilings, a gap that existed because HPA maxReplicas was treated as an infrastructure implementation detail rather than a capacity number requiring the same review as, say, RDS instance sizing.

Diagnosis Steps

1. 1.Run `kubectl get hpa checkout-api-hpa` and compare the CURRENT and MAX columns directly, rather than assuming a plateau in replica count means HPA is broken.
2. 2.Run `kubectl describe hpa checkout-api-hpa` to confirm the reported CPU utilization percentage is still above the target threshold even at max replicas.
3. 3.Check `kubectl get nodes` and cluster autoscaler logs to rule out whether the ceiling was actually node capacity rather than the HPA spec itself.
4. 4.Cross-reference the timestamp replicas stopped growing against the HPA spec's maxReplicas value using `kubectl get hpa checkout-api-hpa -o yaml`.

Solution

Immediately patch the HPA's maxReplicas upward via `kubectl patch hpa checkout-api-hpa --patch '{"spec":{"maxReplicas":50}}'` to give the deployment headroom to keep scaling — this is a safe, additive, non-disruptive change since it only raises a ceiling and doesn't touch running pods. Confirm the cluster autoscaler's node group max size is also high enough to accommodate the new pod ceiling, since raising maxReplicas without available node capacity just shifts the bottleneck from 'HPA capped' to 'pods stuck Pending for lack of schedulable nodes.' Once replicas grew to roughly 34, p99 latency recovered to 210ms within 3 minutes as load distributed across the larger fleet.

Commands

kubectl get hpa checkout-api-hpa

kubectl patch hpa checkout-api-hpa --patch '{"spec":{"maxReplicas":50}}'

kubectl get nodes -o wide | wc -l

kubectl get pods -n checkout --field-selector=status.phase=Pending

Symptom

6:51 AM — a routine deployment of notifications-worker began, and every new pod entered CrashLoopBackOff within 2-3 seconds of starting, with kubectl logs returning completely empty output. Grafana's memory usage panel for the service, which averages usage over 1-minute windows, showed nothing above 60% of the configured limit, making the crash look inexplicable from the dashboard alone.

Root Cause

kubectl describe pod revealed the actual signal that the dashboard couldn't show: Last State Terminated, Reason: OOMKilled, Exit Code: 137. The application, a JVM-based worker, was allocating a large off-heap buffer pool and JIT-compiling several hot code paths during its first 400 milliseconds of startup, producing a genuine but extremely brief memory spike well above its configured 512Mi limit — a spike so short-lived that it never registered in the 1-minute-averaged Grafana panel, which only ever showed steady-state usage after the crash loop had already been running for several minutes and cAdvisor's scrape interval smoothed the peak away. The memory limit had been set based on observed steady-state usage from a load test months earlier that happened to run the JVM with a warm class cache (from a prior test iteration in the same container), never exercising a truly cold JVM start, so the startup spike had simply never been measured.

Diagnosis Steps

1. 1.Run `kubectl describe pod notifications-worker-xyz | grep -A8 'Last State'` to find Exit Code 137, since `kubectl logs` alone showed nothing.
2. 2.Run `kubectl logs notifications-worker-xyz --previous` to confirm the previous instance also produced zero application-level output before dying, ruling out an application-level crash.
3. 3.Query the Kubernetes events stream directly rather than the averaged dashboard: `kubectl get events --field-selector reason=OOMKilling -A`.
4. 4.Reproduce locally with a tight memory cgroup (`docker run -m 512m`) and a container-level memory sampler at sub-second resolution to actually observe the startup spike the 1-minute dashboard was smoothing away.

Solution

Raise the container's memory limit to accommodate the measured cold-start peak plus a safety margin (in this case from 512Mi to 900Mi, based on a sub-second profiler run showing a 780Mi peak at 340ms into startup), rather than only tuning steady-state usage. Additionally, tune the JVM's startup behavior with -XX:+TieredCompilation and a smaller initial off-heap buffer allocation to reduce the peak itself, since raising the limit alone treats the symptom without addressing why the spike is so large in the first place. Roll the fix out to one canary pod first and watch its actual memory profile through a full cold start before fleet-wide rollout, since the whole incident stemmed from never having observed a true cold start under production-like constraints.

Commands

kubectl describe pod notifications-worker-xyz | grep -A8 'Last State'

kubectl get events --field-selector reason=OOMKilling -A

kubectl patch deployment notifications-worker --patch '{"spec":{"template":{"spec":{"containers":[{"name":"worker","resources":{"limits":{"memory":"900Mi"}}}]}}}}'

kubectl rollout status deployment/notifications-worker

Symptom

A deployment's pods repeatedly start and die; kubectl shows CrashLoopBackOff with a growing restart count and increasing backoff delay between restarts.

Error Message

Back-off restarting failed container app in pod web-6f... (reason: CrashLoopBackOff)

Root Cause

The container process exits almost immediately — usually a bad config/env value, a missing file or secret, an unmet dependency at startup, or a failing liveness probe killing a slow-booting app before it's ready. Kubernetes restarts it with exponential backoff, which looks like a loop.

Diagnosis Steps

1. 1.kubectl describe pod <pod> — read the container State/Last State and Events for the exit reason and code.
2. 2.kubectl logs <pod> --previous — the previous container's logs usually show the actual crash (stack trace, 'config not found').
3. 3.Check exit code: 1/2 = app error, 137 = OOMKilled/SIGKILL, 143 = SIGTERM.
4. 4.Verify referenced ConfigMaps/Secrets exist and keys match; check that a startupProbe (not just liveness) protects slow boots.

Solution

Fix the underlying cause (correct the env/secret, add the missing dependency, or relax/replace the liveness probe with a startupProbe for slow starts). Redeploy. For a bad rollout, roll back to the last good revision immediately while investigating.

Commands

kubectl logs <pod> --previous

kubectl describe pod <pod> | sed -n '/Events/,$p'

kubectl rollout undo deploy/<name>

Symptom

New pods stay in Pending/Waiting and never start; status shows ImagePullBackOff or ErrImagePull.

Error Message

Failed to pull image "registry/app:v2": rpc error: code = Unknown desc = failed to resolve reference / unauthorized

Root Cause

The kubelet can't fetch the image: wrong image name/tag, the tag doesn't exist, a private registry without a valid imagePullSecret, expired registry credentials, or the node can't reach the registry (network/DNS/proxy).

Diagnosis Steps

1. 1.kubectl describe pod <pod> — the Events show the exact pull error (not found vs unauthorized vs timeout).
2. 2.Confirm the image:tag actually exists in the registry (typos, missing tag, wrong registry host).
3. 3.For private registries, check the pod/serviceaccount has a working imagePullSecret and the token isn't expired.
4. 4.From a node or debug pod, test connectivity/auth to the registry (DNS, proxy, firewall).

Solution

Correct the image reference or push the missing tag; attach or refresh a valid imagePullSecret; or fix node egress to the registry. Delete the failing pods so the controller recreates them and pulls successfully.

Commands

kubectl describe pod <pod> | grep -A5 Events

kubectl create secret docker-registry regcred --docker-server=... --docker-username=... --docker-password=...

kubectl get sa default -o yaml   # confirm imagePullSecrets attached

Symptom

kubectl get nodes shows a node as NotReady; pods on it show as Running but are unreachable, and after a delay they may be marked for eviction.

Error Message

node <name> status is now: NodeNotReady; kubelet stopped posting node status

Root Cause

The kubelet on that node stopped reporting healthy — commonly the kubelet or container runtime crashed, the node ran out of disk (DiskPressure) or memory, lost network to the control plane, or the underlying VM died. The control plane can't confirm the pods, so it waits before rescheduling.

Diagnosis Steps

1. 1.kubectl describe node <name> — check Conditions (MemoryPressure, DiskPressure, PIDPressure) and the last heartbeat time.
2. 2.SSH to the node: systemctl status kubelet and the container runtime; check disk with df -h and dmesg for OOM.
3. 3.Verify node→apiserver connectivity (network, security groups, certs).
4. 4.Decide: recover the node vs cordon+drain and let workloads reschedule elsewhere.

Solution

If recoverable, free disk/restart the kubelet/runtime and the node rejoins. If not, cordon and drain (or delete) the node so pods reschedule onto healthy capacity; replace the instance. Ensure controllers (Deployments) exist so pods actually reschedule — bare pods won't.

Commands

kubectl describe node <name> | sed -n '/Conditions/,/Events/p'

kubectl cordon <name> && kubectl drain <name> --ignore-daemonsets --delete-emptydir-data

kubectl delete node <name>   # after replacing the instance

Symptom

Containers restart intermittently under traffic; kubectl shows Last State: Terminated, Reason: OOMKilled, exit code 137, and restart counts climb during peaks.

Error Message

State: Terminated  Reason: OOMKilled  Exit Code: 137

Root Cause

The container exceeded its memory limit (or the node ran out of memory and the kernel OOM-killer picked the container). Usually the memory limit is set too low for real load, there's a memory leak, or requests/limits were never set so scheduling and eviction misbehave.

Diagnosis Steps

1. 1.kubectl describe pod <pod> — confirm Reason: OOMKilled and note the limit vs actual usage.
2. 2.kubectl top pod / metrics or Prometheus — chart container memory over time (steady climb = leak; sawtooth at limit = undersized).
3. 3.Check requests/limits: missing requests hurt scheduling; limits too close to steady usage leave no headroom.
4. 4.For JVM/Node apps, verify the runtime is container-memory-aware (heap sized to the cgroup limit).

Solution

Right-size memory requests/limits from observed peak usage with headroom; fix genuine leaks; make runtimes cgroup-aware (e.g., JVM +UseContainerSupport, correct max-old-space-size). Consider a VerticalPodAutoscaler in recommendation mode to guide the numbers.

Commands

kubectl describe pod <pod> | grep -A3 'Last State'

kubectl top pod <pod> --containers

kubectl set resources deploy/<name> --limits=memory=1Gi --requests=memory=512Mi

Symptom

During a node upgrade, kubectl drain stalls and never completes, repeatedly reporting it cannot evict certain pods.

Error Message

error when evicting pod "web-...": Cannot evict pod as it would violate the pod's disruption budget.

Root Cause

A PodDisruptionBudget requires a minimum number of available replicas, and evicting the pod would drop below it — often because the deployment is already at its floor, replicas are unschedulable elsewhere (no capacity/affinity), or the PDB minAvailable equals the replica count.

Diagnosis Steps

1. 1.kubectl get pdb — compare ALLOWED DISRUPTIONS (0 means the budget is exhausted).
2. 2.Check the owning workload's replica count vs the PDB minAvailable/maxUnavailable.
3. 3.kubectl get pods -o wide — is there capacity/affinity for replacements to schedule on other nodes?
4. 4.kubectl describe node <name> — confirm what's left to evict.

Solution

Restore headroom so eviction is allowed: scale the workload up so a replica can move, add node capacity (Cluster Autoscaler), or relax the PDB (e.g., minAvailable from N to N-1) for the maintenance window. Once ALLOWED DISRUPTIONS > 0, the drain proceeds. Avoid --force/--disable-eviction except as a last resort — it bypasses the safety the PDB provides.

Commands

kubectl get pdb -A

kubectl scale deploy/web --replicas=4    # add headroom so one can move

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

Symptom

kubectl commands time out or return connection errors cluster-wide; running pods keep serving traffic but no changes, scaling, or scheduling work.

Error Message

The connection to the server <api>:6443 was refused / etcdserver: request timed out

Root Cause

The apiserver or etcd is unhealthy — etcd lost quorum (majority of members down or partitioned), the apiserver crashed/OOMed, expired control-plane certificates, or a full etcd disk. Because the apiserver is the only path to etcd, everything management-plane stops while data-plane pods keep running.

Diagnosis Steps

1. 1.From a control-plane node, check static pods: crictl ps / journalctl for kube-apiserver and etcd.
2. 2.Check etcd health and quorum: etcdctl endpoint health/status across all members.
3. 3.Verify certs haven't expired: kubeadm certs check-expiration.
4. 4.Check etcd disk space and fsync latency (a slow/full disk stalls writes).

Solution

Restore quorum first: bring failed etcd members back or recover from a recent snapshot if data is lost. Restart a crashed apiserver (fix OOM by raising limits), renew expired certs, or free etcd disk (compact/defrag). Once etcd has quorum and the apiserver is up, the control plane recovers and reconciliation resumes.

Commands

sudo crictl ps | grep -E 'apiserver|etcd'

ETCDCTL_API=3 etcdctl endpoint status --cluster -w table

sudo kubeadm certs check-expiration

Symptom

Newly created pods sit in Pending indefinitely and never get a node assigned; the workload never reaches the desired replica count.

Error Message

0/6 nodes are available: 3 Insufficient cpu, 3 node(s) had untolerated taint {...}. FailedScheduling

Root Cause

The scheduler can't find a node that satisfies the pod: not enough allocatable CPU/memory for the requests, a nodeSelector/affinity that no node matches, taints without a matching toleration, unsatisfiable topology-spread/anti-affinity, or an unbound PVC waiting for a volume.

Diagnosis Steps

1. 1.kubectl describe pod <pod> — the FailedScheduling event lists the exact reason per node (Insufficient cpu, untolerated taint, no matching node).
2. 2.kubectl get nodes -o wide and kubectl describe node — check allocatable vs requested resources.
3. 3.Review the pod's requests, nodeSelector/affinity, tolerations, and topologySpreadConstraints for something no node can satisfy.
4. 4.If it mentions volumes, check the PVC is Bound (kubectl get pvc) and the StorageClass exists.

Solution

Give the scheduler a viable placement: lower over-inflated requests, add a matching toleration or relax affinity, add node capacity (or let Cluster Autoscaler/Karpenter add a node), or fix the unbound PVC/StorageClass. The pod schedules as soon as one node satisfies all constraints.

Commands

kubectl describe pod <pod> | grep -A5 FailedScheduling

kubectl describe node <node> | sed -n '/Allocated resources/,/Events/p'

kubectl get pvc   # ensure volumes are Bound

Symptom

Applications suddenly can't resolve service names (connection errors to hostnames) right after a default-deny NetworkPolicy was rolled out; IP-based connections still work.

Error Message

dial tcp: lookup my-svc.default.svc.cluster.local: i/o timeout / Temporary failure in name resolution

Root Cause

A default-deny egress NetworkPolicy blocked the pods' DNS queries to CoreDNS (kube-dns) on UDP/TCP 53. NetworkPolicy is deny-by-default once it selects a pod, and the policy forgot to allow egress to the DNS service.

Diagnosis Steps

1. 1.Confirm the timing: resolution broke exactly when the policy was applied.
2. 2.kubectl get networkpolicy -A and inspect egress rules for a missing DNS allow.
3. 3.Test from a pod: nslookup a service name (fails) vs connecting to a known pod IP (works) — isolates DNS specifically.
4. 4.Verify CoreDNS pods are healthy in kube-system (rule out an unrelated DNS outage).

Solution

Add an egress allow rule for DNS to every namespace's policy: permit UDP and TCP port 53 to the kube-system namespace (CoreDNS). Once DNS egress is allowed, name resolution recovers immediately.

Commands

kubectl get networkpolicy -A -o wide

kubectl exec -it <pod> -- nslookup kubernetes.default

kubectl -n kube-system get pods -l k8s-app=kube-dns