Production Use Cases

Context

A retail platform operated 85 worker nodes and 220 production Deployments across three zones. Monthly node image upgrades repeatedly caused 30-60 second checkout brownouts.

Problem

The team used `kubectl drain` correctly, but several services had only two replicas and no PodDisruptionBudget. During node maintenance, voluntary evictions removed too much serving capacity at once. The load balancer saw endpoints disappear faster than replacement pods became Ready, and checkout retries amplified latency.

Solution

They added PodDisruptionBudgets for tier-1 services, raised replicas from two to four for critical APIs, configured maxUnavailable to one, and changed upgrade automation to verify PDB status before draining a node. Readiness probes were tightened to include dependency checks, and maintenance dashboards tracked unavailable replicas by namespace.

Commands

kubectl get pdb -A # Confirms disruption budgets exist before maintenance

kubectl drain ip-10-42-7-91 --ignore-daemonsets --delete-emptydir-data --timeout=10m # Drains one node with eviction API protection

kubectl rollout status deploy/checkout-api -n payments --timeout=180s # Verifies replacement capacity is Ready

Context

A Series-C fintech company processing 12,000 transactions/second across 47 microservices on EKS in us-east-1. A failed deployment during peak hours caused a 4-minute checkout outage costing $380K in lost revenue. The CTO mandated zero-downtime deployments for all tier-1 services within 30 days.

Problem

The platform team discovered that standard rolling updates were not truly zero-downtime. During deployments, the Kubernetes endpoints controller removed terminating pods from Service endpoints immediately, but the AWS ALB target group deregistration took 15-30 seconds to propagate. During this window, the ALB continued sending traffic to pods that were already shutting down, resulting in 502 errors. Additionally, new pods were marked Ready by the kubelet before they had fully warmed their connection pools to PostgreSQL and Redis. The readiness probe (a simple HTTP 200 check on /healthz) passed even though the application was not yet capable of serving production traffic. During deployments of the payments-gateway service, error rates spiked from 0.01% to 3.2% for 45-60 seconds as new pods received traffic before connection pools were established and old pods received traffic after beginning shutdown. The team tried increasing terminationGracePeriodSeconds but this only delayed the problem without solving it. They also tried preStop hooks with sleep commands but could not determine the correct sleep duration across different services with different deregistration times.

Solution

The team implemented a three-layer zero-downtime deployment strategy. First, they added Pod Readiness Gates tied to the AWS Load Balancer Controller. By adding the condition `target-health.elbv2.k8s.aws/payments-gw-tg` to the pod spec's readinessGates field, pods were not considered Ready until the ALB confirmed the target was healthy and receiving traffic. This prevented the endpoints controller from adding pods to the Service before ALB registration completed. Second, they configured PodDisruptionBudgets with `maxUnavailable: 1` for all tier-1 services and set deployment strategy to `maxSurge: 25%` and `maxUnavailable: 0`. The maxUnavailable=0 ensured no old pods were terminated until new pods were fully ready (including passing readiness gates). Third, they added a preStop lifecycle hook with a 15-second sleep followed by a SIGTERM handler that stopped accepting new connections but completed in-flight requests within 30 seconds. The full terminationGracePeriodSeconds was set to 60 to accommodate this. The deployment manifest for the payments-gateway included explicit resource requests to prevent scheduling delays, and the HPA was configured with a stabilization window to prevent scale-down during deployments. They validated the approach by running a load test at 15,000 req/s during a deployment and confirming zero 5xx errors in the ALB access logs.

Commands

kubectl apply -f payments-gateway-deployment.yaml # Deploys with readinessGates, PDB, and preStop hook configured

kubectl rollout status deploy/payments-gateway -n payments --timeout=300s # Watches rollout progress including readiness gate satisfaction

kubectl get pods -n payments -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="target-health.elbv2.k8s.aws/payments-gw-tg")].status}{"\n"}{end}' # Verifies ALB readiness gate status per pod

kubectl get pdb payments-gateway-pdb -n payments # Confirms PDB allows at least 1 disruption: ALLOWED DISRUPTIONS should be >= 1

kubectl describe deploy payments-gateway -n payments | grep -A5 'RollingUpdateStrategy' # Validates maxSurge=25% and maxUnavailable=0

kubectl logs -n payments deploy/payments-gateway --since=5m | grep 'connection pool warm' # Confirms new pods have established DB connections before serving traffic

Context

A B2B payments processor handling settlement batches for 2,200 enterprise clients on GKE. Every weekday at 16:00 UTC, settlement windows trigger a 15x traffic spike from 800 req/s to 12,000 req/s within 2 minutes. The team had 3 baseline pods and needed to scale to 50 pods before the spike overwhelmed the service.

Problem

The initial HPA was configured with CPU-based scaling (targetAverageUtilization: 70%), but CPU metrics lagged the actual traffic spike by 40-60 seconds due to the metrics-server scrape interval and HPA evaluation period. By the time the HPA decided to scale, the existing 3 pods were already saturated at 95% CPU, response latency jumped from 45ms to 2,800ms, and upstream timeout errors cascaded through the settlement pipeline. Circuit breakers tripped in 6 downstream services. The team tried reducing the HPA sync period but CPU still did not reflect the queue depth building in the service. They also tried setting minReplicas=50, but maintaining 50 pods 24/7 cost $18,400/month for a spike that lasted only 35 minutes per day. The settlement batch processing required exactly-once delivery guarantees, so dropped requests meant manual reconciliation costing the operations team 12 hours per incident.

Solution

The team deployed the Prometheus Adapter to expose custom metrics to the Kubernetes metrics API. They instrumented the payments-api with a Prometheus gauge `settlement_queue_depth` that tracked pending settlement messages in the RabbitMQ queue, and a histogram `http_request_duration_seconds` for p99 latency. The HPA was reconfigured to scale on `settlement_queue_depth` with a target value of 100 messages per pod, which gave the HPA a leading indicator 60-90 seconds before CPU saturation. They set `behavior.scaleUp.stabilizationWindowSeconds: 0` and `behavior.scaleUp.policies` to allow adding 100% of current pods every 15 seconds, enabling aggressive scale-up from 3 to 50 in under 90 seconds. Scale-down was configured conservatively with a 300-second stabilization window to prevent flapping. The Prometheus Adapter was configured with a custom rule that queried `sum(rabbitmq_queue_messages{queue="settlements"})` and exposed it as `settlement_queue_depth` on the payments-api deployment. They also added a secondary metric of p99 latency threshold at 200ms. The HPA used the `Max` policy for scale-up, selecting whichever metric demanded more replicas.

Commands

kubectl apply -f prometheus-adapter-config.yaml # Deploys custom metrics rule mapping RabbitMQ queue depth to k8s metrics API

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/payments/deployments/payments-api/settlement_queue_depth # Verifies custom metric is exposed and returning current queue depth

kubectl get hpa payments-api-hpa -n payments -w # Watches HPA scaling decisions in real-time showing TARGETS and REPLICAS columns

kubectl top pods -n payments -l app=payments-api --sort-by=cpu # Monitors CPU distribution across scaled pods during spike

kubectl describe hpa payments-api-hpa -n payments | grep -A20 'Metrics:' # Shows current metric values and scaling decisions for both queue_depth and latency metrics

kubectl get pods -n payments -l app=payments-api --field-selector=status.phase=Running | wc -l # Counts running pods during scale-up event

Context

A SaaS observability platform onboarding 340 enterprise clients onto a shared EKS cluster with 120 nodes across 3 availability zones. Each tenant ran data-ingestion pipelines and Grafana dashboards. A noisy-neighbor incident where one tenant's runaway query consumed 80% of cluster CPU impacted 12 other tenants' dashboards, violating SLA agreements.

Problem

The platform originally deployed all tenants into a single namespace with labels for logical separation. This created three critical problems. First, there was no resource isolation: tenant-acme's analytics job could request unlimited CPU and memory, starving tenant-globex's production ingestion pipeline. ResourceQuota did not exist, so a single kubectl apply with requests: 64Gi memory could claim half the cluster. Second, network isolation was non-existent: any pod could reach any other pod across tenants because the default Kubernetes network policy is allow-all. A misconfigured service in tenant-acme directly queried tenant-globex's internal PostgreSQL because service DNS was discoverable cluster-wide. Third, RBAC was a mess: developers from one tenant could kubectl exec into another tenant's pods because they shared a ClusterRoleBinding. The security audit flagged 23 cross-tenant access violations in a single week. The noisy-neighbor SLA breach triggered a $240K credit payout and two enterprise clients threatened to leave.

Solution

The team designed a namespace-per-tenant architecture with three layers of isolation. Layer 1 - Namespace and RBAC: Each tenant received a dedicated namespace (e.g., tenant-acme, tenant-globex) with a RoleBinding granting tenant-specific ServiceAccounts access only to their namespace. A custom admission controller (OPA Gatekeeper) blocked any pod creation that did not have a tenant label matching the namespace. Layer 2 - ResourceQuota and LimitRange: Each namespace received a ResourceQuota capping total CPU requests at 16 cores, memory at 64Gi, and pod count at 100. A LimitRange set default container requests (250m CPU, 512Mi memory) and limits (2 CPU, 4Gi memory) to prevent unbounded resource consumption. Tenants on the Enterprise plan received higher quotas (32 cores, 128Gi). Layer 3 - NetworkPolicy with Calico: A default-deny-all ingress and egress NetworkPolicy was applied to every tenant namespace. Explicit policies allowed ingress only from the ingress-controller namespace and egress only to the shared-services namespace (for DNS, monitoring exporters) and specific CIDR ranges for external APIs. Cross-tenant traffic was completely blocked. The team used Calico's GlobalNetworkPolicy to enforce cluster-wide baseline rules and namespace-scoped policies for tenant-specific exceptions. They built a Terraform module that provisioned all three layers atomically when onboarding a new tenant, completing in 8 seconds.

Commands

kubectl create namespace tenant-acme --dry-run=client -o yaml | kubectl apply -f - # Creates tenant namespace idempotently

kubectl apply -f tenant-acme-quota.yaml # Applies ResourceQuota: 16 CPU, 64Gi memory, 100 pods max

kubectl apply -f tenant-acme-limitrange.yaml # Sets default requests/limits so no pod runs unbounded

kubectl apply -f tenant-acme-deny-all.yaml # Default deny-all NetworkPolicy for both ingress and egress

kubectl apply -f tenant-acme-allow-ingress.yaml # Allows traffic only from ingress-nginx namespace on port 8080

kubectl get resourcequota -n tenant-acme # Verifies quota enforcement: shows Used vs Hard limits

kubectl auth can-i get pods --as=system:serviceaccount:tenant-acme:tenant-sa -n tenant-globex # Should return 'no' - confirms cross-tenant RBAC isolation

Context

A healthcare data platform running HIPAA-compliant workloads on EKS in us-east-1 (primary) and us-west-2 (DR). The platform managed patient records for 1,400 hospitals and required RPO of 15 minutes and RTO of 5 minutes per compliance requirements. A us-east-1 regional S3 outage in March 2026 rendered the primary cluster inaccessible for 47 minutes.

Problem

The team had a manual DR runbook that required 14 steps across 4 different AWS consoles and took 35 minutes to execute. During the March outage, the on-call engineer was paged at 02:17 AM and began executing the runbook. At step 6, they discovered the Velero backup schedule had silently failed for 3 days because the S3 bucket policy had been modified by a security automation. The most recent valid backup was 72 hours old, violating the 15-minute RPO. Additionally, the DR cluster in us-west-2 had configuration drift: 8 of 23 microservices had been updated in production but not reflected in the DR cluster because ArgoCD was only syncing the primary cluster. The engineer attempted to manually apply the latest manifests but could not determine which ConfigMaps and Secrets were current. The 47-minute outage resulted in a HIPAA incident report filing and $1.2M in contractual penalties. The post-mortem identified three root causes: untested backup integrity, configuration drift between clusters, and manual failover procedures.

Solution

The team implemented automated DR with three components. Component 1 - Velero with Cross-Region Replication: Velero was configured with a 15-minute backup schedule targeting an S3 bucket with cross-region replication to us-west-2. A CronJob ran every 30 minutes to verify backup integrity by performing a test restore into a dedicated validation namespace and checking that critical resources (Deployments, ConfigMaps, Secrets, PVCs) were present. Backup failures triggered PagerDuty alerts with P1 severity. Component 2 - ArgoCD Multi-Cluster with ApplicationSets: ArgoCD was reconfigured with an ApplicationSet using the `clusters` generator to deploy identical applications to both us-east-1 and us-west-2 clusters simultaneously. The DR cluster received the same GitOps manifests but with replicas scaled to 0 (using an overlay in Kustomize). During failover, a single patch changed the replica count annotation, and ArgoCD synced within 30 seconds. Persistent data was handled by RDS with cross-region read replicas that could be promoted in 60-90 seconds. Component 3 - Automated Failover Controller: A custom operator running in a third management cluster (eu-west-1) monitored the primary cluster health via synthetic transactions every 10 seconds. If 6 consecutive checks failed (60 seconds), the operator automatically triggered failover: promoting the RDS read replica, patching ArgoCD ApplicationSet to activate DR replicas, and updating Route53 weighted routing to shift 100% traffic to us-west-2. Total automated failover time: 4 minutes 12 seconds.

Commands

velero schedule create critical-backup --schedule='*/15 * * * *' --include-namespaces=patient-data,clinical-api --snapshot-volumes=true --storage-location=s3-cross-region # Creates 15-minute backup schedule with volume snapshots

velero backup describe critical-backup-20260620-1600 --details # Verifies latest backup completed successfully with all resources captured

velero restore create --from-backup critical-backup-20260620-1600 --namespace-mappings patient-data:dr-validation # Test restore into validation namespace to verify backup integrity

argocd app list --server https://argocd.us-west-2.internal | grep -c 'Synced' # Confirms all DR applications are synced with Git (even if scaled to 0)

kubectl --context=dr-uswest2 patch applicationset patient-platform -n argocd --type=merge -p '{"spec":{"generators":[{"clusters":{"selector":{"matchLabels":{"environment":"active"}}}}]}}' # Activates DR cluster workloads

aws route53 change-resource-record-sets --hosted-zone-id Z1234 --change-batch file://failover-dns.json # Shifts DNS to DR region

Context

An e-commerce platform with a 2.3TB PostgreSQL 14 database serving 4,800 req/s on AKS. The database needed upgrading to PostgreSQL 16 for performance improvements (parallel vacuum, I/O optimizations) required to handle Black Friday traffic projections. The team had a 4-hour maintenance window on a Sunday night but could not afford data loss or extended downtime for a database serving order management.

Problem

Traditional PostgreSQL major version upgrades using pg_upgrade require the database to be offline during the upgrade process. For a 2.3TB database with 847 tables and 1,200 indexes, the estimated pg_upgrade time was 2 hours 15 minutes for the --link mode (which still required exclusive access). The team could not accept 2+ hours of downtime because the order fulfillment pipeline processed weekend returns and exchange requests continuously. A previous attempt at a streaming-replication-based upgrade failed because logical replication from PG14 to PG16 could not handle the custom extensions (pgvector, timescaledb) and large-object types used by the image catalog service. Additionally, the rollback strategy was unclear: if PG16 had unexpected query plan regressions after the upgrade, how would they revert without another 2-hour outage? The team needed a blue-green approach where both database versions ran simultaneously with a fast cutover.

Solution

The team designed a blue-green database migration using Kubernetes Jobs for schema migration, init containers for validation, and pglogical for bi-directional replication during the transition. Phase 1 - Blue (PG14) remains primary while Green (PG16) is provisioned: A Kubernetes Job ran pg_basebackup to create the PG16 instance from a PG14 snapshot, then ran pg_upgrade --link on the copy. This happened on a dedicated node pool with NVMe storage to complete in 45 minutes without impacting production. Phase 2 - Logical replication catchup: A second Job configured pglogical replication from Blue (PG14) to Green (PG16), streaming WAL changes accumulated during the upgrade. An init container on the PG16 StatefulSet verified replication lag was under 100ms before marking the pod Ready. Phase 3 - Cutover via Service selector swap: The application's database Service used a selector pointing to `version: pg14`. The cutover was a single kubectl patch changing the selector to `version: pg16`. Init containers on application pods validated connectivity to the new database endpoint before accepting traffic. Phase 4 - Rollback readiness: Reverse pglogical replication from Green to Blue was configured and running, so the selector could be switched back within 5 seconds if query regressions were detected. A Kubernetes CronJob ran query performance benchmarks every 60 seconds against Green, comparing p99 latencies to the Blue baseline. If p99 exceeded 150% of baseline for 3 consecutive checks, an automated rollback triggered.

Commands

kubectl apply -f pg16-upgrade-job.yaml # Launches Job that runs pg_basebackup + pg_upgrade --link on NVMe node pool (45min)

kubectl wait --for=condition=complete job/pg16-upgrade -n database --timeout=3600s # Waits for upgrade job to finish successfully

kubectl apply -f pglogical-replication-job.yaml # Configures WAL streaming from PG14→PG16, catches up accumulated changes

kubectl exec -n database pg16-0 -- psql -c "SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) FROM pg_replication_slots;" # Verifies replication lag is under 100ms before cutover

kubectl patch svc orders-db -n database -p '{"spec":{"selector":{"version":"pg16"}}}' # CUTOVER: switches traffic from PG14 to PG16 in under 1 second

kubectl get endpoints orders-db -n database # Confirms endpoints now point to PG16 pod IPs

Context

A real-time bidding platform processing 180,000 requests/second across 800 pods on a bare-metal Kubernetes cluster with 45 nodes. The ad-auction service had a strict 10ms p99 latency SLA for bid responses. iptables-based kube-proxy with 2,400 Services created 47,000 iptables rules, adding 3.2ms of kernel-space overhead per packet.

Problem

Performance profiling revealed that iptables rule traversal was consuming 3.2ms per connection establishment because the kernel evaluated rules sequentially (O(n) complexity). With 47,000 rules across 2,400 Services, every new connection triggered a linear scan through the NAT table. During traffic spikes, conntrack table exhaustion (default 131,072 entries) caused connection drops as the table filled with TIME_WAIT entries from the high-connection-rate bidding traffic. The team observed 0.3% packet loss during peak hours when conntrack table overflowed. Additionally, kube-proxy's iptables mode did not support session affinity with timeout granularity finer than seconds, which was required for the bidding protocol's sticky-session requirement (same buyer routes to same pod for 200ms auction windows). The team also needed L7 network policies to restrict which services could call the billing API based on HTTP headers, which iptables-based solutions could not provide without a full sidecar service mesh. A sidecar mesh (Istio/Linkerd) was rejected because it added 1.5ms latency per hop from the userspace proxy, which was unacceptable for the 10ms SLA.

Solution

The team migrated from kube-proxy with iptables to Cilium in eBPF-native mode, replacing both kube-proxy and providing L3/L4/L7 network policy enforcement entirely in kernel space. The migration was performed node-by-node using a canary approach. Step 1 - Cilium was deployed as a DaemonSet alongside existing kube-proxy, running in 'partial replacement' mode where Cilium handled only pods on nodes where it was active. Step 2 - A NodeSelector was used to gradually label nodes for Cilium (starting with 3 staging nodes, then 10% of production, then 50%, then 100%). On each labeled node, the Cilium agent took over Service load-balancing using eBPF maps (O(1) hash lookup replacing O(n) iptables scan). Step 3 - Once all nodes ran Cilium, kube-proxy was disabled by setting `--kube-proxy-replacement=strict` and removing the kube-proxy DaemonSet. Step 4 - Cilium's bandwidth manager was enabled using eBPF-based EDT (Earliest Departure Time) rate limiting, replacing the tc-based bandwidth plugin. Step 5 - L7 CiliumNetworkPolicies were applied to restrict billing-api access based on HTTP method and path, enforced in-kernel without sidecar proxies. Conntrack was replaced by Cilium's eBPF-based CT map with 512K entries and configurable GC intervals.

Commands

helm install cilium cilium/cilium --namespace kube-system --set kubeProxyReplacement=strict --set k8sServiceHost=10.0.0.1 --set k8sServicePort=6443 --set bpf.masquerade=true --set bandwidthManager.enabled=true # Installs Cilium with full kube-proxy replacement and bandwidth management

kubectl -n kube-system exec ds/cilium -- cilium status --verbose | grep KubeProxyReplacement # Confirms eBPF kube-proxy replacement is active: 'Strict'

kubectl -n kube-system exec ds/cilium -- cilium bpf ct list global | wc -l # Shows eBPF conntrack entries (replaces kernel conntrack table)

kubectl -n kube-system exec ds/cilium -- cilium bpf lb list # Displays eBPF Service map with all 2,400 services as hash entries

kubectl apply -f cilium-l7-policy-billing.yaml # Applies L7 network policy restricting billing API to POST /v1/charges from auction-service only

kubectl -n kube-system exec ds/cilium -- cilium monitor --type=policy-verdict -n bidding # Live monitors policy enforcement decisions for the bidding namespace

Context

A production e-commerce checkout API runs on a 3-node GKE cluster (n2-standard-4) with 12 replicas spread across 3 zones. Peak traffic is 4,200 req/sec during flash sales. The platform team schedules weekly Deployment image updates every Tuesday at 02:00 UTC. Before introducing PDBs, a single bad rollout drained 8 of 12 pods simultaneously when the cluster autoscaler replaced a node and the Deployment maxUnavailable default allowed too much churn.

Problem

During a routine Kubernetes upgrade of the checkout-api Deployment from v2.14.0 to v2.15.0, the combination of a voluntary node drain for OS patching and an aggressive rolling update strategy caused simultaneous pod terminations across availability zones. With only 4 pods remaining briefly, Redis connection pools exhausted, upstream payment gateway timeouts spiked to 18%, and P99 latency jumped from 120ms to 2.8 seconds for roughly 11 minutes. Incident postmortem revealed no guardrail prevented more than 50% of replicas from being unavailable at once. The SRE team also discovered that cluster-autoscaler scale-down events interacted badly with maxSurge=3, effectively doubling disruption during node consolidation. Stakeholders demanded a policy that guarantees minimum checkout capacity during both planned maintenance and application rollouts without blocking legitimate node upgrades entirely.

Solution

The team implemented a PodDisruptionBudget named checkout-api-pdb with spec.minAvailable set to 9, ensuring at least 75% of the 12 replicas stay running during voluntary disruptions such as node drains and rolling updates. They changed the Deployment strategy to RollingUpdate with maxSurge=2 and maxUnavailable=1 so only one pod terminates at a time while two new pods can start on spare capacity. Readiness probes were tightened to require successful health checks against the payment sandbox endpoint before marking pods ready, preventing traffic shift to half-initialized containers. For node maintenance, operators used kubectl drain with --ignore-daemonsets and verified PDB status showed ALLOWED disruptions=3 before proceeding. They documented a runbook tying PDB minAvailable to SLO math: 9 pods at 350 req/sec each sustains 3,150 req/sec, above the 3,000 req/sec contractual minimum. Integration tests in staging validated that a simultaneous drain of one node plus a Deployment rollout completed in 22 minutes without breaching availability. Alertmanager rules fired when available replicas dropped below 10 for more than 60 seconds, giving on-call early warning during edge cases.

Commands

kubectl apply -f checkout-api-pdb.yaml

kubectl get pdb checkout-api-pdb -n production -o wide

kubectl describe deployment checkout-api -n production | grep -A5 Strategy

Context

A video transcoding microservice on EKS processes queue-backed jobs. Baseline load is 800 jobs/hour across 6 worker pods (c6i.2xlarge nodes). CPU-based HPA kept 6–8 replicas because workers idle-wait on I/O; meanwhile RabbitMQ queue depth regularly hit 15,000 messages during upload spikes, causing 45-minute processing backlogs.

Problem

The transcoding workers appeared healthy by CPU metrics—typically 25–35% utilization—while the real bottleneck was unconsumed queue depth. The existing HorizontalPodAutoscaler targeted average CPU at 70%, scaling from 6 to 8 pods during spikes that actually required 18–22 workers. Product teams reported SLA misses: 1080p transcodes promised within 10 minutes regularly took 38 minutes when evening upload traffic surged. Attempts to lower CPU targets caused thrashing: pods scaled up, pulled large container images, then scaled down before processing meaningful work. Custom business metrics were not exposed to the metrics API server. Without queue-depth-driven scaling, the team over-provisioned static replicas to 14 during peak hours, wasting roughly $4,200/month in compute. Leadership required autoscaling tied to RabbitMQ ready message count with scale-up latency under 90 seconds.

Solution

Engineers deployed prometheus-adapter to surface RabbitMQ queue depth as a custom metric transcoding_queue_depth. They annotated the worker Deployment with autoscaling alpha annotations initially, then migrated to autoscaling/v2 HPA referencing external metrics from Prometheus. The HPA spec set minReplicas=6, maxReplicas=24, with two metrics: external metric transcoding_queue_depth targetAverageValue=800 (messages per pod) and a secondary CPU utilization at 75% as a ceiling guard. KEDA was evaluated but the team standardized on prometheus-adapter for consistency with existing observability. Scale-up behavior used stabilizationWindowSeconds=0 with policies adding up to 4 pods per 60 seconds; scale-down used 300-second stabilization to prevent flapping. RabbitMQ exporter scraped queue transcoding.jobs.ready every 15 seconds. Load tests injected 20,000 messages and verified HPA reached 22 replicas within 75 seconds. Grafana dashboards correlated replica count, queue depth, and job completion rate for tuning. Adapter config mapped Prometheus query sum(rabbitmq_queue_messages_ready{queue="transcoding.jobs"}) to the external metrics API with proper namespace and label selectors. RBAC granted the adapter extension-apiserver-authentication-reader and custom-metrics-resource-reader ClusterRoles. Staging burn-in ran two weeks comparing recommended replica counts against manual operator decisions before production cutover.

Commands

kubectl apply -f prometheus-adapter-config.yaml

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/media/jobs/transcoding_queue_depth" | jq

kubectl autoscale deployment transcoding-worker --custom-metric=transcoding_queue_depth --target-value=800 --min=6 --max=24 -n media

Context

A SaaS analytics dashboard serves 18 microservices behind NGINX Ingress on a 5-node AKS cluster. Public hostname pattern *.analytics.example.com handles 1,350 req/sec combined. Previously operators manually uploaded TLS certificates from a commercial CA every 90 days; two outages occurred when certs expired on internal staging hosts that mirrored production routes.

Problem

Manual certificate lifecycle did not scale as the team grew from 8 to 34 Ingress resources across dev, staging, and production namespaces. Expired certificates on staging-api.analytics.example.com caused 502 errors for 6 hours before detection because monitoring checked only production hostnames. Wildcard cert sharing violated security policy after a tenant isolation audit—each customer subdomain required distinct certificates. Renewals required change tickets, kubectl secret creation, and Ingress annotation updates, averaging 45 minutes per cert. Let's Encrypt rate limits were hit during a botched automation script that requested 12 duplicate certs in one hour. Developers needed self-service TLS for preview environments without sharing cluster-wide secrets. The platform team mandated automated ACME HTTP-01 or DNS-01 issuance with 30-day renewal alerts and zero manual kubectl secret edits for standard routes.

Solution

The team installed cert-manager v1.14 via Helm with a ClusterIssuer letsencrypt-prod using DNS-01 against Azure DNS for wildcard support on *.analytics.example.com subdomains. Each Ingress received the annotation cert-manager.io/cluster-issuer: letsencrypt-prod and tls blocks referencing secret names matching hostnames. For internal-only services, a separate ClusterIssuer used HTTP-01 via the public Ingress class nginx-external. cert-manager Certificate resources were templated in Helm charts so application teams only declared hosts in values.yaml. A Kyverno policy rejected Ingress objects missing TLS sections in production namespaces. PrometheusRule alerts fired when certificate expiry was under 21 days. Staging used letsencrypt-staging issuer for CI validation before promoting charts. After migration, 34 Certificates reached Ready=True within 4 minutes of Ingress creation during a test deploy of 6 new preview environments. Azure DNS service principal credentials were stored in cert-manager-solver-credentials sealed secrets with quarterly rotation. NGINX Ingress controller was configured to reload gracefully on secret rotation events without dropping active connections. A runbook documented troubleshooting for Challenge failures including DNS propagation delays and incorrect zone IDs. Certificate duration was set to 90 days with renewBefore 720h to allow two retry cycles before expiry.

Commands

kubectl apply -f clusterissuer-letsencrypt-prod.yaml

kubectl get certificates -A

kubectl describe certificate dashboard-tls -n production

Context

A healthcare data platform hosts 8 hospital tenants on a shared OpenShift 4.14 cluster with 12 worker nodes (m5.2xlarge). Each tenant receives a dedicated namespace (tenant-hosp-01 through tenant-hosp-08) running 5–12 pods apiece. Regulatory audit required proof that Tenant A cannot reach Tenant B's services or shared PHI storage endpoints without explicit allowlists.

Problem

Initial deployment relied on namespace naming conventions and RBAC alone, which auditors rejected because default Kubernetes networking allows any pod to reach any pod cluster-wide. A penetration test demonstrated that a compromised debug pod in tenant-hosp-03 could curl tenant-hosp-07's PostgreSQL service on port 5432 and reach an unprotected internal API on 10.96.0.0/12. Shared services—logging (Fluent Bit), metrics (Prometheus), and a central identity provider—needed selective access from all tenants without opening lateral movement paths between tenant namespaces. Some legacy apps used hostNetwork pods, bypassing standard NetworkPolicy until remediated. The CNI (OVN-Kubernetes) supported NetworkPolicy but none were defined. Compliance deadline was 60 days with potential contract loss covering 240,000 patient records across tenants.

Solution

Platform engineers adopted a default-deny posture: each tenant namespace received a NetworkPolicy deny-all ingress and egress, then layered explicit policies. Allow ingress from the shared ingress-nginx namespace to tenant frontend pods on port 8080. Allow egress to kube-system CoreDNS on UDP 53, to observability namespace on TCP 9090 and 9200, and to shared-auth on TCP 443. Deny all other cross-namespace traffic. Tenant-to-tenant communication was prohibited unless a bilateral policy was approved via ticket. They deployed a policy generator in CI that rendered NetworkPolicy YAML from a tenant registry CSV. Non-compliant hostNetwork workloads were migrated to standard pod networking over 3 sprints. Quarterly audits used netpol-test DaemonSet pods to verify connectivity matrices. Documentation mapped each policy to HIPAA technical safeguard 164.312(e). Rollout occurred namespace-by-namespace during maintenance windows with connectivity smoke tests. Egress to external FHIR endpoints was restricted to a shared egress gateway namespace with IP allowlisting at the firewall layer. Calico network sets were avoided in favor of native NetworkPolicy for portability. Each tenant onboarding checklist included automated netpol conformance tests that ran 47 connectivity assertions before namespace promotion to production status.

Commands

kubectl apply -f policies/tenant-hosp-03-default-deny.yaml

kubectl get networkpolicy -n tenant-hosp-03

kubectl exec -n tenant-hosp-03 debug-pod -- curl -m 2 http://api.tenant-hosp-07.svc.cluster.local:8080

Context

An internal ERP system migrated from a managed RDS instance to in-cluster PostgreSQL 15 for cost control on a 4-node RKE2 cluster using Longhorn storage. The database serves 420 concurrent connections from 9 application pods and handles 1,100 transactions/sec at month-end close.

Problem

Early attempts used a single-replica Deployment with a ReadWriteOnce PVC, causing data corruption risk when Kubernetes rescheduled the pod to another node while the volume remained attached elsewhere. Rolling updates restarted the only postgres pod without ordered recovery, once producing 22 minutes downtime during a minor version bump. Without stable network identity, application connection strings using Service DNS worked but backup sidecars could not rely on predictable pod names for WAL archiving to S3. Replication was required for RPO under 5 minutes, yet Deployments lack ordinal identity needed for primary/replica topology. Storage class defaults allowed volume expansion but not guaranteed IOPS; month-end batch jobs caused disk latency spikes to 45ms avg, breaching app timeouts. DBAs demanded StatefulSet semantics: stable pod names, ordered rollout, and per-replica PVCs.

Solution

The team deployed PostgreSQL via StatefulSet postgres with replicas=3, serviceName=postgres-headless, and volumeClaimTemplates requesting 200Gi from longhorn-fast StorageClass with 6,000 IOPS. Pod postgres-0 ran as primary; postgres-1 and postgres-2 were streaming replicas using Patroni for leader election and automatic failover. PodManagementPolicy was Parallel for replicas 1–2 but ordered readiness for postgres-0 during initial bootstrap. Applications connected through Service postgres-primary (selector role=primary) on port 5432. Backup used pgBackRest sidecar on each pod with WAL push to S3; stable pod names enabled consistent stanza configuration. Updates applied by incrementing StatefulSet pod template and rolling one ordinal at a time: kubectl rollout status statefulset/postgres. Init containers verified PVC mount permissions and fsGroup 999. Anti-affinity spread postgres pods across 3 nodes minimum. Connection pooling moved to PgBouncer Deployment with transaction mode, reducing raw connections from 420 to 80 on the primary. Synchronous replication was configured with synchronous_standby_names set to ANY 1 for RPO zero on approved maintenance windows. Monitoring via postgres_exporter tracked replication lag, bloat, and checkpoint frequency with alerts at lag exceeding 5 seconds.

Commands

kubectl apply -f postgres-statefulset.yaml

kubectl get pods -l app=postgres -o wide

kubectl exec postgres-0 -- patronictl list

Context

A genomics research org runs nightly batch pipelines on GKE Standard with 3 baseline nodes (n2-highmem-8) and node pool batch-workers autoscaling 0–20 nodes. Typical jobs spawn 40–80 pods requiring 16Gi RAM each; monthly full-genome reprocessing adds 600 Job pods completing over 6 hours.

Problem

Without cluster autoscaler, jobs queued for hours when pending pods exceeded schedulable capacity on 3 nodes—roughly 12 large pods max after system overhead. Researchers manually scaled the node pool via Terraform before big runs, often over-provisioning 18 nodes for 72 hours at $1,900 extra per event. When they forgot, SLAs slipped: a 600-pod run stayed Pending for 4.2 hours until on-call noticed Slack alerts. Conversely, leaving 15 idle nodes wasted budget. Some pods stayed Pending due to insufficient CPU despite free memory because the scheduler packs by requests, not actual usage. Cluster autoscaler was installed but misconfigured: max-node-provision-time was default and expander was random, causing suboptimal zone balance. GPU node pools were excluded incorrectly from autoscaling annotations.

Solution

Platform team reconfigured cluster-autoscaler with --nodes=1:20 for batch-workers, expander=least-waste, and skip-nodes-with-local-storage=false after validating Longhorn topology. Node pool tags gained gke-cluster-autoscaler-enabled=true. Pod resource requests were right-sized so 600 jobs each requesting 14Gi RAM and 4 CPU could schedule accurately. PriorityClasses ensured critical control-plane adjacent workloads preempted best-effort test jobs. They set PodDisruptionBudgets on long-running services separately so batch scale-down did not evict production namespaces sharing the default pool—batch moved to dedicated tainted pool batch-workload=true. Autoscaler scale-down-delay-after-add was 10m to keep nodes warm through bursty Job creation. Monitoring tracked cluster_autoscaler_unschedulable_pods and scale-up latency. Terraform managed min/max; CA handled real-time. Job manifests added nodeSelector and tolerations matching the batch pool taint so the scheduler signaled CA correctly. Over-provisioning using pause pods was rejected in favor of accurate requests after VPA analysis on representative genome pipeline containers. Scale-down utilization threshold was tuned to 0.5 so nodes drained only when half their allocatable capacity sat idle for 10 minutes. Weekly reports compared CA scale events against cloud billing line items to validate savings.

Commands

kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

kubectl get nodes -l cloud.google.com/gke-nodepool=batch-workers

kubectl describe pod genome-job-abc123 | grep -A3 Events

Context

A fintech payments team manages 47 Kubernetes manifests across 3 environments on EKS. Release frequency increased from biweekly to daily; 6 engineers commit to a mono-repo. Production runs 28 Deployment replicas handling 6,800 req/sec with strict change audit requirements.

Problem

kubectl apply from CI pipelines caused configuration drift: staging matched Git but production had 11 hand-patched ConfigMaps from firefighting sessions. Rollbacks took 25–40 minutes because nobody trusted which manifest version was live. A bad ConfigMap edit pushed directly to production disabled fraud-check rules for 19 minutes, processing 3,200 transactions without screening. Compliance required every prod change linked to a Git SHA. Blue/green was deemed too costly at 56 replica pairs, but big-bang rollouts risked customer impact. Teams needed automated sync from Git, diff visibility, and canary analysis before full promotion. Existing CI only built images; it did not reconcile cluster state. Multiple kubectl contexts increased human error risk.

Solution

Argo CD was installed with App-of-Apps pattern: root app bootstrapped environment apps prod-payments, staging-payments, each tracking main with environment overlays in Kustomize. Production Application spec included syncPolicy.automated prune and selfHeal disabled initially—promotions required manual sync after CI checks. Image updates flowed via Argo CD Image Updater watching ECR tags matching ^v[0-9]+\.[0-9]+\.[0-9]+$. For progressive delivery, Argo Rollouts replaced Deployment on payment-api with canary steps: 10% traffic 5 min, 30% 10 min, 100%, analyzing Prometheus metrics error-rate < 0.5% and P99 < 400ms via AnalysisTemplate. Git became sole source of truth; kubectl apply blocked by OPA Gatekeeper except break-glass RBAC. Slack notifications integrated via Argo Notifications on sync success, degradation, and rollback. Drift detection ran every 3 minutes; 23 drift events were auto-healed in staging during the first month. SSO via OIDC mapped GitHub teams to Argo CD project roles so only payments-sre could sync production. Pre-sync hooks ran kubeconform validation and conftest policy checks against organizational Rego bundles. Rollback procedures documented both argo rollouts undo and Git revert with cherry-pick for hotfix branches. Audit logs exported to Splunk retained sync actor, commit SHA, and diff summary for SOC2 evidence.

Commands

kubectl get applications -n argocd

kubectl argo rollouts get rollout payment-api -n production

kubectl argo rollouts promote payment-api -n production

Context

A media streaming platform operates a 24-node bare-metal Kubernetes cluster (kubeadm, v1.29) serving 9,200 concurrent HLS sessions. Monthly kernel security patches require rolling node reboots. Each node hosts 35–42 pods including DaemonSets, CDN edge caches, and API gateways.

Problem

Early maintenance used kubectl drain without pod-specific preparation, causing abrupt CDN cache pod termination and 404 spikes for 180,000 users during one maintenance window. Default grace period 30 seconds was insufficient for connections draining on gateway pods handling 850 req/sec per node. PodDisruptionBudgets existed for APIs but not for cache tiers. DaemonSets blocked drains until operators used --ignore-daemonsets without documenting which system agents required post-reboot verification. One drain evicted 38 pods simultaneously when eviction budgets were misconfigured. Local SSD cache warmup after reboot took 14 minutes, during which origin fetch rate tripled and overloaded upstream object storage at 12 Gbps aggregate. Maintenance windows exceeded the allotted 4 hours, triggering customer SLA credits.

Solution

SRE built a maintenance runbook executed by a scripted cordon-drain-verify loop. Pre-step: scale CDN cache Deployment to add 20% surplus replicas cluster-wide. Annotate gateway pods with preStop hook sleeping 45 seconds after SIGTERM to allow load balancer deregistration from external HAProxy. Verify each critical PDB shows ALLOWED >= 1 before drain. Command sequence: kubectl cordon node-12; kubectl drain node-12 --ignore-daemonsets --delete-emptydir-data --grace-period=60 --timeout=20m. Post-drain: confirm node SchedulingDisabled, watch for Pending pods, run synthetic playback tests from 6 edge PoPs. After reboot: kubectl uncordon; wait for NodeReady; validate kubelet and CNI; run cache warmup job seeding top 10,000 objects. Maintenance moved to 02:00–06:00 local with status page updates. Only one node at a time; max 4 nodes per window. PodDisruptionBudget for gateway tier was temporarily raised from minAvailable 8 to 10 during drain windows. Eviction API rate was monitored to avoid thundering herd on etcd when 38 pods rescheduled simultaneously. A dedicated maintenance namespace hosted the drain-orchestrator Job with Leases for leader election so two operators could not drain overlapping nodes. Node problem detector alerts were silenced for planned reboots to reduce pager noise.

Commands

kubectl cordon worker-12.example.com

kubectl drain worker-12.example.com --ignore-daemonsets --delete-emptydir-data --grace-period=60 --timeout=20m

kubectl uncordon worker-12.example.com

Context

A logistics tracking API on a 7-node DigitalOcean Kubernetes cluster ran 16 replicas with requests set to 2 CPU / 4Gi during initial launch panic. Actual production metrics showed 0.3 CPU and 900Mi per pod at 2,100 req/sec aggregate. Monthly node cost was $3,780 for workloads that fit theoretically on 4 nodes.

Problem

Engineers copied resource requests from load-test peaks 18 months prior, before query optimization reduced CPU need by 70%. HPA scaled replica count based on CPU percentage of those inflated requests, keeping 16 pods when 7 sufficed. Scheduler marked nodes 78% allocated by requests while actual utilization averaged 22% CPU and 31% memory. New teams could not deploy to production namespaces due to artificial quota exhaustion. FinOps flagged $1,600/month waste. Down-sizing requests manually risked OOM kills without data-driven recommendations. Horizontal scaling masked vertical inefficiency. Leadership wanted automated recommendation and optional auto-apply in non-production first, with production requiring review. VPA admission webhook concerned the team due to pod restart behavior during updates.

Solution

They installed VPA (recommender, updater, admission-controller) in kube-system. Each target Deployment received a VPA object in recommendation-only mode initially: updateMode Off for 21 days to collect histograms. VPA recommended 450m CPU and 1.1Gi memory per pod for tracking-api. Staging switched to updateMode Auto with minAllowed 300m/768Mi and maxAllowed 1 CPU/2Gi bounds. After two weeks stable, production used updateMode Initial so only new pods picked right-sized requests without restarting running pods during business hours. Replicas were manually reduced from 16 to 9 after requests updated, then HPA minReplicas lowered to 5. Goldilocks dashboard exposed recommendations to developers. Change advisory board approved rollout with rollback manifests pinned. OOM monitoring alert fired zero times post-change over 60 days. VPA recommender interval was set to 1 minute with historyLength 8 days to capture weekly traffic patterns. Pod QoS class remained Burstable after right-sizing since limits stayed at 1 CPU and 2Gi. Cluster overcommit ratio improved from 3.5:1 requested-to-actual CPU to 1.4:1, freeing schedulable headroom for 22 new microservices without adding nodes.

Commands

kubectl apply -f vpa-tracking-api.yaml

kubectl describe vpa tracking-api-vpa -n logistics

kubectl get pods -n logistics -l app=tracking-api -o custom-columns=NAME:.metadata.name,CPU_REQ:.spec.containers[0].resources.requests.cpu,MEM_REQ:.spec.containers[0].resources.requests.memory

Context

A microservices mesh on a 15-node Kubernetes cluster (130 services, 890 pods) generated DNS lookup storms after gRPC adoption: each RPC could trigger 3–5 SRV/A lookups. CoreDNS defaulted to 2 replicas on 512m CPU limit, handling baseline 12,000 queries/sec cluster-wide with sporadic 5-second timeouts during deploy waves.

Problem

Application logs showed net.Dial timeouts to *.svc.cluster.local during rolling updates when 200 pods restarted simultaneously, each cold-starting DNS caches. CoreDNS metrics exposed coredns_dns_request_duration_seconds_count p99 at 4.2 seconds during incidents. The default Corefile forwarded all external queries upstream for every pod restart burst, saturating upstream resolver rate limits at 1,000 qps. ndots:5 in pod dnsConfig caused unnecessary search list expansions—5 search paths per lookup for partial names. Autoscaling was absent; manual replica patch to 4 helped briefly but CPU throttling at 512m limit caused drops. kube-dns autoscaler existed but max was 2 based on outdated node count formula cores=1 replica per 256 cores. Service mesh sidecars doubled DNS volume. SRE needed higher replica count, cache tuning, reduced search domains, and NodeLocal DNSCache for hot path optimization.

Solution

Team deployed NodeLocal DNSCache DaemonSet on all 15 nodes binding 169.254.20.10 via link-local IP, forwarding to CoreDNS ClusterIP for cache miss. CoreDNS ConfigMap updated with prometheus plugin, cache 9000 success TTL 30, and load balance plugin. Pod dnsConfig cluster-wide via mutating webhook set ndots:2 and searches limited to namespace.svc.cluster.local svc.cluster.local cluster.local. CoreDNS HPA scaled replicas 4–10 on CPU 70% and custom metric DNS QPS via prometheus-adapter. Resource limits raised to 1 CPU / 1Gi per CoreDNS pod. Liveness probes tuned. After changes, load test simulated 400 simultaneous pod starts generating 45,000 qps peak—NodeLocal absorbed 78% locally cached. Upstream forward policy added cache 30 for external zones. kubelet clusterDNS was patched per-node to 169.254.20.10 with clusterDomain unchanged. CoreDNS PodDisruptionBudget minAvailable 3 prevented scale-down below quorum during HPA fluctuations. Negative testing confirmed NXDOMAIN responses cached correctly without poisoning successful lookups. Documentation warned teams against hardcoding kube-dns ClusterIP in legacy ConfigMaps, providing migration examples for fully qualified service names.

Commands

kubectl edit configmap coredns -n kube-system

kubectl get hpa coredns -n kube-system

kubectl apply -f node-local-dns.yaml

Context

SaaS company running 200+ microservices on AKS spending K/month on on-demand nodes. Most services were stateless.

Problem

All pods ran on on-demand node pools. Spot instances are 60-80% cheaper but the team feared disruption from evictions.

Solution

Tiered scheduling: critical services (payment, auth) pinned to on-demand with nodeSelector. Stateless services use two deployments — 1 replica on-demand for guaranteed availability, rest on spot with tolerations. Added PodDisruptionBudgets for graceful eviction handling.

Commands

kubectl label nodes <node> node-pool=spot

kubectl taint nodes <node> spot=true:PreferNoSchedule

Context

A checkout API served steady daytime traffic but saw 5–8x spikes during flash sales. The team had provisioned a fixed 12 replicas — wasteful off-peak and still occasionally saturated at peak.

Problem

Fixed replica counts force a bad trade-off: size for peak and waste money 20 hours a day, or size for average and fall over during spikes. They needed capacity to track demand automatically.

Solution

They set CPU requests accurately (the HPA scales on utilization relative to requests), added a HorizontalPodAutoscaler targeting 60% CPU between 4 and 40 replicas, and later added a custom metric (requests-per-second via Prometheus Adapter) since CPU lagged the real signal. Scale-down stabilization was raised to avoid flapping.

Commands

kubectl autoscale deploy checkout --cpu-percent=60 --min=4 --max=40

kubectl get hpa checkout --watch

kubectl describe hpa checkout   # see current/target metrics and scaling events

Context

A payments service needed safer releases after a bad deploy caused a 20-minute partial outage. The default RollingUpdate had shifted all traffic to a broken version before anyone noticed.

Problem

RollingUpdate replaces pods gradually but still sends real traffic to every new pod immediately, so a subtle regression reaches 100% of users as the rollout completes — with no fast, clean rollback of *traffic*.

Solution

For most services they kept RollingUpdate but added strict readiness probes and maxSurge/maxUnavailable tuning plus a PodDisruptionBudget. For the payments service they adopted canary with Argo Rollouts: shift 5%→25%→50%→100% while watching error-rate/latency SLOs, auto-abort on breach. Blue-green (two full environments, switch the Service selector) was reserved for a stateful component needing instant, all-or-nothing cutover.

Commands

kubectl set image deploy/payments api=payments:v2 --record

kubectl rollout status deploy/payments

kubectl argo rollouts get rollout payments --watch

Context

A multi-team cluster ran dev, staging, and several product namespaces flat — by default every pod could reach every other pod, so a compromised staging pod could talk straight to production databases.

Problem

Kubernetes networking is allow-all by default. Without policy, lateral movement is trivial and PCI/segmentation requirements can't be met.

Solution

They rolled out a default-deny ingress policy per sensitive namespace, then added explicit allow rules only for the flows that should exist (e.g., api namespace may reach db namespace on 5432). They verified the CNI enforced policy (Calico/Cilium — the default kubenet does not) and tested with temporary debug pods.

Commands

kubectl label ns production tier=prod

kubectl apply -f default-deny-ingress.yaml

kubectl exec -it debug -- wget -qO- --timeout=3 db.production:5432   # should time out

Context

A partner integration needed to allowlist a fixed egress/ingress IP, but every time the team recreated the LoadBalancer Service the cloud assigned a new address, breaking the partner's firewall rule.

Problem

By default cloud LoadBalancer Services get an ephemeral IP that changes on recreation, which is incompatible with external allowlisting and DNS stability.

Solution

They pre-reserved a static IP in the cloud provider and pinned it via loadBalancerIP (or the provider-specific annotation on newer clouds), plus set externalTrafficPolicy: Local to preserve client source IPs where needed. The reserved address was documented and handed to the partner once.

Commands

gcloud compute addresses create web-ip --region us-central1

kubectl patch svc web -p '{"spec":{"loadBalancerIP":"34.x.x.x"}}'

kubectl get svc web -o wide   # EXTERNAL-IP should match the reserved address

Context

A new API image passed CI but introduced a memory regression that only surfaced under production load an hour after rollout. On-call needed to revert immediately without redeploying from source.

Problem

Rebuilding and redeploying the previous version from Git takes precious minutes during an incident, and getting the exact prior config right by hand is error-prone.

Solution

Because Deployments keep revision history, on-call rolled back to the previous known-good revision with a single command, then paused rollouts while the fix was prepared. They also set revisionHistoryLimit sensibly and standardized on --record/annotations so each revision was identifiable.

Commands

kubectl rollout history deploy/api

kubectl rollout undo deploy/api            # to previous revision

kubectl rollout undo deploy/api --to-revision=7

Context

A backend team of eight wanted every engineer to run the full microservice stack locally before pushing, without needing cloud access or stepping on a shared cluster.

Problem

Developing against a shared remote cluster caused collisions and slow feedback, and not everyone had cloud permissions. They needed a fast, disposable, production-like Kubernetes locally.

Solution

They standardized on Minikube with the Docker driver, scripted `minikube start` with pinned Kubernetes version and resource sizing, enabled the ingress and metrics-server addons, and pointed local image builds at Minikube's Docker daemon to skip a registry round-trip. A Makefile wrapped the whole flow.

Commands

minikube start --driver=docker --kubernetes-version=v1.30.2 --cpus=4 --memory=8g

minikube addons enable ingress && minikube addons enable metrics-server

eval $(minikube docker-env) && docker build -t myapp:dev .   # build straight into the cluster

Context

On-call engineers were escalating routine pod failures because they weren't sure which kubectl commands to run in what order, wasting time during incidents.

Problem

Ad-hoc `kubectl get pods` alone rarely explains a failure, and inconsistent debugging steps meant slow, inconsistent triage across the team.

Solution

They codified a triage runbook: describe the pod for events, read current and previous logs, check resource usage and node health, then exec or use an ephemeral debug container for live inspection. The sequence was turned into a one-page cheat sheet and a `kubectl` alias set.

Commands

kubectl describe pod <pod> | sed -n '/Events/,$p'

kubectl logs <pod> --previous --tail=200

kubectl debug -it <pod> --image=busybox --target=<container>

Context

A team had a dozen internal web apps, each initially exposed with its own cloud LoadBalancer Service, driving up cost and leaving TLS handled inconsistently per app.

Problem

One LoadBalancer per app is expensive and operationally messy, and each team was terminating TLS differently (or not at all).

Solution

They deployed a single NGINX Ingress controller behind one LoadBalancer, moved every app to a ClusterIP Service, and routed by hostname via Ingress objects. cert-manager issued and auto-renewed Let's Encrypt certificates, centralizing TLS. DNS pointed all hostnames at the one LB.

Commands

helm install ingress-nginx ingress-nginx/ingress-nginx

kubectl apply -f cluster-issuer-letsencrypt.yaml

kubectl get ingress -A   # host -> service routing, TLS secret per host

Context

Batch jobs and evening traffic peaks left pods Pending for lack of nodes, while a statically-sized node pool sat half-idle overnight — paying for capacity nobody used.

Problem

A fixed node count can't be right for both peak and trough: it either strands Pending pods at peak or wastes money off-peak.

Solution

They enabled Cluster Autoscaler on the node group so Pending pods triggered scale-up and underused nodes were drained and removed after a cooldown. Later they moved to Karpenter, which provisions right-sized nodes just-in-time from a broader instance set, improving bin-packing and spot usage.

Commands

kubectl get pods --field-selector=status.phase=Pending -A

kubectl -n kube-system logs deploy/cluster-autoscaler | grep -i scale

kubectl get nodes -L node.kubernetes.io/instance-type   # watch capacity change

Context

A service hard-coded config and read a database password from a baked-in file, so promoting the same image from dev to prod meant rebuilding it, and the password sat in the image layers.

Problem

Config and secrets baked into images break the 'build once, promote everywhere' principle and leak sensitive data into image history and Git.

Solution

They moved non-sensitive settings to ConfigMaps and the password to a Secret, injected both as env vars/mounted files, and kept one image across environments with per-environment ConfigMaps/Secrets. For real protection they enabled etcd encryption-at-rest and adopted the External Secrets Operator to sync secrets from a cloud secrets manager, so the source of truth never lived in Git or etcd.

Commands

kubectl create configmap app-config --from-literal=LOG_LEVEL=info

kubectl create secret generic db --from-literal=password='...'

kubectl set env deploy/app --from=configmap/app-config

Context

Every CI pipeline and app used the namespace 'default' ServiceAccount, and several humans had cluster-admin 'to make things work' — so a leaked CI token or a compromised pod had broad cluster access.

Problem

Shared, over-privileged identities mean any single compromise (a CI token, one pod) can read secrets or modify workloads across the cluster.

Solution

They gave each workload and pipeline its own ServiceAccount bound to a narrow Role scoped to just what it needed, removed standing cluster-admin in favor of time-bound elevation, set automountServiceAccountToken: false where pods never call the API, and audited access with `kubectl auth can-i`. Cloud workloads used Workload Identity/IRSA instead of static cloud keys.

Commands

kubectl create serviceaccount deployer -n ci

kubectl create rolebinding ci-deploy --role=deployer --serviceaccount=ci:deployer -n staging

kubectl auth can-i --list --as=system:serviceaccount:ci:deployer -n staging