Kubernetes Intermediate Questions

Think of an OpenShift upgrade like renovating a multi-story building floor by floor while tenants keep living in it. You don't shut the whole building down — you start with the management office (control plane), then renovate one floor at a time (worker nodes), making sure tenants can always use at least some floors.

The upgrade starts with the Cluster Version Operator (CVO), which is the orchestrator. When you initiate an upgrade (via console or oc adm upgrade), the CVO first validates the upgrade path — not all version jumps are supported. For example, 4.14 can go to 4.15 but not directly to 4.17. The CVO checks the upgrade graph from Red Hat's Cincinnati service to confirm the path is valid and no known blocking issues exist.

Once validated, the CVO begins updating Cluster Operators one by one in dependency order. These are the built-in platform services: the authentication operator, ingress operator, monitoring operator, machine-config operator, and about 30 others. Each operator updates its own managed components (containers, configs, RBAC). The CVO waits for each operator to report 'Available=True, Degraded=False' before proceeding to the next. If any operator gets stuck, the entire upgrade pauses.

Next, the control plane nodes upgrade. The Machine Config Operator (MCO) renders a new MachineConfig for the master MachineConfigPool, which includes the new kubelet version, CRI-O version, and OS packages. Master nodes are updated one at a time (the MCO respects maxUnavailable, defaulting to 1 for masters). Each master is cordoned, drained, rebooted with the new config, and uncordoned. Since etcd requires quorum (2 of 3 nodes), only one master can be down at a time.

Finally, worker nodes upgrade through the worker MachineConfigPool. The MCO renders the new MachineConfig, then updates workers according to the MCP's maxUnavailable setting (default is 1). Each worker is cordoned (no new Pods scheduled), drained (existing Pods evicted respecting PDBs), rebooted with new OS and configs, and uncordoned. This is where PDB issues most commonly stall upgrades — if a PDB prevents all Pods from being evicted, the drain hangs indefinitely.

The most common upgrade failure: a worker node stuck in 'draining' because a PodDisruptionBudget prevents eviction. The MCO logs and oc get nodes will show the node as SchedulingDisabled. You need to identify the blocking PDB with oc get pdb -A and either adjust it or manually handle the stuck Pods.

Think of a PDB like a fire safety regulation for a building. The rule might say 'at least 2 of 3 emergency exits must remain open at all times.' During a renovation, workers can close one exit at a time. But if they try to close two simultaneously, the fire marshal blocks the work until one reopens. The regulation doesn't create more exits — it just prevents too many from being closed at once.

A Pod Disruption Budget tells Kubernetes: 'during voluntary disruptions, you must keep at least N pods of this application running (minAvailable) or take down at most M pods at a time (maxUnavailable).' Voluntary disruptions include node draining (kubectl drain), cluster upgrades, Cluster Autoscaler scale-down, and Karpenter consolidation. PDBs do NOT protect against involuntary disruptions like node crashes, OOM kills, or hardware failures — those bypass PDBs entirely.

Here's the mechanism: when Kubernetes needs to evict a Pod (during a node drain), it calls the Eviction API. The Eviction API checks all PDBs that select this Pod. If evicting it would cause the number of healthy Pods to drop below minAvailable (or the number of unavailable Pods to exceed maxUnavailable), the eviction is denied. The drain operation retries, but if the PDB condition never resolves, the drain hangs forever.

This is the #1 cause of stuck OpenShift upgrades. Real scenario: you have an application with 5 replicas and a PDB of minAvailable=5. During an upgrade, the MCO tries to drain a worker node. One of the 5 Pods is on that node. Evicting it would drop healthy Pods to 4, violating minAvailable=5. The drain is blocked. The node stays in SchedulingDisabled state. The MachineConfigPool shows UPDATING=True but never completes. The entire upgrade stalls waiting for this one node.

The fix depends on the situation: (1) if the PDB is too aggressive (minAvailable equals replica count), reduce it to allow at least one disruption, (2) if the application can scale up temporarily, increase replicas so there's headroom, (3) as a last resort in an emergency, delete the PDB temporarily, let the drain complete, then recreate it. Option 3 is risky because the application briefly loses PDB protection.

A critical interview point: PDBs do NOT create additional pods or nodes. They are purely a constraint — a gate that blocks eviction requests. They're essential for production workloads (preventing rolling updates from taking too many replicas offline), but must be carefully tuned relative to replica count. The rule of thumb: maxUnavailable should be at least 1, or minAvailable should be at most (replicas - 1).

Think of MachineConfig like a building blueprint that specifies how every apartment on a floor should be configured — what appliances are installed, what the thermostat settings are, what fixtures are in the bathroom. Instead of a maintenance worker going door-to-door making changes (SSH into each node), you update the blueprint and the building management system (MCO) automatically applies changes to every apartment matching that floor plan. If you manually changed a thermostat, the system would revert it on the next maintenance cycle.

A MachineConfig is a Kubernetes custom resource that describes node-level configuration: kernel parameters (sysctl settings), systemd unit files, files to write to the node filesystem (like chrony.conf for NTP), container runtime settings, and OS-level packages. When you create or modify a MachineConfig, the Machine Config Operator (MCO) renders it into a final configuration, combines it with other MachineConfigs, and produces a 'rendered' config that represents the complete desired state of a node.

A MachineConfigPool (MCP) groups nodes and determines which MachineConfigs apply to them. OpenShift has three default MCPs: 'master' (control plane nodes), 'worker' (compute nodes), and optionally 'infra' (infrastructure nodes running routers, monitoring). Each MCP has a node selector that matches nodes by labels. When a MachineConfig changes, the MCO identifies which MCP is affected, then rolls out the update to nodes in that pool — cordoning, draining, applying the config, rebooting, and uncordoning each node.

The MCP controls the rollout strategy via maxUnavailable. For masters, this is always 1 (etcd quorum). For workers, you can increase it to update multiple nodes in parallel — balancing speed against risk. If you set maxUnavailable=3 on a 10-node worker pool, three nodes update simultaneously, reducing upgrade time but accepting that 30% of capacity is temporarily offline.

Why not just SSH in and make changes? Because OpenShift uses immutable infrastructure. The MCO continuously reconciles actual node state against the rendered MachineConfig. If you SSH in and change a sysctl value, it works temporarily — but on the next node reboot or MCO reconciliation, your change gets overwritten. Manual changes are also not auditable, not version-controlled, and not consistently applied across nodes. MachineConfig gives you GitOps-compatible, declarative, auditable node management.

A common pitfall: creating a MachineConfig with a syntax error in a systemd unit or an invalid kernel parameter can render nodes unbootable. The MCO will keep trying to apply it, and nodes may get stuck in a reboot loop. Always test MachineConfig changes on a non-production cluster or a dedicated MCP with one test node first.

Think of it like two different approaches to upgrading a fleet of delivery trucks. The EKS approach is like buying new trucks (new node group), gradually transferring packages and drivers to the new trucks, and then selling the old ones. The OpenShift approach is like driving each truck to the shop one at a time, upgrading the engine and transmission while the rest of the fleet keeps delivering, then putting it back on the road.

With EKS, the control plane upgrade is managed by AWS — you click 'upgrade' or run aws eks update-cluster-version, and AWS handles upgrading the API server, etcd, and other control plane components behind the scenes (it's a managed service, so you don't touch the masters). For worker nodes, the standard practice is to create a new managed node group running the target Kubernetes version, cordon/drain the old node group, let workloads reschedule onto new nodes, then delete the old node group. Some teams use Karpenter, which handles this more dynamically by replacing nodes as they're drained.

With OpenShift, everything is in-place. The Cluster Version Operator orchestrates the entire upgrade: Cluster Operators update first, then the Machine Config Operator updates master nodes one by one (cordon → drain → reboot with new RHCOS version → uncordon), then worker nodes through MachineConfigPools. The nodes themselves are upgraded — same machines, new OS and packages. You don't create new infrastructure.

Key differences in practice: EKS upgrades are more forgiving — if something goes wrong with the new node group, the old one is still running, so you can roll back by rerouting traffic. OpenShift upgrades are harder to roll back because nodes are modified in-place (though OpenShift does support limited rollback). EKS requires you to manage the node upgrade strategy yourself (or use tools like eksctl, Karpenter, or Terraform), while OpenShift's MCO handles node lifecycle automatically.

EKS typically has less downtime risk because old and new nodes coexist during migration. OpenShift has a tighter upgrade window because each node goes offline briefly for reboot. However, OpenShift's approach is simpler operationally — you run one command and the platform handles everything. EKS gives more flexibility but requires more orchestration effort.

A gotcha specific to EKS: you must upgrade add-ons (VPC CNI, CoreDNS, kube-proxy) separately from the control plane, and version compatibility matrices matter. OpenShift bundles everything together, reducing the chance of version mismatches.

Think of it like a committee that makes decisions by majority vote. If you have 3 committee members, you need 2 to agree (majority) to pass any decision. If one member is sick, the remaining 2 can still vote and make decisions. But if you only had 2 members and one got sick, you'd have 1 out of 2 — not a majority — so no decisions can be made and everything stops.

The reason is etcd, the distributed key-value store that holds all Kubernetes cluster state. Etcd uses the Raft consensus algorithm, which requires a strict majority of members to agree on any write. This majority is called quorum. For 3 members, quorum = 2 (you can lose 1). For 5 members, quorum = 3 (you can lose 2). For 2 members, quorum = 2 (you can lose 0 — making 2 members WORSE than 1 for availability).

When one control plane node fails in a 3-node setup, here's what happens: etcd continues operating because 2 of 3 members still form quorum. The API server pods on the remaining 2 nodes handle all requests (the load balancer in front of them routes around the failed node). The scheduler and controller manager use leader election — one was active, the others were on standby. If the active leader was on the failed node, a new leader is elected within seconds. From the user's perspective, kubectl commands might have a brief hiccup (~5-10 seconds) during leader re-election, but the cluster continues operating normally.

However, losing TWO of three control plane nodes is catastrophic: etcd loses quorum (only 1 of 3 remaining), and all writes fail. The API server can serve reads from the remaining etcd member but cannot process any creates, updates, or deletes. Existing workloads on worker nodes keep running (the kubelet continues managing pods independently), but you cannot deploy anything new, scale, or recover pods that fail. The cluster is in a read-only degraded state until quorum is restored.

Why not 5 or 7 control plane nodes? Each etcd write must be acknowledged by a majority before it's committed. More members means more network round trips and higher write latency. For most clusters, the trade-off of 3 nodes (survive 1 failure, fast writes) is optimal. Large enterprise clusters sometimes use 5 nodes for extra resilience, but 7+ is almost never justified because the write performance penalty outweighs the marginal availability gain.

Think of this like a locked filing cabinet. Only one person can have the key at a time. If the person holding the key (the stuck Pod) becomes unresponsive, you can't just give a copy of the key to someone else — that would mean two people writing to the same files simultaneously, causing corruption. You have to get the original key back first (confirm the original Pod is fully stopped), then hand it to the replacement.

When a Pod using a ReadWriteOnce PersistentVolume becomes not ready but the node is still running, here's the sequence: the kubelet marks the Pod as not ready, the readiness probe fails, and the Pod is removed from Service endpoints. However, the volume remains attached to the node and mounted in the Pod's filesystem. Kubernetes will not detach a volume from a node while any Pod on that node still references it — even if that Pod is in a bad state.

If the Pod is stuck in Terminating (the node acknowledges the delete but the process won't stop), you can try kubectl delete pod --grace-period=0 --force. But this only removes the Pod from the API server — it does NOT guarantee the process is stopped on the node. The kubelet still needs to kill the container. If the node itself is unreachable (network partition, kernel hang), the Pod stays in Terminating indefinitely because the API server can't confirm the container is gone.

This is where it gets dangerous: if you force-delete the Pod and the scheduler creates a replacement on a different node, Kubernetes tries to attach the volume to the new node. But the volume is still attached to the old node. Cloud providers (AWS, GCP) will deny the attach because the volume is in-use. The new Pod sits in ContainerCreating with a 'Multi-Attach error' until the old attachment is released. On AWS EBS, this force-detach can take 5-10 minutes, and if the old node was writing data at the moment of detach, you risk filesystem corruption.

The safe recovery process: (1) Confirm the old node's status — if the node is reachable, let the kubelet handle cleanup. (2) If the node is unreachable, verify the node is truly dead (not just network-partitioned). (3) Only after confirming the node won't write to the volume, delete the VolumeAttachment object or force-detach from the cloud console. (4) Wait for the volume to show as Available before letting the new Pod start. (5) After attachment, run filesystem checks if corruption is suspected.

For StatefulSets specifically, Kubernetes is extra cautious: it will NOT create a replacement Pod until the old Pod is confirmed deleted (including volume cleanup). This 'at-most-one' guarantee prevents dual-mount scenarios but means recovery takes longer.

Think of a restaurant kitchen with a ticket system. In a push model, the waiter walks the order directly to the chef. In a pull model, the chef watches a ticket board and picks up new orders as they appear. Both get the food made, but the pull model means the chef always knows the current state of all orders without anyone having to push each one individually.

In Kubernetes, deployments typically follow one of two patterns. Push-based CI/CD tools like Jenkins or GitHub Actions run kubectl apply or helm upgrade as a pipeline step, directly sending manifests to the cluster API server. Pull-based GitOps tools like ArgoCD or Flux watch a Git repository and automatically reconcile the cluster state with what is declared in the repo. Many teams use a hybrid: CI builds and pushes images, then updates a Git repo, which triggers ArgoCD to deploy.

Internally, a typical pipeline has stages: code checkout, unit tests, Docker build with a tagged image (using git SHA or semantic version), image push to ECR or another registry, manifest update (either in-pipeline or via a Git commit to an infra repo), and deployment to the target cluster. For Kubernetes specifically, the Deployment controller handles rolling updates by creating a new ReplicaSet, scaling it up, and scaling the old one down. The pipeline may also run integration tests, smoke tests, or canary analysis after deployment.

At production scale, teams separate their CI (build and test) from CD (deploy). The CI pipeline produces a versioned artifact. The CD pipeline or GitOps controller handles promotion across environments: dev, staging, UAT, production. Environment-specific values are managed through Helm values files, Kustomize overlays, or ArgoCD ApplicationSets. Teams monitor deployment success through rollout status checks, readiness probe results, and automated rollback on failure.

The non-obvious gotcha is that push-based deployments can drift from the declared state if someone makes manual kubectl changes. GitOps tools detect and correct this drift automatically, but they add complexity around secret management and multi-cluster configuration. Teams that start with push-based pipelines often migrate to GitOps as the number of clusters and services grows beyond what manual pipeline management can handle reliably.

Think of moving into a new apartment. Common problems are the moving truck arriving at the wrong address (image pull errors), the apartment not having enough power outlets (resource limits), the door lock not matching your key (RBAC errors), and the furniture not fitting through the doorway (resource quota). Each problem looks different but follows a predictable troubleshooting pattern.

In Kubernetes CI/CD, deployment failures cluster around a few categories. Image-related failures happen when the image tag does not exist in the registry, registry credentials are expired, or the image was pushed to a different repository than the manifest references. Resource failures occur when the namespace has a ResourceQuota and the new deployment exceeds CPU or memory limits. Configuration failures happen when a ConfigMap or Secret referenced by the pod does not exist in the target namespace or has different keys than the application expects.

The troubleshooting sequence is predictable. First, check the Deployment rollout status with kubectl rollout status. If the rollout is stuck, describe the Deployment to see the events. Then check the ReplicaSet events to see why new pods are not being created. If pods exist but are not ready, check pod events with kubectl describe pod, then check container logs with kubectl logs. For image pull errors, the events will show ErrImagePull or ImagePullBackOff with a specific error message. For resource issues, the events will show FailedScheduling or quota exceeded messages.

At production scale, the most impactful issues are deployments that pass in lower environments but fail in production. This usually happens because of environment-specific differences: different resource quotas, different network policies blocking connectivity, different secrets or certificates, or different node configurations. Teams prevent this by making environments as similar as possible, using the same Helm chart with different values, and running integration tests in a staging environment that mirrors production networking and security policies.

The non-obvious gotcha is that a deployment can appear successful — kubectl rollout status reports completion — but the application is still broken. This happens when readiness probes are too lenient (checking only that the HTTP port is open, not that the application can actually serve requests). Teams should use deep health checks that verify database connectivity, downstream service availability, and application-specific readiness before marking a pod as ready.

Think of an overcrowded bus. When the bus exceeds its weight limit, the driver must ask some passengers to leave. Passengers without tickets (BestEffort pods) are asked first, then those with partial tickets (Burstable pods), and finally full-fare passengers (Guaranteed pods) are the last to go. The bus driver does not choose randomly — there is a clear priority order based on who has the strongest claim to stay.

In Kubernetes, pod eviction is the kubelet's mechanism for protecting node stability. When a node runs low on a critical resource — memory, temporary (ephemeral) storage, or process IDs — the kubelet begins evicting pods to reclaim that resource. This is different from preemption (which is the scheduler removing lower-priority pods to make room for higher-priority ones) and different from API-initiated eviction (which is used during node drain for maintenance).

Internally, the kubelet monitors resource usage against configurable eviction thresholds. The default soft eviction threshold for memory is memory.available < 100Mi, and for disk is nodefs.available < 10%. When a threshold is breached, the kubelet sets the corresponding node condition (MemoryPressure, DiskPressure, PIDPressure) and begins ranking pods for eviction. The ranking uses QoS class: BestEffort pods (no resource requests or limits) are evicted first, Burstable pods (requests set but lower than limits) are evicted next based on how much they exceed their requests, and Guaranteed pods (requests equal limits for all containers) are evicted last.

At production scale, the most common eviction cause is ephemeral storage exhaustion from container logs, emptyDir volumes, or container writable layers growing unbounded. Memory-based evictions happen when applications have memory leaks or when resource limits are set too low for actual workload requirements. Teams should monitor node conditions, set appropriate resource requests and limits to ensure critical pods get Guaranteed QoS, configure log rotation to prevent disk pressure, and use PodDisruptionBudgets to limit the impact of evictions on service availability.

The non-obvious gotcha is that eviction thresholds have both soft and hard variants. Soft evictions give pods a grace period to terminate cleanly, while hard evictions kill pods immediately. If the hard eviction threshold is hit (e.g., memory.available < 50Mi), the kubelet kills pods without waiting for graceful shutdown, which can cause data loss or incomplete request processing. Architects should ensure hard thresholds are never reached by setting soft thresholds with enough buffer.

Think of a light switch connected to a circuit breaker. You flip the switch (container starts), the circuit overloads (container crashes), and the breaker trips (Kubernetes waits before retrying). Each time you try again, the breaker waits longer before allowing another attempt. CrashLoopBackOff is Kubernetes telling you that the container keeps failing and the wait time between restarts is increasing.

In Kubernetes, CrashLoopBackOff is not a separate error state — it is the backoff delay that kubelet applies after repeated container crashes. The container exits with a non-zero code, kubelet restarts it after 10 seconds, it crashes again, kubelet waits 20 seconds, then 40, then 80, capping at 300 seconds (5 minutes). The pod status shows CrashLoopBackOff during these waiting periods and Error or Completed when the container actually exits.

The most common root causes fall into categories. Application errors: the application throws an unhandled exception during startup because a required database is unreachable, a configuration file is malformed, or a required API key is missing. Configuration errors: the container command or args field is wrong (pointing to a script that does not exist in the image), the image tag points to a version with a different entrypoint, or a required environment variable is not set. Resource errors: the container is OOMKilled immediately on startup because the memory limit is too low for the JVM heap or the application's baseline memory footprint. Probe errors: an aggressive liveness probe kills the container before it finishes starting up, especially for Java applications with long startup times.

At production scale, the diagnostic sequence is: first check exit code with kubectl describe pod (exit code 1 = application error, 137 = OOMKilled/SIGKILL, 143 = SIGTERM). Then check previous container logs with kubectl logs --previous since the current container may have already crashed. Check whether the container image recently changed with kubectl rollout history. Verify that ConfigMaps, Secrets, and PersistentVolumeClaims referenced by the pod actually exist in the namespace.

The non-obvious gotcha is that CrashLoopBackOff can be caused by a liveness probe that is too aggressive during startup. If the liveness probe starts checking before the application is ready and the initialDelaySeconds is too short, the probe fails, kubelet kills the container, it restarts, and the cycle continues. The fix is to use a startup probe with a longer timeout to protect the liveness probe during application initialization, or to increase the liveness probe's initialDelaySeconds and failureThreshold.

Think of renting a car for a road trip. Self-managed nodes are like buying a car — you choose the model, handle maintenance, insurance, and repairs. Managed Node Groups are like a long-term lease — the dealer handles servicing, but you choose the model and drive it yourself. EKS Auto Mode is like a ride-sharing service — you just say where you want to go, and someone else handles the car, driving, and routing.

EKS deployment typically starts with creating the control plane (API server, etcd, scheduler) which is fully managed by AWS. Then you choose how to run worker nodes. Managed Node Groups are the most common choice: you specify instance types, desired capacity, and AMI family, and AWS creates an Auto Scaling Group, handles AMI updates, and manages node lifecycle. Self-managed nodes give you full control — you create your own ASG with custom AMIs, custom bootstrap scripts, and custom instance configurations, but you manage patching and upgrades yourself.

EKS Auto Mode, introduced in late 2024, takes automation further. Instead of specifying instance types and node groups, you let AWS choose the optimal compute for your workloads. Auto Mode manages the Kubernetes components typically installed as add-ons: kube-proxy, CoreDNS, VPC CNI, EBS CSI driver, and pod identity agent. It also handles GPU scheduling and Karpenter-style intelligent node provisioning. The tradeoff is less control over specific instance types and node configurations in exchange for significantly reduced operational burden.

At production scale, most enterprise teams use Managed Node Groups with specific instance families defined per workload type: compute-optimized (c6i) for API services, memory-optimized (r6i) for caching layers, and GPU instances (p4d, g5) for ML workloads. Teams deploy EKS using Terraform with the terraform-aws-modules/eks module, which handles the VPC, security groups, IAM roles, OIDC provider, and node group configurations. Add-ons like VPC CNI, CoreDNS, and EBS CSI driver are managed as EKS add-ons with version pinning.

The non-obvious gotcha is that Managed Node Groups handle rolling updates differently than you might expect. When you update the AMI or instance type, MNG creates new nodes and drains old ones, but it respects PodDisruptionBudgets. If a PDB blocks draining, the update stalls silently. Teams should monitor node group update status and set appropriate PDBs. With Auto Mode, you lose the ability to SSH into nodes or run DaemonSets for custom monitoring agents, which can be a blocker for teams with specific compliance or debugging requirements.

Think of a Kubernetes Service like a restaurant's main phone number. When customers call that number, a receptionist (kube-proxy) routes the call to whichever waiter (pod) is currently available. The restaurant doesn't give customers direct waiter phone numbers because waiters come and go with shift changes. The main number stays constant while the routing behind it updates automatically.

A Service is an abstraction that provides a stable network identity (ClusterIP, DNS name) for a set of pods. When you create a Service with a selector like app: payments-api, the endpoints controller (part of kube-controller-manager) watches for pods matching that label and creates an Endpoints resource listing their IP:port combinations. This Endpoints object updates automatically as pods are created, deleted, or become not-Ready.

Internally, the flow works like this: when a client pod sends a packet to the Service ClusterIP (e.g., 10.96.45.12:8080), the packet hits the node's network stack. kube-proxy has programmed iptables rules (or IPVS virtual servers) that intercept packets destined for that ClusterIP. These rules perform DNAT (Destination NAT), rewriting the destination IP from the Service ClusterIP to one of the pod IPs listed in the Endpoints object. The selection uses random probability by default (iptables mode) or round-robin/least-connections (IPVS mode). The return traffic from the pod is SNATed back so the client sees responses coming from the Service IP, not the pod IP directly.

In production, readiness probes are critical for this flow. A pod is only added to the Endpoints list when its readiness probe passes. If a pod's readiness probe fails, the endpoints controller removes it from the Endpoints object within seconds, and kube-proxy updates its rules to stop sending traffic there. This is why a missing or misconfigured readiness probe is dangerous: a pod that is starting up but not ready to serve traffic will receive requests and return errors.

A common gotcha: if your Service selector does not match any pod labels, the Endpoints object will be empty and all requests will fail. This is the number-one cause of 'Service not working' issues. Another subtle issue: headless Services (clusterIP: None) skip kube-proxy entirely and return pod IPs directly via DNS, which is used by StatefulSets for stable network identities.

Think of maxSurge and maxUnavailable like staffing rules during a shift change at a hospital. maxSurge is how many extra nurses you can have on the floor simultaneously (overtime budget), and maxUnavailable is how many nurse positions can be empty at once (minimum staffing). A hospital might say 'we can have 1 extra nurse on overtime (maxSurge=1) but no positions can be vacant (maxUnavailable=0)' meaning the new shift arrives before the old shift leaves.

When you update a Deployment's pod template (new image, env vars, etc.), the Deployment controller creates a new ReplicaSet with the updated spec and begins scaling it up while scaling the old ReplicaSet down. The speed and safety of this transition is controlled by the strategy.rollingUpdate.maxSurge and strategy.rollingUpdate.maxUnavailable parameters. Both can be absolute numbers or percentages of the desired replica count.

Here is the exact sequence with replicas=4, maxSurge=1, maxUnavailable=1: The controller can have at most 5 total pods (4 + maxSurge=1) and at least 3 available pods (4 - maxUnavailable=1). It starts by creating 1 new pod (now 5 total: 4 old + 1 new). Once the new pod is Ready, it terminates 1 old pod (now 4 total: 3 old + 1 new, with 1 old terminating making 3 available, which satisfies the minimum). It then creates another new pod (5 total again), waits for it to be Ready, terminates another old one, and repeats until all 4 pods are running the new version. The entire process respects both constraints at every step.

In production, the two most common configurations are: (1) maxSurge=25%, maxUnavailable=25% (the default) which balances speed and safety, allowing the update to happen in about 4 rounds for most replica counts; (2) maxSurge=1, maxUnavailable=0 which is the safest option because no old pod is terminated until its replacement is proven Ready. The second option means your cluster temporarily runs more pods than desired, so you need spare capacity, but it guarantees zero capacity reduction during the update.

The critical gotcha: maxUnavailable counts pods that are not Ready, not pods that are Terminating. If a new pod fails its readiness probe, it counts as unavailable, which may block the rollout from proceeding. A common failure mode is a broken readiness probe on the new version that never passes: the rollout creates maxSurge new pods, they all fail readiness, and the rollout stalls with both old and new pods running but no progress being made. This is when kubectl rollout undo becomes necessary.

Think of ConfigMaps and Secrets like the difference between a restaurant's public menu (ConfigMap) and the combination to the safe in the back office (Secret). Everyone can see the menu, it changes regularly, and it tells you what's available. The safe combination is shared only with authorized managers, stored separately from the menu, and handled with extra care. Both are 'configuration' for how the restaurant operates, but they require different levels of protection.

ConfigMaps store non-sensitive configuration data: application settings, feature flags, configuration file contents, environment-specific parameters like database hostnames (without credentials), log levels, and timeout values. They are stored in etcd as plaintext and are visible to anyone with read access to the namespace. You would use a ConfigMap for things like DATABASE_HOST=postgres.production.svc, LOG_LEVEL=info, or an entire nginx.conf file mounted as a volume.

Secrets store sensitive data: database passwords, API keys, TLS certificates, OAuth tokens, and SSH keys. They are base64-encoded (not encrypted by default!) in etcd, but have several additional protections: RBAC can restrict Secret access separately from other resources, Secrets are stored in tmpfs (memory-only) on nodes rather than written to disk, they can be configured for encryption-at-rest in etcd via EncryptionConfiguration, and kubectl get secrets does not display their values by default. It is critical to understand that base64 encoding is NOT encryption -- anyone who can read the Secret object can decode it trivially.

In production, the choice is straightforward: if leaking the value would be a security incident, it goes in a Secret. If it would merely be inconvenient or confusing, it goes in a ConfigMap. However, many teams go further and integrate external secret managers (HashiCorp Vault, AWS Secrets Manager) using the External Secrets Operator, which creates Kubernetes Secrets from external sources. This provides actual encryption, rotation, and audit logging that native Kubernetes Secrets lack.

A common gotcha: when you update a ConfigMap, pods mounting it as a volume see the change within 1-2 minutes (kubelet sync period), but pods using it as environment variables do NOT see the change until they are restarted. This catches many teams off guard. A pattern to force restarts is adding a ConfigMap hash annotation to the pod template so that changes trigger a rolling update. Secrets behave the same way regarding volume mounts vs environment variables.

Think of dynamic provisioning like ordering a custom-built storage unit. You fill out a form (PVC) saying 'I need 100GB of fast SSD storage.' The form gets sent to a specific furniture company (StorageClass/provisioner) that builds the storage unit to your exact specifications and delivers it (PV). Without dynamic provisioning, an administrator would need to pre-build storage units of various sizes and hope one matches what you need.

A PersistentVolumeClaim is a user's request for storage. It specifies what the user needs: how much capacity (100Gi), what access mode (ReadWriteOnce for single-node access, ReadWriteMany for multi-node), and which StorageClass to use. The StorageClass is a cluster-level resource that defines HOW volumes are provisioned: which cloud provisioner to call (ebs.csi.aws.com for AWS EBS, pd.csi.storage.gke.io for GCP), what parameters to pass (volume type, IOPS, encryption), and what reclaim policy to use when the PVC is deleted (Retain keeps the data, Delete destroys it).

The dynamic provisioning flow works as follows: (1) User creates a PVC referencing storageClassName: fast-ssd. (2) The PV controller sees the unbound PVC and finds the StorageClass named fast-ssd. (3) It calls the CSI provisioner specified in the StorageClass (e.g., ebs.csi.aws.com). (4) The CSI provisioner calls the AWS API to create an EBS volume with the specified type (gp3) and size (100Gi). (5) A PersistentVolume object is created in Kubernetes representing the actual EBS volume. (6) The PV is bound to the PVC. (7) When a pod references the PVC, the kubelet mounts the EBS volume into the pod's container.

In production, StorageClasses are critical for multi-tier storage strategies. A typical cluster has several StorageClasses: standard (gp3, general purpose, default), fast-ssd (io2, high IOPS for databases), archive (sc1, cold HDD for logs), and replicated (multi-AZ replication for critical data). Each tier has different cost, performance, and durability characteristics. Developers select the appropriate tier by specifying storageClassName in their PVC.

A common production issue is zone affinity: EBS volumes are AZ-specific, so a PV created in us-east-1a can only be mounted by pods on nodes in us-east-1a. If the node in that AZ goes down and the pod is rescheduled to us-east-1b, it cannot mount the volume. This is why StatefulSet PVCs with cross-AZ scheduling need careful topology configuration using volumeBindingMode: WaitForFirstConsumer in the StorageClass, which delays volume creation until the pod is scheduled, ensuring the volume is created in the same AZ as the pod.

Think of Kubernetes DNS like a building's internal phone directory. Instead of memorizing every department's extension number (IP address), employees dial the department name ('accounting') and the PBX system (CoreDNS) translates it to the right extension. When departments move offices (pods reschedule), the directory updates automatically so callers always reach the right place without changing their speed-dial.

Kubernetes runs CoreDNS as the cluster DNS server, typically deployed as a Deployment with 2 replicas in the kube-system namespace, fronted by a Service with a well-known ClusterIP (usually 10.96.0.10). When a pod is created, the kubelet configures its /etc/resolv.conf to point to this CoreDNS Service IP as the nameserver, with search domains that allow short-name resolution.

The DNS resolution hierarchy works as follows: For a Service named payments-api in namespace production, CoreDNS creates an A record at payments-api.production.svc.cluster.local pointing to the Service's ClusterIP. Pods in the same namespace can use the short name payments-api because the search domains in /etc/resolv.conf include production.svc.cluster.local. Pods in other namespaces use payments-api.production or the full FQDN. For headless Services (clusterIP: None), CoreDNS returns the individual pod IPs instead of a single ClusterIP, which is how StatefulSets provide stable DNS names for each pod (e.g., postgres-0.postgres-headless.database.svc.cluster.local).

Internally, CoreDNS watches the Kubernetes API server for Service and Endpoints changes using its kubernetes plugin. When a new Service is created or Endpoints change, CoreDNS updates its in-memory record set within seconds. It also handles external DNS resolution: if a pod queries for api.stripe.com, CoreDNS forwards the request to upstream nameservers configured in its Corefile (typically the node's DNS or a cloud provider's resolver like 169.254.169.253 on AWS).

In production, DNS is one of the most common sources of mysterious failures. Issues include: (1) ndots:5 default causing external domain lookups to try 5 internal suffixes first (e.g., api.stripe.com.production.svc.cluster.local before the real domain), adding latency for external calls. (2) CoreDNS pod resource exhaustion under high query volume causing timeouts. (3) DNS cache TTL mismatches when Services are recreated with new ClusterIPs but clients cache the old IP. The ndots issue is particularly insidious: reducing ndots from 5 to 2 in the pod's dnsConfig can cut external DNS resolution time by 80% for applications making many external API calls.

Think of the difference like assigned seating versus general admission at a concert. A Deployment is general admission -- you get 5 seats somewhere, and if you leave and come back, you might get different seats. A StatefulSet is assigned seating -- seat 1 is always seat 1 with the same person, same view, same neighbors, and if you step out for a snack, your seat is reserved when you return. Databases need assigned seating because each instance has a specific role and its own data.

A StatefulSet provides three guarantees that Deployments do not: (1) Stable, unique network identifiers -- pods are named deterministically (postgres-0, postgres-1, postgres-2) rather than randomly (postgres-7d4f8b-x9k2p). Each pod gets a DNS record via a headless Service (postgres-0.postgres-headless.database.svc.cluster.local) that persists across pod rescheduling. (2) Stable, persistent storage -- each pod gets its own PVC through volumeClaimTemplates, and that specific PVC reattaches to the same pod if it restarts. Pod postgres-0 always gets PVC data-postgres-0, even after deletion and recreation. (3) Ordered, graceful operations -- pods are created sequentially (0, then 1, then 2) and terminated in reverse order (2, then 1, then 0).

For databases, these guarantees are essential. Consider a PostgreSQL cluster with primary-replica replication: postgres-0 is the primary (accepts writes), postgres-1 and postgres-2 are replicas (stream WAL from primary). The replicas are configured to connect to postgres-0.postgres-headless for replication. If postgres-0 were a Deployment pod with a random name, every time it restarted with a new name, all replicas would lose their replication source. With a StatefulSet, postgres-0 always comes back as postgres-0 with the same DNS name and the same data volume, so replicas automatically reconnect.

Ordered startup matters for initialization: postgres-0 must be running and initialized before postgres-1 can start streaming WAL from it. StatefulSets guarantee this ordering by default (though you can use podManagementPolicy: Parallel to opt out for services that don't need it). Similarly, during scale-down, the highest-ordinal pod is removed first -- you never accidentally delete the primary (ordinal 0) before the replicas.

In production, StatefulSets have operational trade-offs: they are slower to recover from node failures because of the at-most-one semantics (the controller will not create a replacement pod until it can confirm the old one is fully terminated and its volume is detached). They are harder to debug because you cannot simply delete a stuck pod and let a fresh one take over -- the replacement must be the same ordinal with the same volume. Rolling updates proceed one pod at a time in reverse ordinal order by default, making them slower than Deployments. These trade-offs are acceptable for stateful workloads where data consistency matters more than recovery speed.

Think of requests and limits like airplane seat reservations. The request is your confirmed seat reservation -- the airline guarantees you that seat exists. The limit is the maximum overhead bin space you can use -- you can store luggage up to that amount, but security will stop you from exceeding it. If the airline overbooks (node memory pressure), passengers without reservations (BestEffort) are bumped first, then those with partial reservations (Burstable), and guaranteed-class passengers are bumped last.

Resource requests tell the Kubernetes scheduler how much CPU and memory a pod needs guaranteed. The scheduler only places a pod on a node if the node's allocatable resources minus all existing pod requests is greater than or equal to the new pod's requests. This is a hard constraint -- if no node has enough unrequested capacity, the pod stays Pending. Crucially, the scheduler does NOT look at limits or actual usage, only requests. This means a node can be 'full' from a scheduling perspective (all allocatable resources are requested) while actual utilization is only 30%.

Resource limits tell the kernel (via cgroups) the maximum a container can use. For CPU, exceeding the limit results in throttling -- the container is paused for the remainder of the cgroup period (typically 100ms). The container is NOT killed for exceeding CPU limits. For memory, exceeding the limit results in an OOM kill -- the kernel terminates the process immediately. This asymmetry is critical: CPU limits cause latency spikes, memory limits cause crashes.

The QoS class determines eviction priority during node memory pressure: (1) Guaranteed (requests == limits for all containers in the pod) -- lowest eviction priority, these pods are evicted last; (2) Burstable (at least one container has a request, but requests != limits) -- medium priority; (3) BestEffort (no requests or limits set on any container) -- highest eviction priority, evicted first. Within the Burstable class, pods using more memory relative to their request are evicted before those using less.

In production, the most common mistake is setting CPU limits too low. A pod with requests=100m and limits=200m will be throttled the instant it needs more than 200m, even if the node has 3 idle cores. Many teams now run without CPU limits entirely (only requests) to avoid artificial throttling while still allowing the scheduler to make good placement decisions. Memory limits remain essential because without them, a memory leak in one pod can OOM the entire node, killing all pods on it. The recommended pattern is: always set memory requests and limits (equal for critical services), always set CPU requests, and consider omitting CPU limits unless you need strict isolation.

Think of debugging a non-starting pod like diagnosing why a car won't start. You follow a systematic order: is there fuel (image available)? Does the engine turn over (container starts)? Does it stall after starting (crash loop)? Is it running but won't move (readiness probe failing)? Each stage has different diagnostic tools and different root causes.

The debugging workflow follows the pod lifecycle phases. Phase 1 - Pending: The pod has not been scheduled to a node. Run kubectl describe pod <name> and check the Events section at the bottom. Common failures: 'Insufficient cpu/memory' (no node has enough allocatable resources -- check node capacity with kubectl describe nodes), 'no nodes available to schedule pods' (taints without matching tolerations), '0/10 nodes are available: 10 pod has unbound PersistentVolumeClaims' (PVC not bound). The fix depends on the failure: add nodes, adjust requests, fix taints, or provision storage.

Phase 2 - ImagePullBackOff: The container runtime cannot pull the image. Events show 'Failed to pull image' with reasons like authentication failure (wrong imagePullSecret), image not found (typo in tag), or registry unreachable (network policy blocking egress to registry). Debug with kubectl describe pod and check imagePullSecrets in the pod spec.

Phase 3 - CrashLoopBackOff: The container starts and immediately exits. Run kubectl logs <pod> --previous to see the output from the last crashed instance. Common causes: missing environment variables the app requires, database connection failures, misconfigured command/args in the pod spec, or the container exiting with a non-zero exit code. The --previous flag is critical because the current container may have already crashed and been replaced.

Phase 4 - Running but not Ready: The container is running but the readiness probe is failing. Check kubectl describe pod for 'Unhealthy' events showing which probe fails and the HTTP response or exec output. The pod will not receive Service traffic while not Ready. Debug by exec-ing into the pod and manually running the probe command: kubectl exec <pod> -- curl localhost:8080/healthz.

Phase 5 - Running and Ready but not responding to external traffic: The issue is network-level. Check the Service endpoints (kubectl get endpoints <svc> -- is the pod listed?), check NetworkPolicies blocking ingress, and verify the container is listening on the correct port (kubectl exec <pod> -- netstat -tlnp).

For containers that crash too quickly to exec into, use ephemeral debug containers: kubectl debug <pod> --image=busybox --target=<container> attaches a debug container that shares the pod's process namespace, letting you inspect the filesystem and network without the application needing to be running.

Think of renting a storage unit. The StorageClass is the catalog that lists available unit types (climate-controlled, standard, fireproof). The PersistentVolumeClaim is your rental application form where you specify how much space you need and what kind. The PersistentVolume is the actual physical storage unit assigned to you. With dynamic provisioning, you never need to pre-build units; the facility constructs one on demand the moment your application is approved.

In Kubernetes, storage decouples the lifecycle of data from the lifecycle of pods. A PersistentVolume (PV) represents a piece of storage in the cluster, whether it is an AWS EBS volume, a GCP Persistent Disk, an NFS share, or a local SSD. It has a capacity, access modes (ReadWriteOnce, ReadWriteMany, ReadOnlyMany), and a reclaim policy that determines what happens when the claim is released. A PersistentVolumeClaim (PVC) is a namespaced request that binds to a PV matching its requirements. The StorageClass is the blueprint that tells Kubernetes which CSI driver to use, what parameters to pass (volume type, IOPS, encryption), and what reclaim policy to apply.

Dynamic provisioning eliminates the need for cluster administrators to pre-create PVs manually. When a PVC references a StorageClass, the provisioner watches for unbound claims, calls the cloud API through the CSI driver to create the actual disk, then creates a PV object and binds it to the PVC. The pod mounts the PVC by name and gets access to the underlying volume. This flow means developers only need to know what size and class they want, while the platform handles the infrastructure. CSI (Container Storage Interface) is the standard plugin mechanism that connects Kubernetes to any storage backend, whether cloud-managed or on-premises.

In production, storage configuration directly impacts availability and cost. Setting the wrong reclaim policy to Delete means data is destroyed when a PVC is removed, which can be catastrophic for database volumes. Using ReadWriteOnce with a Deployment that has multiple replicas causes scheduling failures because the volume can only attach to one node. Volume binding mode WaitForFirstConsumer ensures the PV is provisioned in the same availability zone as the pod, avoiding cross-AZ attachment failures. Teams running StatefulSets for databases like PostgreSQL or Kafka typically use volumeClaimTemplates with a StorageClass that has reclaimPolicy set to Retain.

The non-obvious gotcha is that PVCs bound to a PV cannot be resized unless the StorageClass has allowVolumeExpansion set to true. Another common production issue is zone mismatch: a PV provisioned in us-east-1a cannot be mounted by a pod scheduled in us-east-1b. Teams discover this after a node failure forces pod rescheduling. Additionally, if the default StorageClass is accidentally deleted or changed, new PVCs remain Pending indefinitely with no obvious error unless you describe the PVC and check events.

Think of three different staffing models at a company. A Deployment is like a pool of identical customer service agents where any agent can handle any call and they are interchangeable. A StatefulSet is like a team of specialists where each person has their own desk, their own filing cabinet, and clients who always ask for them by name. A DaemonSet is like having one security guard assigned to every floor of the building, no more and no less, regardless of how many floors exist.

A Deployment manages stateless pods through a ReplicaSet. Pods get random names like payments-api-7f8d9c-x4k2q and can be killed and recreated freely because they hold no unique state. Scaling is trivial because every replica is identical. Rolling updates replace pods one by one without caring about order. This makes Deployments ideal for web servers, APIs, workers processing from queues, and any application where one replica can substitute for another.

A StatefulSet manages stateful pods with guarantees that Deployments cannot provide. Each pod gets a stable, predictable hostname like payments-db-0, payments-db-1, payments-db-2. These names persist across restarts and rescheduling. Each pod gets its own PersistentVolumeClaim through volumeClaimTemplates, so payments-db-0 always mounts the same volume even after rescheduling to a different node. Pods are created in order (0 before 1 before 2) and terminated in reverse order. This ordered lifecycle is critical for distributed systems like PostgreSQL with streaming replication, Kafka brokers that need stable broker IDs, or ZooKeeper nodes that form a quorum. The headless Service associated with a StatefulSet creates DNS records like payments-db-0.payments-db-headless.payments.svc.cluster.local for direct pod addressing.

A DaemonSet ensures exactly one pod runs on every node (or a subset of nodes matching a nodeSelector or affinity). When a new node joins the cluster, the DaemonSet controller automatically schedules a pod on it. When a node is removed, the pod is garbage collected. This makes DaemonSets perfect for node-level concerns: Fluentd or Filebeat for log collection, Prometheus node-exporter for host metrics, Datadog agents, CNI plugins, or storage drivers that need to run on every machine.

In production, choosing the wrong controller causes real problems. Running a database as a Deployment means all replicas share no stable identity, making replication configuration fragile and data loss likely during rescheduling. Running a stateless API as a StatefulSet adds unnecessary ordering constraints that slow scaling and complicate rollouts. Using a Deployment for a log collector means some nodes might run zero or multiple collectors depending on scheduling.

The non-obvious gotcha with StatefulSets is that deleting a StatefulSet does not delete the PVCs it created. This is intentional to prevent data loss, but it means orphaned volumes accumulate cost. Another surprise is that StatefulSet rolling updates proceed one pod at a time in reverse ordinal order, and if pod N-1 fails its readiness check, the rollout halts entirely. Teams must monitor StatefulSet rollouts carefully because a single unhealthy replica blocks the entire update.

Think of a hotel key card system. Instead of giving every guest a master key (hardcoded credentials), the front desk verifies your identity (OIDC provider), issues a card (short-lived token) that only opens your specific room (IAM role), and the card expires at checkout. If someone steals the card, it stops working soon and only ever opened one room. IRSA works the same way for pods accessing AWS services.

Without IRSA, teams typically use one of three insecure patterns: storing AWS access keys in Kubernetes Secrets (which can leak through RBAC, etcd backups, or misconfigured pod access), assigning an IAM instance profile to the entire node (which gives every pod on that node the same permissions), or using tools like kube2iam that intercept the metadata endpoint (which adds complexity and latency). IRSA eliminates all three by giving each pod its own identity that AWS trusts directly.

The mechanism works through several coordinated components. First, the EKS cluster has an OIDC provider registered with AWS IAM. This tells AWS to trust tokens issued by the Kubernetes API server. Second, an IAM role is created with a trust policy that specifies which Kubernetes service account in which namespace can assume it. The trust policy condition checks the OIDC issuer, the audience, and the subject (system:serviceaccount:namespace:sa-name). Third, the Kubernetes ServiceAccount is annotated with eks.amazonaws.com/role-arn pointing to the IAM role. Fourth, when a pod using that ServiceAccount starts, the EKS pod identity webhook injects a projected service account token volume and sets AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables. The AWS SDK in the application reads these, calls STS AssumeRoleWithWebIdentity, and receives temporary credentials.

In production, IRSA provides least-privilege access at the pod level. The payments-api can access only the S3 bucket it needs and the SQS queue it consumes from, while the checkout-worker in the same namespace can access DynamoDB but not S3. If one pod is compromised, the blast radius is limited to its specific IAM role permissions. Tokens are automatically rotated (default expiry is 12 hours, configurable down to 15 minutes), and credential theft is detectable through CloudTrail.

The non-obvious gotcha is that IRSA requires the application to use an AWS SDK version that supports web identity token authentication (SDK v2 or AWS SDK for Go v1.25+, Python boto3 1.9.220+). Legacy applications that only read from environment variables for static keys will not work without code changes. Another common issue is trust policy misconfiguration: if the namespace or service account name in the condition does not match exactly, AssumeRole fails silently and the pod falls back to node-level permissions or gets AccessDenied. EKS Pod Identity is the newer alternative that simplifies the trust policy setup but requires the EKS Pod Identity Agent DaemonSet.

Think of the difference between a car dashboard and a mechanic's diagnostic computer. The dashboard shows predefined gauges: speed, fuel, temperature (monitoring). You know something is wrong when the temperature gauge is high, but you cannot ask why. The diagnostic computer lets you query any sensor, trace any circuit, and correlate signals to find the root cause (observability). Both are necessary, but observability lets you investigate problems you never anticipated.

Monitoring is the practice of collecting, aggregating, and alerting on predefined metrics. In Kubernetes, this means tracking CPU usage, memory consumption, pod restart counts, request rates, error rates, and latency percentiles. Tools like Prometheus scrape metrics endpoints, store time-series data, and fire alerts when thresholds are breached. Monitoring answers known questions: is the service up? Is CPU above 80 percent? Are there more than five restarts in ten minutes? The limitation is that monitoring only answers questions you thought to ask in advance.

Observability is a property of a system that lets you understand its internal state from external outputs. The three pillars are metrics (numeric measurements over time), logs (discrete events with context), and traces (request paths across services). An observable system lets you ask new questions without deploying new instrumentation. When a user reports that checkout is slow for customers in Europe but not the US, observability tools let you slice latency by region, trace a specific request through fifteen microservices, and correlate with database query times — even if you never built a dashboard for that exact scenario.

Centralized logging is critical in Kubernetes because the platform treats pods as disposable. When a pod crashes, Kubernetes restarts it and the previous container's logs are only accessible through kubectl logs --previous until the pod is rescheduled or evicted, at which point they vanish entirely. During an incident at 3 AM, the pod that caused the problem may have already been replaced three times. Without centralized logging (Fluentd/Fluent Bit shipping to Elasticsearch, CloudWatch, or Loki), the evidence is gone. Centralized logging also enables searching across hundreds of pods simultaneously, correlating events across namespaces, and retaining logs for compliance.

In production, the common mistake is treating metrics and logs as separate concerns. A mature observability stack correlates them: a Prometheus alert for high error rate links to a Grafana dashboard, which links to Loki log queries filtered by the same timeframe, labels, and pod. Distributed tracing with OpenTelemetry connects a single user request across service boundaries, showing exactly where latency accumulates. The goal is reducing mean time to detection (MTTD) and mean time to resolution (MTTR) by making every failure diagnosable without SSH access to individual containers.

The non-obvious gotcha is log volume cost. A busy Kubernetes cluster can generate gigabytes of logs per hour. Without structured logging (JSON format with consistent fields), log search becomes grep across unstructured text. Without log-level filtering at the agent, debug logs flood the central store and inflate storage costs. Teams must balance verbosity against cost, typically shipping warn and error in production while keeping debug available for temporary escalation during incidents.

Think of a restaurant handling a dinner rush. VPA is like giving each chef a bigger workstation when they are cramped (vertical scaling of individual workers). HPA is like calling in more chefs when orders pile up (horizontal scaling of worker count). Cluster Autoscaler is like opening additional kitchen rooms when there is no space for more chefs (infrastructure scaling). Each addresses a different bottleneck, and they work best when coordinated.

The Horizontal Pod Autoscaler (HPA) watches metrics and adjusts the replica count of a Deployment or StatefulSet. By default it uses CPU utilization, but it can target memory, custom metrics (requests per second from Prometheus), or external metrics (SQS queue depth from CloudWatch). HPA evaluates every 15 seconds (configurable), calculates the desired replica count using the formula desiredReplicas = currentReplicas * (currentMetric / targetMetric), and scales accordingly. It respects minReplicas and maxReplicas boundaries and has stabilization windows to prevent flapping.

The Vertical Pod Autoscaler (VPA) analyzes historical resource usage and recommends or automatically adjusts the CPU and memory requests and limits for containers. It operates in three modes: Off (only recommends), Initial (sets resources only at pod creation), and Auto (evicts and recreates pods with updated resources). VPA solves the problem of developers guessing resource requests — either setting them too high (wasting cluster capacity) or too low (causing throttling and OOMKills). It uses a recommender component that analyzes metrics over time and an updater that evicts pods needing adjustment.

The Cluster Autoscaler watches for pods stuck in Pending state because no node has sufficient resources. When it detects unschedulable pods, it evaluates which node group can accommodate them and triggers the cloud provider to add nodes. Conversely, it removes underutilized nodes (below 50 percent utilization by default) after a cooldown period, draining pods safely using PodDisruptionBudgets. It works with AWS Auto Scaling Groups, GCP Managed Instance Groups, or Azure VM Scale Sets.

In production, coordination between these three autoscalers requires careful planning. The critical rule is never to use HPA and VPA on the same metric for the same pod, because they will fight: HPA tries to add replicas while VPA tries to resize existing ones, creating oscillation. The recommended pattern is HPA on CPU with VPA on memory in recommendation-only mode, or HPA on custom metrics while VPA handles resource right-sizing in Initial mode. Cluster Autoscaler must respond within 30-60 seconds to add nodes or traffic will be dropped during spikes. Teams configure node pool warm-up strategies or use Karpenter for faster, more flexible node provisioning.

The non-obvious gotcha is scaling lag. HPA reacts in seconds but new pods need time to start (image pull, readiness probe). Cluster Autoscaler takes 1-3 minutes to provision new nodes. During a sudden traffic spike, the system can drop requests for several minutes. Mitigation strategies include setting higher minReplicas during known peak windows, using PodPriority to preempt less critical workloads, over-provisioning with pause pods (low-priority placeholder pods that get evicted instantly to free capacity), and combining with KEDA for event-driven scaling that responds to queue depth before CPU rises.

Think of a mall directory kiosk. Visitors look up the store name (host), follow the floor and section (path), and expect to reach the store (backend). If the directory has a typo, the store moved, the hallway is blocked, or the store is closed, visitors cannot get there. Ingress troubleshooting follows the same logic: verify every link in the chain from the external request to the running pod.

An Ingress resource is a routing declaration, not a router itself. It tells an Ingress controller (NGINX, ALB, Traefik, Istio Gateway) how to route external traffic based on hostname and URL path. The controller watches Ingress objects, updates its routing table (nginx.conf, ALB rules, envoy routes), and directs traffic to the backend Service and port specified. If any part of this chain is misconfigured, traffic either returns 404, 503, or times out with no obvious error.

The troubleshooting sequence starts at the DNS and load balancer layer. Verify that the domain resolves to the correct IP or ALB hostname. Check whether the load balancer health checks pass for the Ingress controller pods. Then inspect the Ingress resource: does the host field match the exact domain being requested? Does the path match the URL pattern (prefix vs exact)? Is the backend service name and port correct? A common mistake is specifying the container port instead of the Service port, or using a service name that does not exist in the same namespace.

Next, check the Ingress controller itself. Is the controller pod running and ready? Has it synced the Ingress resource (describe the Ingress and look for events or Address field population)? Check controller logs for configuration reload errors, upstream connection failures, or TLS certificate problems. For NGINX Ingress Controller, the logs show every routing decision and upstream selection. An empty Address field on the Ingress means the controller has not processed it, often because the ingressClassName does not match or the controller is filtering by namespace.

Behind the Ingress, the Service must have healthy endpoints. Run kubectl get endpoints to confirm the Service has pod IPs listed. Empty endpoints mean either no pods match the Service selector, pods exist but fail readiness probes, or the selector labels are mismatched. Even with endpoints populated, the target port must match what the container listens on. A Service targeting port 8080 when the app listens on 3000 will show connection refused in controller logs.

The non-obvious gotcha is TLS misconfiguration. If the Ingress specifies a TLS section but the Secret does not exist, contains an expired certificate, or has a subject name that does not match the host field, some controllers serve a default fake certificate while others reject the connection entirely. Another common issue is path precedence: if you have both /api and /api/v2 paths, the ordering and pathType (Prefix vs Exact vs ImplementationSpecific) determine which rule matches. Some controllers require a trailing slash; others do not.

Think of debugging a dropped phone call. You could blame the phone, but the issue might be the SIM card, the cell tower, the network backbone, or the destination carrier. A systematic technician tests each link in the chain from the caller to the receiver. In Kubernetes, a user report of service down could mean failure at any of ten distinct layers, and guessing wastes precious minutes during an outage.

The senior troubleshooting formula follows the full request path: Application, Container, Pod, Service, Endpoints, Ingress, Load Balancer, DNS, Network, Cloud Infrastructure. Starting at the application layer, check if the process inside the container is running and responding. Inspect logs for uncaught exceptions, OOMKills, configuration errors, or dependency failures. At the container layer, check if the container started successfully, whether liveness and readiness probes are passing, and if there are restart loops. Container state shows Waiting (image pull failure, CrashLoopBackOff), Running, or Terminated (exit code reveals the reason).

At the pod layer, check scheduling status: is the pod Pending (insufficient resources, node affinity, taints), Running but not Ready (failing readiness probe), or Evicted (node pressure)? Describe the pod to see events like FailedScheduling, FailedMount, or BackOff. At the Service layer, verify the Service selector matches pod labels exactly and the Service port maps to the correct container port. At the Endpoints layer, confirm the Service has populated endpoints — empty endpoints is one of the most common causes of 503 errors that looks like an application problem but is actually a configuration issue.

At the Ingress layer, check that host and path rules are correct, the controller has synced (Address field populated), and TLS secrets are valid. At the Load Balancer layer, verify health check paths, target group registration, and security groups. At the DNS layer, confirm the domain resolves to the correct endpoint and TTLs have propagated. At the Network layer, check NetworkPolicies, security groups, NACLs, and pod-to-pod connectivity. At the Cloud Infrastructure layer, check node health, API rate limits, IAM permissions, and service quotas.

In production, this layered approach is critical because symptoms at one layer often originate from another. A 503 at the load balancer might be caused by empty endpoints, which is caused by failing readiness probes, which is caused by a database connection timeout, which is caused by a security group change. Following the chain systematically reveals the root cause in minutes rather than hours. Experienced engineers also parallelize: one person checks the application path while another checks infrastructure, then they compare findings.

The non-obvious gotcha is confirmation bias. Engineers see a recent deployment and assume the deploy caused the issue, but correlation is not causation. The deploy might be coincidental while the real cause is a cloud provider networking issue, a certificate expiration, or a resource quota being hit. The troubleshooting formula forces you to verify evidence at each layer rather than jumping to conclusions. Document what you checked, what you found, and what you ruled out — this prevents circular debugging during stressful incidents.

Think of a doctor performing a physical exam. They do not start with an MRI. They check vital signs first: heart rate, blood pressure, temperature, breathing. These quick checks narrow the problem from thousands of possibilities to a few likely diagnoses. The first ten kubectl commands serve the same purpose: they give you a rapid situational awareness of what is broken, where it is broken, and what changed recently.

The first command, kubectl get pods -A (or scoped to the affected namespace), provides the highest-signal overview. It immediately shows CrashLoopBackOff, ImagePullBackOff, Pending, Evicted, or Terminating states. The -A flag catches problems in supporting namespaces like ingress, monitoring, or cert-manager that might affect the application indirectly. From this output, you identify which pods need deeper investigation.

The second and third commands, kubectl describe pod and kubectl logs --previous, give you the story of what happened. Describe shows events (scheduling failures, probe failures, volume mount errors), conditions, and container state transitions. Logs with --previous retrieve output from the last crashed container, which is essential because the current container might already be a fresh restart with no relevant history. Together, these two commands explain why a pod is unhealthy.

The fourth through sixth commands focus on the traffic path. kubectl get events --sort-by=.metadata.creationTimestamp reveals cluster-level problems like node pressure, quota exhaustion, or webhook failures. kubectl get endpoints verifies that the Service has healthy backends — empty endpoints is the silent killer that causes 503s with no application logs. kubectl describe ingress shows whether external routing is configured correctly and whether the controller has synced.

The seventh through ninth commands assess infrastructure. kubectl get nodes shows node readiness, version skew, and conditions like MemoryPressure or DiskPressure. kubectl top pods reveals which pods are consuming the most CPU and memory, potentially causing throttling or OOMKills. kubectl top nodes shows overall cluster capacity and whether scheduling failures are due to exhausted resources.

The tenth command, kubectl rollout status, connects the issue to recent changes. If a deployment is progressing (waiting for new pods to become ready) or degraded (deadline exceeded), it tells you whether the current incident correlates with a deploy. Combined with rollout history, you can quickly determine if a rollback is appropriate.

The non-obvious gotcha is command order and scope. Running these commands namespace-scoped when the problem is cluster-wide (node pressure, DNS failure, cert-manager down) wastes critical minutes. Conversely, running them cluster-wide when you already know the affected service adds noise. Senior engineers adapt the sequence based on the alert context: a PagerDuty alert with a namespace and deployment name lets you skip the broad scan and go directly to describe and logs.

Think of an open office floor plan versus one with locked rooms. By default, Kubernetes networking is the open floor plan: any pod can talk to any other pod on any port across any namespace. This is the flat network model. NetworkPolicies add walls and locked doors. Once you put one wall around a pod, everything that is not explicitly allowed through a door is blocked. But pods without any wall remain completely open.

Kubernetes networking follows a fundamental rule: all pods can communicate with all other pods without NAT. This is the flat network requirement that every CNI plugin (Calico, Cilium, Weave, AWS VPC CNI) must implement. While this simplifies service discovery and connectivity, it means a compromised pod in the staging namespace can reach the database pods in the production namespace. This is where NetworkPolicies provide segmentation.

A NetworkPolicy is a namespaced resource that selects pods using labels (podSelector) and defines allowed ingress (incoming) and egress (outgoing) traffic rules. Each rule can specify allowed sources or destinations by pod labels (podSelector), namespace labels (namespaceSelector), or IP ranges (ipBlock with CIDR notation). The critical behavior to understand is that NetworkPolicies are additive for the selected pods: if a pod matches any policy, only traffic explicitly allowed by all applicable policies is permitted. There is no deny rule — you achieve deny-all by creating a policy that selects pods but specifies no allowed traffic.

The standard production pattern is defense in depth. First, apply a default deny-all policy to each sensitive namespace that blocks all ingress and egress for all pods. Then layer specific allow policies on top: the payments-api can receive traffic from the ingress controller and the API gateway on port 8080, can send traffic to payments-db on port 5432, and can reach external payment processors on specific CIDR ranges. This ensures that even if a vulnerability is exploited in one service, lateral movement is restricted.

In production, NetworkPolicies are only enforced if the CNI plugin supports them. AWS VPC CNI alone does not enforce NetworkPolicies — you need to add Calico or Cilium as a policy engine. Kubenet (the default in some environments) also does not support them. Teams have deployed NetworkPolicy YAMLs assuming they were protected, only to discover during a security audit that the CNI never enforced them. Always verify enforcement by testing that a blocked connection actually fails.

The non-obvious gotcha is DNS resolution. A default deny-all egress policy blocks DNS queries to CoreDNS (port 53), which breaks service discovery for all pods in the namespace. You must explicitly allow egress to the kube-system namespace on port 53 (TCP and UDP) in your deny-all policy, or all pods will fail to resolve service names. Another surprise is that NetworkPolicies are namespace-scoped: a policy in namespace A cannot directly reference pods in namespace B by name, only by namespace labels. If namespace labels are not set correctly, cross-namespace rules silently fail to match.

Think of upgrading a restaurant's menu. A rolling update is like changing one table's menu at a time until all tables have the new menu — gradual with no extra cost but hard to undo if customers complain. Blue-green is like printing a complete new set of menus and swapping them all at once while keeping the old ones ready — instant switchback but double the printing cost. Canary is like giving the new menu to one table first, watching their reaction, and only rolling it out if they order successfully — safest but slowest.

The rolling update is Kubernetes' native deployment strategy. When you update a Deployment's pod template, the controller creates a new ReplicaSet and gradually scales it up while scaling the old one down. The maxSurge parameter controls how many extra pods can exist during the transition (e.g., 25% means 12 becomes 15 temporarily), and maxUnavailable controls how many pods can be missing (e.g., 25% means at least 9 of 12 must always be ready). Traffic flows to both old and new pods simultaneously during the rollout. If the new pods fail readiness probes, the rollout pauses. Rollback is a single command: kubectl rollout undo.

Blue-green deployment runs two complete environments: blue (current live version) and green (new version). Both are fully deployed and warmed up before any traffic switch. Once the green environment passes health checks and smoke tests, the Service selector or Ingress routing is updated to point all traffic to green. If problems emerge, switching back to blue is instantaneous because it is still running. The cost is double the resources during the transition. In Kubernetes, this is implemented by having two Deployments (payments-api-blue and payments-api-green) and changing the Service selector or using Ingress annotations to switch.

Canary deployment sends a small fraction of production traffic to the new version (typically 1-5% initially) while the majority continues on the stable version. If metrics (error rate, latency, business KPIs) look healthy over a defined period, traffic is gradually shifted: 5%, 25%, 50%, 100%. If any metric degrades, traffic reverts to the stable version automatically. This provides the highest safety because real production traffic validates the new version with minimal blast radius. In Kubernetes, canary deployments are typically managed by tools like Argo Rollouts, Flagger, or Istio traffic splitting rather than native Kubernetes primitives.

In production, the choice depends on your risk tolerance, infrastructure budget, and rollback speed requirements. Rolling updates are free (no extra resources) and native, but you cannot easily test the new version in isolation before it receives traffic. Blue-green is excellent for major version changes that might need instant rollback, but it doubles resource cost. Canary is ideal for high-traffic services where even a 1% error rate affects thousands of users and you need metric-driven progressive delivery. Many teams use rolling updates for internal services and canary for user-facing APIs.

The non-obvious gotcha is database compatibility. All three strategies may have old and new application versions running simultaneously (during the rollout window). If the new version changes the database schema, both versions must work with both the old and new schema. This requires backward-compatible migrations: add columns as nullable, never rename in place, deploy the new schema before the new code, and remove old columns only after the old code is fully gone. Teams that forget this have both versions hitting the database and one version crashes on schema mismatch.

Think of securing a building. RBAC is the badge system that controls who can enter which floors and rooms (API permissions). Pod Security Standards are the building codes that prevent tenants from doing dangerous things like removing fire exits or storing explosives (runtime restrictions). NetworkPolicies are the internal walls and locked corridors that prevent someone on one floor from accessing another floor without authorization (network segmentation). Each layer addresses a different attack vector, and all three are needed for comprehensive security.

RBAC (Role-Based Access Control) governs who can interact with the Kubernetes API and what operations they can perform. A Role defines permissions (verbs like get, list, create, delete on resources like pods, secrets, deployments) within a namespace. A ClusterRole defines permissions cluster-wide. RoleBindings and ClusterRoleBindings associate roles with users, groups, or service accounts. Without RBAC, a compromised service account could read Secrets from other namespaces, create privileged pods, or delete critical workloads. Properly scoped RBAC ensures the payments-api ServiceAccount can only read its own ConfigMaps and Secrets, not those belonging to other teams.

Pod Security Standards (the replacement for the deprecated PodSecurityPolicy) define three levels: Privileged (unrestricted), Baseline (prevents known privilege escalations), and Restricted (heavily hardened). These are enforced through the Pod Security Admission controller using namespace labels. Restricted mode prevents running as root, using host networking, mounting hostPath volumes, adding Linux capabilities, and running privileged containers. This matters because a container escape from a privileged pod gives full root access to the host node, which compromises all pods on that node and potentially the entire cluster.

NetworkPolicies, as the third layer, restrict which pods can communicate with which other pods and external systems. Even if an attacker compromises a pod, NetworkPolicies prevent lateral movement to the database, secrets store, or other microservices. Combined with RBAC preventing the compromised pod's ServiceAccount from reading other Secrets, and Pod Security preventing privilege escalation to the host, the blast radius of a single compromised container is contained to that container's existing data and network connections.

In production, these three controls must be deployed together because each has blind spots. RBAC alone cannot prevent a pod from connecting to a database it should not access (that is a network concern). NetworkPolicies alone cannot prevent a pod from running as root and escaping to the host. Pod Security alone cannot prevent a compromised pod from calling the Kubernetes API to read secrets. Defense in depth means that bypassing one control does not grant full access.

The non-obvious gotcha is that Pod Security Admission only warns or denies at pod creation time — it does not retroactively affect running pods. If you add Restricted enforcement to a namespace with existing non-compliant pods, those pods continue running until they are recreated. Another common gap is that RBAC for ServiceAccounts often starts too permissive (using default ServiceAccount with broad permissions) and is never tightened. Teams should create dedicated ServiceAccounts per workload with minimal permissions and disable token automounting for pods that do not need API access.