Kubernetes Architect Questions

Think of a restaurant opening for dinner. The kitchen can start cooking only after the exhaust fan and point-of-sale terminal are running, but those support systems must keep working after the first meal is served. A native sidecar container is that support system: it starts before or alongside the main work, keeps running, and is not the reason the restaurant exists.

In Kubernetes, native sidecars solve the long-standing awkwardness of treating log shippers, mesh proxies, and credential refreshers as normal containers. A normal app container has no special startup relationship to the main workload. A traditional init container runs to completion before app containers start. A native sidecar sits between those ideas: it is listed under initContainers, but restartPolicy: Always tells Kubernetes to keep it alive as a supporting container.

Internally, kubelet processes init containers in order. When it reaches an init container with restartPolicy: Always, it starts that container and continues once the sidecar has started successfully. The app containers then start while the sidecar remains running. If the sidecar has a readiness probe, Kubernetes includes that probe when calculating whether the Pod is Ready. During shutdown, Kubernetes terminates app containers first, then sidecars, which is important for log flushing and proxy drain behavior.

At production scale, this changes rollout and observability practices. Teams should monitor sidecar restart count, readiness probe failures, log shipping lag, and whether the sidecar adds startup latency. For Jobs, the sidecar no longer keeps the Job alive forever after the main container has completed, which removes a common operational trap in batch workloads that need a helper process.

The non-obvious gotcha is that native sidecars are still part of the Pod resource budget and lifecycle. A slow or unhealthy sidecar can block readiness for an otherwise healthy app, and a sidecar with an aggressive readiness probe can cause endpoints to flap under dependency pressure. Architects should treat sidecars as production dependencies with SLOs, not invisible helper containers.

Think of a building lobby with two security checks. One guard can read a badge and a printed rule list immediately, while another guard must call a remote office for approval. ValidatingAdmissionPolicy is the fast local rule list; a validating webhook is the remote phone call. Both can be useful, but the second one can delay every person entering the building if the phone line is slow.

In Kubernetes, admission control is the checkpoint between an API request and persisted cluster state. ValidatingAdmissionPolicy gives cluster administrators a native way to reject invalid resources with Common Expression Language, or CEL, which is a safe expression language designed for policy checks. Instead of operating a separate webhook service, TLS certificates, network path, scaling rules, and timeout behavior, the API server evaluates the expression directly.

The flow is precise. A ValidatingAdmissionPolicy defines match constraints and validation expressions. A ValidatingAdmissionPolicyBinding connects that policy to the resources and namespaces where it applies and specifies actions such as Deny. Optional parameter resources can make one abstract policy reusable across teams. When a matching CREATE or UPDATE request reaches admission, Kubernetes evaluates the CEL expression against the incoming object. If the expression returns false, the configured failure behavior decides whether the request is rejected.

In production, this is ideal for rules like requiring labels, limiting replica counts, blocking privileged settings, or enforcing image registry patterns. Operators should monitor admission rejection rates, API server latency, policy rollout changes, and namespace selectors. Policies should be rolled out gradually because one bad rule can block deployments across many services. Use audit or warning-style validation before deny when the blast radius is unknown.

The gotcha is treating CEL as a total replacement for admission webhooks. CEL should not call external systems and cannot mutate resources. If policy needs live inventory from a CMDB, vulnerability scanner, signing service, or custom approval database, a webhook or controller may still be required. Architects should prefer CEL for fast predictable (deterministic) checks and reserve webhooks for cases where the policy genuinely needs external state.

Think of a hospital assigning specialized equipment to surgeries. A room is not enough; the scheduler must know whether the right scanner, technician, and access path are available before booking the operation. Dynamic Resource Allocation, or DRA, gives Kubernetes a similar model for GPUs, network adapters, accelerators, and other devices that cannot be represented well by a simple integer resource count.

DRA exists because extended resources are too blunt for many modern devices. A workload may need a GPU with specific memory, a network device with a particular capability, or a device pool managed by a vendor driver. DeviceClass describes categories of devices. ResourceClaim says what a workload needs. ResourceClaimTemplate lets Kubernetes generate per-Pod claims. ResourceSlice advertises devices attached to nodes so the scheduler can reason about where the Pod can run.

The internal workflow crosses multiple components. A driver creates ResourceSlices for its device pools. A workload references a ResourceClaim or ResourceClaimTemplate. If a template is used, the resourceclaim-controller creates a claim. The scheduler filters ResourceSlices, allocates a matching device, updates the ResourceClaim status, and schedules the Pod to a compatible node. The kubelet and device driver on that node then prepare device access for the container.

Production adoption requires more than enabling a feature. Platform teams should watch unschedulable Pods, ResourceClaim conditions, ResourceSlice inventory freshness, kubelet device metrics, driver logs, and device health status where available. They should test behavior during device failure, node drain, driver restart, and cluster upgrade. DRA also introduces RBAC and security questions because drivers may report device status and influence scheduling decisions.

The subtle gotcha is assuming allocation means uniform capacity across replicas. Some DRA features allow prioritized alternatives, so different Pods in the same ReplicaSet may receive different device types if the workload permits it. If the application assumes identical hardware, performance or correctness can drift between replicas. Architects need clear device classes, workload tolerances, and dashboards that expose which claim was allocated to which Pod.

Think of a shopping mall. The mall operator controls entrances, fire doors, and security rules, while each store controls its own sign and product layout. Traditional Ingress often turns the mall operator into the person editing every store sign. Gateway API gives the building owner and store owner separate but connected responsibilities.

Gateway API was designed to make Kubernetes traffic management more expressive and role-oriented than the original Ingress API. Ingress is useful but often overloaded with controller-specific annotations, which blur ownership and make portability harder. Gateway API introduces clearer resources for infrastructure providers, cluster operators, and application teams so each group can manage the layer it actually owns.

The flow starts with GatewayClass, which identifies the implementation or controller. A Gateway represents a deployed data-plane entry point with listeners such as HTTPS on port 443. HTTPRoute, GRPCRoute, or other route resources define application-level routing rules and attach to a Gateway when allowed by namespace and listener policy. The controller reconciles those resources into load balancer, proxy, or service mesh configuration.

In production, Gateway API helps with multi-team environments. Platform engineers can standardize TLS, listener ports, allowed namespaces, and shared infrastructure. App teams can publish or update service routes without editing a central Ingress file. Operators should monitor route attachment status, accepted conditions, listener conflicts, certificate readiness, and controller reconciliation errors. This shifts troubleshooting from annotation archaeology to explicit status fields.

The gotcha is that Gateway API does not magically remove governance. If allowedRoutes is too permissive, one team can still attach unexpected hostnames or paths. If it is too strict, teams see routes that never attach and traffic silently stays on the old path. Architects need namespace policy, hostname ownership, certificate automation, and clear dashboards showing which routes are accepted by which Gateway.

Think of a highway toll system. Traditional kube-proxy is like a toll booth where every car stops, gets checked, and is directed to its lane. EBPF with Cilium is like an electronic pass reader embedded in the road surface — the car never stops, the toll is processed at wire speed, and the road itself knows which lane to direct traffic into without a booth.

Cilium replaces the iptables-based kube-proxy and the user-space proxy model used by traditional service meshes. Instead of maintaining thousands of iptables rules that the kernel evaluates linearly, Cilium attaches eBPF programs to network hooks inside the kernel. These programs handle service IP translation, load balancing across endpoints, network policy enforcement, and even some L7 protocol parsing without packets ever leaving kernel space. This eliminates the context switches between kernel and user space that Envoy-based sidecars require for every connection.

Internally, Cilium uses several eBPF map types to store service endpoints, identity labels, policy rules, and connection tracking state. When a packet arrives, the eBPF program attached to the network interface or socket looks up the destination service, selects a backend Pod using consistent hashing or round-robin, rewrites headers, and forwards the packet — all within a single kernel function call chain. Hubble, the observability layer built on top of Cilium, taps into these eBPF data paths to provide flow logs, DNS visibility, and HTTP metrics without injecting any proxy.

At production scale, Cilium handles over 5,000 production deployments as of 2025, including platforms at Adobe, Bell Canada, and multiple hyperscalers. Teams should monitor eBPF program load errors, map memory usage, endpoint synchronization latency, dropped flow events in Hubble, and kernel version compatibility. Cilium requires Linux kernel 5.10 or later for full feature support, and some advanced features like bandwidth manager or BBR congestion control need even newer kernels.

The non-obvious gotcha is that Cilium does not fully replicate every L7 feature of Envoy-based meshes. While it handles mTLS via SPIFFE identities, basic HTTP routing, and L7 policy, complex traffic management like retries with budgets, circuit breaking with outlier detection, or gRPC-aware load balancing may still require a sidecar or gateway proxy. Architects should map their actual L7 requirements before declaring a full service mesh unnecessary, because removing sidecars and then re-adding them later is a painful migration.

Think of a franchise restaurant chain. The headquarters defines the menu, branding, and recipes, but each location adjusts portion sizes, local ingredients, and opening hours. Karmada works the same way for Kubernetes: you define workloads once in a central control plane, and propagation policies decide which clusters receive them while override policies customize each deployment for its target environment.

Karmada, a CNCF incubation project, addresses the operational complexity of managing workloads across multiple clusters in different regions, clouds, or environments. Its architecture mirrors a single Kubernetes cluster with a dedicated API server, etcd, scheduler, and controller manager, but instead of scheduling Pods to nodes, it schedules resources to member clusters. Users interact with the Karmada API server using standard kubectl commands and Kubernetes manifests.

Internally, PropagationPolicy and ClusterPropagationPolicy define which resources go where. The policy specifies resource selectors, target clusters by name or label, and a scheduling strategy — either Duplicated (run identical copies in every target) or Divided (split replicas across clusters by weight). The Karmada scheduler evaluates cluster health, available capacity, and spread constraints before binding resources. OverridePolicy and ClusterOverridePolicy then patch manifests for each target, adjusting image registries, resource limits, environment variables, or annotations per cluster without modifying the source template.

At production scale, architects must decide several things: whether to use push mode (Karmada pushes to clusters) or pull mode (agents in member clusters pull), how to handle failover when a member cluster becomes unhealthy, how to integrate with existing GitOps tools like ArgoCD or Flux without creating conflicting reconciliation loops, and how to manage RBAC across the Karmada control plane and member clusters. Monitoring should cover resource propagation status, binding conditions, cluster health checks, scheduler decisions, and override application success.

The non-obvious gotcha is that Karmada adds a layer of abstraction that can obscure debugging. When a Deployment fails in a member cluster, the error may appear in the member cluster's events but not bubble up cleanly to the Karmada control plane. Status aggregation across clusters is improving but not yet seamless, and teams that skip investing in cross-cluster observability will struggle with incident response. Architects should also plan for the Karmada control plane itself as a critical dependency — if it goes down, existing workloads continue running but new changes cannot propagate.

Think of a library card catalog. When the library has a few thousand books, one cabinet handles lookups fine. But when the library grows to millions of books and hundreds of librarians are searching simultaneously, you either need a faster cabinet, multiple cabinets organized by subject, or a way to archive old cards. Etcd is that card catalog for Kubernetes — every resource definition, status update, and event is a card in the catalog. Etcd is the sole persistent store for Kubernetes cluster state. Every API server read and write flows through etcd, making its performance the ceiling for cluster responsiveness. For large clusters — those with tens of thousands of Pods, thousands of Services, or high churn from controllers and operators — etcd becomes the bottleneck before CPU, memory, or network do. The key metrics are fsync latency (which depends on disk IOPS), database size, number of keys, leader election frequency, and peer round-trip time between etcd members.

Internally, etcd uses a B-tree index with multi-version concurrency control, or MVCC, keeping every revision of every key until compacted. Compaction removes old revisions, and defragmentation reclaims disk space after compaction. Without regular compaction, the database grows unboundedly. Kubernetes runs automatic compaction every five minutes by default, but operators must also schedule defragmentation because compaction alone does not free physical disk space. On cloud providers, using provisioned IOPS SSD volumes (like gp3 with 6000+ IOPS on AWS) is critical because etcd performance degrades sharply when fsync latency exceeds 10 milliseconds.

At production scale, the first architectural decision is separating the events store. Kubernetes Events are high-volume, short-lived objects that create write pressure without carrying critical state. Running a dedicated etcd instance for Events reduces load on the main etcd cluster significantly. AWS EKS offers provisioned control plane tiers (XL, 2XL, 4XL) that scale etcd database limits up to 16 GB for clusters running AI and ML workloads with many custom resources. When even separated events and tuned compaction are insufficient, true etcd sharding — distributing different API groups to separate etcd clusters — or virtual clusters that maintain independent etcd instances per tenant become the next scaling lever.

The non-obvious gotcha is that etcd performance problems often manifest as API server timeouts or slow kubectl responses, and teams blame the API server rather than looking at etcd disk latency. A single slow etcd member in a three-node cluster can drag down the entire quorum because the leader waits for a majority of followers to acknowledge writes. Architects should alert on p99 fsync duration, database size approaching 8 GB, and any leader changes, because a leader election storm during high write load can cascade into control-plane unavailability.

Think of an apartment building with two security options. The sidecar model gives every apartment its own security guard who checks visitors at the apartment door — effective but expensive. The ambient model puts a guard at the building entrance who checks IDs for everyone, and only apartments that need advanced screening get a shared floor-level inspector. You get security everywhere with far fewer guards.

Istio ambient mesh reached general availability with Istio 1.22 in late 2024 and has become production-stable through 2025 and into 2026. It fundamentally changes how the data plane is deployed. Traditional Istio injects an Envoy sidecar into every Pod, which adds memory overhead (typically 50-100 MB per Pod), increases startup latency, and creates operational complexity around sidecar injection, upgrade ordering, and resource accounting. Ambient mesh removes all of this by separating L4 and L7 concerns into shared infrastructure.

The architecture has two layers. Ztunnel is a lightweight Rust-based proxy that runs as a DaemonSet on every node. It handles all L4 concerns: mTLS encryption and identity using SPIFFE certificates, TCP-level authorization policy, and basic connection routing. Ztunnel performance has improved 75 percent over recent releases and adds negligible latency. For workloads that need L7 features — HTTP routing, retries, header-based authorization, traffic splitting — architects deploy waypoint proxies, which are shared Envoy instances scoped to a namespace or service account rather than injected per Pod.

In production migration, teams should start by enabling ambient mode on a namespace using the label istio.io/dataplane-mode=ambient. Existing sidecar workloads can coexist with ambient workloads during migration. The key evaluation points are: L7 feature gaps between sidecar and waypoint proxy configurations, whether multi-cluster ambient mesh is mature enough (alpha planned for Istio 1.27), how existing Istio AuthorizationPolicy and VirtualService resources translate, and whether shared ztunnel on a node creates a blast radius concern where a ztunnel crash affects all Pods on that node.

The non-obvious gotcha is that ambient mesh changes the failure domain. In sidecar mode, a proxy crash affects one Pod. In ambient mode, a ztunnel crash can disrupt networking for every Pod on that node. This makes ztunnel reliability, resource limits, and upgrade strategy (rolling DaemonSet updates) critical. Architects should also verify that their observability stack captures ztunnel metrics and waypoint proxy metrics in the same dashboards, because the telemetry surface shifts from per-pod to per-node and per-namespace.

Think of seating guests at a wedding reception. You want to spread friends evenly across tables so no table is overcrowded and no group is isolated. The wedding planner checks how many people are at each table and seats the next guest at the most empty one, but if a table does not exist yet (no physical table has been set up), the planner cannot seat anyone there even if the venue has room. Topology spread constraints in Kubernetes work the same way.

Kubernetes topology spread constraints are declared in the Pod spec under topologySpreadConstraints. Each constraint specifies a topologyKey (a node label like topology.kubernetes.io/zone or kubernetes.io/hostname), a maxSkew (the maximum allowed difference in Pod count between the most-populated and least-populated domain), a whenUnsatisfiable behavior (DoNotSchedule or ScheduleAnyway), and a labelSelector to identify which Pods count toward the spread calculation.

Internally, the scheduler evaluates topology spread during the Filter and Score phases. In the Filter phase, it eliminates nodes where placing the Pod would violate the maxSkew when whenUnsatisfiable is DoNotSchedule. In the Score phase, it ranks remaining nodes by how well they balance the distribution. The scheduler considers the topologyKey label on existing nodes to define domains — a domain only exists if at least one node carries that label value. It then counts matching Pods per domain and calculates whether the new Pod can land in each domain without exceeding maxSkew.

At production scale, the interaction with cluster autoscaling creates subtle failures. If a node pool in one availability zone scales to zero, that zone disappears from the scheduler's topology map. The scheduler only sees zones with active nodes, so it may consider a two-zone spread sufficient even when three zones are available. When maxSkew is 1 and whenUnsatisfiable is DoNotSchedule, the scheduler can leave Pods pending because it cannot place them in a zone that has no nodes, and the autoscaler may not create a node in the missing zone because it does not see pending Pods that specifically require it. This chicken-and-egg problem is one of the most common production issues with topology spread constraints.

The non-obvious gotcha is that topology spread constraints count all matching Pods, including ones that are terminating, not-ready, or failing. During a rolling update, old Pods being terminated still count toward the spread calculation, which can cause new Pods to be unschedulable until the old ones are fully removed. Architects should set minDomains to explicitly declare how many zones the spread should consider, use node affinity in combination with spread constraints to ensure the autoscaler knows about expected zones, and monitor for unschedulable Pods with topology spread violation events.

Think of a restaurant kitchen during dinner rush. The horizontal approach is adding more cooks to handle more orders. The vertical approach is giving each cook a bigger stove and more counter space so they can cook faster. If you try both approaches based on the same signal — how backed up the order queue is — the kitchen oscillates between adding cooks and upgrading stoves in a confusing loop. You need different signals for each scaling axis.

The Horizontal Pod Autoscaler watches metrics like CPU use, memory use, or custom metrics, and adjusts the number of Pod replicas to meet a target value. The Vertical Pod Autoscaler observes actual resource consumption over time and recommends or applies changes to the resource requests and limits of individual Pods. When both use CPU as their metric, VPA may increase a Pod's CPU request, which changes the Pod's use percentage, which then triggers HPA to scale down replicas, which increases use again, creating an oscillation loop.

The recommended production pattern separates their concerns. Run VPA in recommendation mode (updateMode: Off) or target only memory, while HPA scales on CPU or custom metrics. Alternatively, use VPA in Auto mode for stateful workloads where horizontal scaling is impractical — databases, caches, or single-instance controllers — and reserve HPA for stateless services that benefit from replica scaling. Some teams use VPA recommendations to set initial resource requests in CI/CD pipelines rather than letting VPA mutate Pods at runtime, which avoids the Pod restart that VPA triggers when updating in-place.

At production scale, KEDA (Kubernetes Event-Driven Autoscaler) fills a gap that neither HPA nor VPA addresses well. KEDA scales based on external event sources — message queue depth in Kafka or SQS, pending items in a Redis stream, HTTP request rate from Prometheus, or custom metrics from any source with a KEDA scaler. Critically, KEDA can scale Deployments to zero replicas when there is no work, which standard HPA cannot do (HPA's minimum is one). This makes KEDA the right choice for batch processors, event consumers, and asynchronous workers where idle cost matters. KEDA works by deploying ScaledObject resources that create HPA objects under the hood, so it integrates with existing Kubernetes autoscaling infrastructure.

The non-obvious gotcha is that VPA in Auto mode restarts Pods when it changes resource requests, which can cause brief service disruption and interact badly with PodDisruptionBudget limits. Teams often discover this during peak traffic when VPA decides to right-size all replicas simultaneously. Architects should set VPA update policies with minReplicas and eviction requirements, and test VPA behavior during high-traffic scenarios before enabling Auto mode on critical services.

Think of shipping containers at a port. Standard OCI containers are like the steel containers that fit every crane, truck, and ship — universal, well-understood, and can carry anything. WebAssembly modules are like pneumatic tube capsules that travel at incredible speed through dedicated tubes, but they only carry small, specific items and the tube network does not reach everywhere yet. WASM is not replacing the shipping container; it is adding a fast lane for workloads that fit.

WebAssembly, originally designed for browser execution, has expanded to server-side use through the WebAssembly System Interface (WASI), which was stabilized in late 2024. WASI provides a portable, capability-based interface for WASM modules to access host features like networking, file I/O, and HTTP. On Kubernetes, WASM workloads are scheduled using standard Pod specs but with a RuntimeClass that points to a containerd shim like runwasi. The shim loads the WASM module into a runtime like Wasmtime or Spin instead of starting a Linux container. From the scheduler's perspective, the Pod looks normal; from the node's perspective, the workload starts in under a millisecond and uses a fraction of the memory.

The integration architecture uses Kubernetes RuntimeClass to route Pods to the correct runtime. A node must have the WASM containerd shim installed alongside the standard runc shim. Spin, which joined the CNCF Sandbox, provides a higher-level developer experience where WASM components are defined as Spin applications and deployed as Kubernetes Deployments with the appropriate RuntimeClass annotation. At SUSECON 2025, Fermyon demonstrated sub-0.5ms cold starts for WASM functions on Kubernetes, compared to hundreds of milliseconds for traditional container cold starts.

In production, WASM is ready for specific workload profiles: computationally intensive serverless functions, plugin systems requiring sandboxed execution, edge deployments with extreme startup-time constraints, and short-lived event processors. However, WASM containers running inside Docker's containerd integration are actually 65-325ms slower than regular containers for startup because the WASM runtime initialization inside a container adds overhead. The sub-millisecond benefit only applies to native WASM runtimes, not WASM-inside-containers. General-purpose web applications and stateful microservices still face gaps in mature networking, persistent storage, debugging tools, and library ecosystem support.

The non-obvious gotcha is assuming WASM replaces containers today. The component model — allowing WASM modules to compose with each other — is still maturing. Most production WASM on Kubernetes in 2026 runs alongside containers, not instead of them. Architects should use WASM for high-density, short-lived, compute-bound functions and keep traditional containers for everything else, planning for coexistence rather than migration.

Think of a building superintendent who checks every apartment every morning. If a tenant requested a repair, the superintendent inspects the current state, compares it to the work order, does what is needed, and notes the result. If the superintendent is interrupted mid-repair, they pick up where they left off the next morning because the work order still exists. A Kubernetes Operator controller works the same way — it continuously reconciles the world toward what the Custom Resource says it should be.

The Operator pattern was introduced to encode human operational knowledge into software. A CRD (Custom Resource Definition) defines the schema for a new Kubernetes resource — for example, a PostgresCluster resource that describes a desired database cluster. The controller watches the API server for changes to these resources and runs a Reconcile function each time a resource is created, modified, deleted, or when a requeue timer fires. The Reconcile function reads the current resource state, inspects the actual cluster state (Pods, Services, PVCs, ConfigMaps), computes the difference, and takes action to close the gap.

Internally, the controller uses an informer to cache resource state locally and a work queue to process reconciliation events. When the informer detects a change, it enqueues the resource key. The controller dequeues it and calls Reconcile. If Reconcile returns an error or a requeue request, the controller puts the key back in the queue with exponential backoff. This level-triggered design means the controller does not need to track every individual event — it just needs to know the current desired state and current actual state each time Reconcile runs.

At production scale, the most common architectural mistakes are: writing non-repeatable without side effects (idempotent) reconciliation logic that creates duplicate resources on re-runs, failing to update the status subresource with meaningful conditions so users and other controllers cannot observe progress, creating requeue storms by always returning a short requeue interval regardless of whether work is needed, skipping finalizers so external resources (cloud load balancers, DNS entries, storage) are orphaned when the Custom Resource is deleted, and attempting complex multi-step workflows in a single Reconcile call without checkpointing progress in status.

The non-obvious gotcha is that Reconcile can be called at any time for any reason — not just when the Custom Resource changes. Node pressure, informer resyncs, and dependent resource changes all trigger reconciliation. If Reconcile has side effects that are not idempotent (like sending a notification or creating a cloud resource without checking if it already exists), the operator will misbehave under real-world conditions. Architects should design every operation inside Reconcile to be safely repeatable and should test operators with chaos engineering to ensure they recover from partial failures and controller restarts.

Think of hiring staff for a catering company. Cluster Autoscaler is like posting a job ad for a specific role — you have predefined job descriptions (node groups), and when you need more people, you hire from those templates. Karpenter is like a staffing agency that looks at the actual tasks on the board, finds a person with exactly the right skills and availability, and places them immediately. The agency is faster and more flexible, but you need to trust it with your hiring criteria.

Cluster Autoscaler has been the standard Kubernetes node scaling solution since early Kubernetes versions. It works by monitoring pending Pods that cannot be scheduled due to insufficient resources, then scaling up one of the configured node groups (Auto Scaling Groups on AWS, Managed Instance Groups on GCP, or VM Scale Sets on Azure). It also scales down by identifying underutilized nodes and draining them. The key limitation is that node groups must be pre-configured with specific instance types, and the autoscaler chooses among existing groups rather than selecting the optimal instance type for each workload.

Karpenter, originally created by AWS and now a CNCF project, takes a fundamentally different approach. Instead of managing node groups, Karpenter watches for unschedulable Pods and directly provisions compute from the cloud provider API, choosing the best instance type based on Pod resource requirements, node selectors, affinity rules, and topology spread constraints. NodePool resources define constraints like allowed instance families, availability zones, capacity types (on-demand or spot), and expiration policies. Karpenter evaluates all pending Pods together and can bin-pack them onto a single optimally-sized instance rather than scaling a node group one unit at a time.

At production scale, Karpenter typically provisions nodes in under 60 seconds compared to 2-5 minutes for Cluster Autoscaler, because it skips the Auto Scaling Group scaling process and calls the EC2 API directly. Karpenter also handles node disruption proactively through consolidation, which replaces underutilized nodes with cheaper or better-fitting ones, and expiration, which rotates nodes to pick up AMI updates. However, Karpenter is currently most mature on AWS, with Azure support in development and GCP support community-driven. Teams needing multi-cloud portability or running on GKE or AKS may still prefer Cluster Autoscaler.

The non-obvious gotcha is that Karpenter's flexibility requires careful constraint definition. Without proper NodePool limits on instance families, maximum Pods per node, or total cluster capacity, Karpenter can provision very large or very expensive instances. It can also create infrastructure drift if the team's Terraform or IaC does not account for Karpenter-managed nodes. Architects should set explicit NodePool constraints, integrate Karpenter's provisioned nodes into their monitoring and cost dashboards, and understand that Karpenter manages node lifecycle independently of any external node group definition.

Think of a coworking space. Dedicated clusters are like each company renting its own building — maximum isolation but expensive. Namespaces are like shared open-plan desks — cheap but noisy, with limited privacy. Virtual clusters are like private offices inside the coworking building — each company gets its own locked room with its own reception desk, mailbox, and key card system, while sharing the building's elevators, HVAC, and parking garage. Vcluster creates a virtual Kubernetes cluster by running a minimal control plane (kube-apiserver, kube-controller-manager, and a backing store like SQLite, embedded etcd, or external etcd) as a StatefulSet inside a host cluster namespace. Tenants interact with the virtual cluster's API server using their own kubeconfig. When a tenant creates a Deployment, the virtual cluster's syncer component translates it into real resources on the host cluster — Pods run on host nodes, but they appear to live in the virtual cluster's namespace. From the tenant's perspective, they have full cluster-admin access, can install their own CRDs, create any namespace, and manage RBAC independently.

The isolation boundaries differ significantly from plain namespaces. With namespaces, tenants share the same API server, the same CRDs, the same admission policies, and the same RBAC system. One tenant's misconfigured webhook or CRD can affect all tenants. With vcluster, each tenant has independent RBAC, CRDs, admission policies, and namespace naming. The host cluster enforces compute isolation through resource quotas, network policies, and Pod security standards on the host namespace that backs the virtual cluster. The syncer translates virtual resources to host resources with name-mangling to prevent conflicts.

At production scale, virtual clusters are used for development environments, CI/CD isolation, multi-tenant SaaS platforms, and testing environments where each team needs cluster-admin access without the cost of dedicated clusters. The host cluster operator controls compute density, network isolation, storage classes, and node pool assignments. Monitoring should cover the virtual cluster control plane resource usage, syncer health, host namespace resource quotas, and any mismatch between virtual and host resource state.

The non-obvious gotcha is that vcluster does not provide kernel-level isolation. Virtual cluster Pods run on the same host nodes and share the same Linux kernel, container runtime, and network fabric as other tenants. A container escape or kernel vulnerability affects all tenants on that node. For hard multi-tenancy with untrusted workloads, architects still need dedicated node pools per tenant, Pod sandboxing (gVisor or Kata Containers), or fully separate clusters. Virtual clusters are best suited for soft multi-tenancy where tenants are trusted but need operational independence.

Think of a wedding seating planner with very specific rules. The basic planner checks table capacity and guest preferences. But this wedding also requires that certain groups of guests must all be seated simultaneously (gang scheduling), some tables are reserved for VIPs until the last minute (capacity reservation), and guests from rival families must never share an aisle (anti-affinity). Standard rules cannot express all of this, so the planner adds specialized checkers at different stages of the seating process. Kubernetes scheduler plugins work exactly this way.

The Kubernetes scheduling framework replaced the old policy-based scheduler configuration with a plugin architecture. The scheduler processes each Pod through a pipeline of extension points: PreFilter (validate and preprocess), Filter (eliminate ineligible nodes), PostFilter (handle unschedulable Pods), PreScore (prepare scoring data), Score (rank eligible nodes), Reserve (tentatively claim resources), Permit (wait or approve), PreBind (prepare external resources), and Bind (commit the Pod to a node). Each extension point can have multiple plugins that run in order.

Scheduling profiles allow a single kube-scheduler binary to expose multiple scheduler personalities. Each profile has a name and its own set of enabled, disabled, and configured plugins. A Pod selects its scheduler by setting spec.schedulerName. This means architects can run a default profile for general workloads and a specialized profile for GPU workloads, batch jobs, or latency-sensitive services without deploying separate scheduler binaries. The scheduler-plugins project from Kubernetes SIGs provides production-grade plugins like Coscheduling (gang scheduling for batch workloads that need all Pods scheduled together), Capacity Scheduling (enforcing elastic quotas across namespaces), and Trimaran (scoring based on real-time node use from metrics server).

At production scale, custom scheduler plugins require careful testing because they affect every Pod placement decision. A slow PreFilter or Score plugin increases scheduling latency for all Pods using that profile. A buggy Filter plugin can make nodes ineligible when they should be available, causing Pods to remain pending. Plugin ordering matters because earlier plugins in the chain can mask or override later ones. Architects should measure scheduler latency percentiles (scheduling_duration_seconds), unschedulable Pod counts, and plugin-specific metrics before and after enabling custom plugins.

The non-obvious gotcha is debugging scheduling failures with custom plugins. When a Pod is unschedulable, the scheduler event says which extension point rejected it, but the interaction between multiple plugins can create emergent behavior that is hard to trace. For example, a topology spread constraint combined with a capacity reservation plugin can create scenarios where Pods are pending not because of resource shortage but because the combination of constraints has no feasible solution. Architects should use the scheduler's verbose logging, the scheduling-queue metrics, and dry-run scheduling tools to validate plugin interactions before production deployment.

Imagine a bank with two headquarters in different cities. If one building loses power, customers walk into the other branch and get the same service because both branches share the same customer records and follow identical procedures. Multi-region Kubernetes is exactly this — two or more EKS clusters running identical workloads with shared data, and a traffic director (Route 53) that automatically sends customers to whichever branch is open.

Multi-region EKS starts with provisioning identical clusters in at least two AWS regions — typically us-east-1 and us-west-2 for geographic diversity. Each cluster is created using Terraform with identical node group configurations, VPC layouts, and add-on versions. The clusters share the same container images from a central ECR registry (with cross-region replication enabled), the same Helm charts from a central chart museum, and the same ArgoCD ApplicationSets that deploy workloads to both clusters simultaneously. The key principle is that both clusters must be able to serve production traffic independently at all times — this means each cluster runs the full stack including payments-api, settlements-processor, and fraud-detector, not a partial subset.

The networking layer is where the magic happens. Route 53 is configured with health-check-based failover routing policies. Each cluster exposes health endpoints through an Application Load Balancer, and Route 53 health checks probe these endpoints every 10 seconds. When the primary region fails a health check three consecutive times, Route 53 automatically updates DNS to route traffic to the secondary region. For more sophisticated routing, you can use Route 53 weighted routing to split traffic 80/20 across regions during normal operations, which keeps the secondary cluster warm and validates it continuously handles real production traffic. AWS Global Accelerator can be layered on top for anycast IP addressing, reducing DNS propagation delays during failover from minutes to seconds.

Data replication is the hardest part. For the settlements-db (Amazon RDS PostgreSQL), you set up cross-region read replicas with asynchronous replication. During failover, you promote the replica to a primary — this involves a brief write outage (typically 1-2 minutes) and potential data loss of a few seconds of transactions. For banking workloads, this is mitigated by implementing idempotency keys on all payment transactions so that retries after failover do not result in duplicate transfers. DynamoDB Global Tables provide active-active multi-region writes with last-writer-wins conflict resolution, suitable for session state and caching layers. Secrets and certificates are synchronized using AWS Secrets Manager with cross-region replication, ensuring both clusters can access TLS certificates and database credentials.

In production at a bank, multi-region adds significant operational complexity. You need synchronized deployments — a version mismatch between regions during failover can cause data compatibility issues. ArgoCD ApplicationSets with a rolling sync strategy handle this by deploying to the secondary region first, running smoke tests, then deploying to the primary. You need regular failover drills — at least quarterly — where you intentionally fail over production traffic to the secondary region and verify all services function correctly. These drills must be coordinated with compliance officers and documented for audit trails. Monitoring must be region-aware, with Grafana dashboards showing per-region SLIs and cross-region replication lag.

The biggest gotcha is cost and the active-passive trap. Running a second EKS cluster that only handles failover traffic means you are paying double for compute that sits idle 99% of the time. Smart teams run active-active, where both regions serve traffic continuously, which amortizes cost and ensures the secondary region is always production-validated. Another gotcha is cross-region latency — if payments-api in us-east-1 needs to call fraud-detector in us-west-2, that is 60-80ms of additional latency per hop. Services must be fully self-contained within each region. Finally, Terraform state management across regions requires separate state files per region with a shared backend (S3 with DynamoDB locking), and you must test that destroying and recreating one region does not affect the other.

Full DNS Resolution Path

1. Application makes DNS query for my-service.my-namespace.svc.cluster.local 2. Container's /etc/resolv.conf is configured by kubelet with nameserver <CoreDNS ClusterIP>, search <namespace>.svc.cluster.local svc.cluster.local cluster.local, and ndots:5 3. ndots behavior: If the query has fewer than 5 dots, the resolver appends each search domain and tries them ALL before trying the original name. api.example.com generates 4 queries before the actual one. 4. CoreDNS receives the query via the ClusterIP service (kube-dns), which load balances to CoreDNS pods 5. CoreDNS checks its plugins: kubernetes plugin for cluster-internal names, forward plugin for external names 6. For internal services: CoreDNS queries the Kubernetes API (via watch cache) to resolve Service → ClusterIP or Endpoints 7. Response returned through the chain back to the pod

Migration Strategy (4 Phases)

Phase 1: Foundation (Week 1-2) - Deploy identical infrastructure on target cluster using GitOps (ArgoCD) - Set up cross-cluster networking (VPC peering, transit gateway, or service mesh federation) - Establish data replication for stateful workloads (database replicas, PV snapshots)

Phase 2: Parallel Run (Week 3-4) - Deploy applications to target cluster alongside source - Route shadow/mirror traffic to target for validation (Istio traffic mirroring) - Compare responses between clusters for correctness - Monitor latency, error rates, resource usage on target

Phase 3: Gradual Traffic Shift (Week 5-6) - Use weighted DNS (Route53 weighted routing) or service mesh to shift traffic: 5% → 25% → 50% → 100% - At each step: validate SLIs, error budgets, latency percentiles - Stateful workload cut-over: promote read-replica to primary, update connection strings

Phase 4: Cleanup (Week 7) - Keep source cluster in standby for 1 week as rollback - Decommission source cluster after validation period

RBAC Architecture for Multi-Tenant Clusters

Titus Architecture vs Kubernetes

Multi-Layer Resource Management

etcd is the consistent, distributed key-value store holding all Kubernetes cluster state: Pod specs, Secrets, ConfigMaps, lease data, and RBAC rules. Every API server read and write for mutable state ultimately interacts with etcd. High availability deploys etcd as a Raft cluster where one leader handles writes and replicates log entries to followers.

Raft requires a majority of members to acknowledge a write before it commits. For N members, quorum is floor(N/2)+1. A 3-member cluster tolerates 1 failure (2 of 3 = quorum). A 5-member cluster tolerates 2 failures. Critically, a 4-member cluster still requires 3 for quorum—also tolerating only 1 failure—so the fourth node adds latency and operational cost without improving resilience. This is why etcd clusters always use odd counts.

When quorum is lost—imagine 2 of 3 etcd members permanently fail—the remaining member cannot elect a leader or accept writes. kubectl commands fail. Controllers stop reconciling. Existing Pods on worker nodes continue running (kubelet operates from local cache), but no scheduling, scaling, or Service updates occur. This split-brain protection is preferable to serving stale or divergent cluster state.

Recovery options include etcd snapshot restore to a fresh cluster, disaster recovery procedures documented per distribution (kubeadm, RKE, EKS manages etcd), and preventive monitoring of etcd latency, db size, and leader elections. Regular automated snapshots to object storage are non-negotiable for self-managed clusters.

Analogy: etcd quorum is a panel of judges requiring majority agreement before publishing a verdict. With three judges, one may be absent and rulings continue. If only one judge remains, court adjourns—no new rulings until quorum is restored—even though convicted prisoners (running Pods) remain in their cells.

Architect-level discussions cover stacked vs external etcd topology, TLS peer encryption, defragmentation schedules, and performance tuning (SSD, network latency under 10ms between members).

Monitor etcd db size and set auto-compaction retention to prevent unbounded growth. Leader election metrics (etcd_server_leader_changes_seen_total) spiking indicate network instability. For managed Kubernetes, understand what the provider backs up and test restore into an isolated environment quarterly—assuming the control plane is someone else's problem until it isn't is a common organizational failure mode.

Upgrading an operator in production is risky because operators manage custom resources representing running workloads. A new version may change CRD schemas or reconciliation logic.

Step 1: Check release notes for breaking CRD changes between versions. Step 2: Back up all existing custom resources: kubectl get sparkapplications -A -o yaml > backup.yaml Step 3: Dry-run the upgrade: helm upgrade --dry-run to preview changes. Step 4: Upgrade the controller deployment. The new controller picks up existing CRs and reconciles. Step 5: Watch controller logs for errors. Verify existing pods are not restarted.

For multi-version jumps (e.g., Spark Operator 3.1 to 3.5), check webhook configs, RBAC changes, and deprecated CRD fields. Always test on a staging namespace first.

Traditional CNIs (kube-router, Calico in iptables mode) implement NetworkPolicy by generating iptables/ipset rules and chains. The kernel walks these chains per-packet. With a handful of policies this is fine; with thousands of policies and large clusters, three things degrade: (1) iptables-restore time during scale-out — rule programming becomes a control-plane bottleneck during mass pod churn, (2) per-packet latency grows because chain traversal is not O(1), and (3) debugging is hard because the effective rule set is the product of many overlapping chains.

eBPF-based CNIs like Cilium attach programs directly to the kernel's networking hooks (TC, XDP, socket layer) and represent policy as hash maps and identity-based lookups (Cilium assigns each pod a 'security identity' derived from labels, not just an IP). Policy evaluation becomes a map lookup keyed by identity rather than chain traversal, so it scales much better with policy count and connection rate. eBPF also enables features iptables can't do efficiently: kube-proxy replacement (no more iptables/IPVS for service routing), transparent encryption, and L7-aware policies without a sidecar.

The tradeoffs: eBPF CNIs require a relatively modern kernel (5.x+ recommended for full feature set), upgrades touch live kernel-loaded programs so you need to validate kernel/Cilium version compatibility carefully, and observability tooling (Hubble for Cilium) is a new piece of operational surface to learn. For very large multi-tenant clusters with heavy NetworkPolicy usage and high pod churn, the eBPF model is generally the better fit; for smaller or more conservative environments, the simplicity and maturity of iptables-based CNIs may outweigh the scalability ceiling.

Kubernetes treats etcd as the single source of truth, and every mutating API request goes through Raft consensus: the etcd leader appends the entry to its write-ahead log, replicates it to followers, and only acknowledges the write once a majority (quorum) of members have persisted it to disk (fsync, not just memory — this is what makes etcd durable across crashes). This means write latency has a hard floor set by disk fsync latency on however many nodes are needed for quorum, and any added network latency between etcd members directly inflates write latency.

Under load, several distinct failure modes show up and need different fixes

1. Disk-bound latency: etcd is extremely fsync-sensitive. If etcd shares a disk with other I/O-heavy workloads, or runs on non-SSD storage, fsync latency spikes and every write in the cluster slows down. Diagnose with etcd_disk_wal_fsync_duration_seconds — Google/etcd's own guidance treats >10ms p99 as a red flag for production etcd. 2. Quorum loss / leader churn: if network partitions or resource starvation cause repeated leader elections (etcd_server_leader_changes_seen_total incrementing), every election pauses writes cluster-wide for the duration of the election, which can be a second or more under contention. 3. API server overload independent of etcd: large List/Watch requests (e.g., a controller doing an unbounded LIST of all Pods cluster-wide) can starve the API server's request queue even when etcd itself is healthy — distinguish this by comparing apiserver_request_duration_seconds against etcd's own latency metrics; if etcd is fast but the API server is slow, the bottleneck is API server-side (admission webhooks, conversion, or queue depth), not etcd. 4. Large key/value churn: high rates of resource updates (e.g., frequent Endpoints/EndpointSlice updates from a churny Service) inflate etcd's DB size and revision history, which slows compaction and can eventually trigger 'etcdserver: mvcc: database space exceeded' errors if compaction/defrag isn't running regularly.

The debugging path: check etcd's own metrics first (fsync latency, leader changes) to rule in/out etcd as the bottleneck, then check API server request latency by verb and resource to see if it's etcd-bound or API-server-bound, then check what's generating the write/read volume (a misbehaving controller is the most common real-world cause).

Before fine-grained kubelet authorization, many monitoring agents received nodes/proxy access because that was the practical way to reach kubelet endpoints. That permission is broad and can expose more kubelet API surface than a metrics scraper needs.

In Kubernetes v1.36, fine-grained kubelet authorization is GA and always active. The kubelet first checks a specific subresource for common paths, then falls back to nodes/proxy for compatibility. This creates a migration path: existing charts continue to work, while new or hardened deployments can move to scoped permissions.

A production migration should inventory ClusterRoles that grant nodes/proxy, identify why each workload needs kubelet access, and replace broad rules with subresources such as nodes/metrics and nodes/stats where possible. Then use admission policy or policy-as-code to flag new broad grants. Watch for mixed-version clusters and third-party Helm charts that have not yet adopted the new RBAC model.

SSA merges structurally using the OpenAPI schema, so behavior depends on how types describe their lists. Built-in types are annotated, but a CRD without x-kubernetes-list-type: map/set and merge keys is treated as atomic — an apply replaces the entire list instead of merging elements, which can wipe entries other managers added. Second, SSA only governs the apply path; a plain kubectl patch, an update, or a subresource write still overwrites an SSA-owned field and does not surface an SSA conflict, so mixing imperative edits can still bite you. Third, plan the Helm story: Helm 4 adopts server-side apply for new releases, but releases created under the old model must be migrated explicitly rather than switching automatically. Validate these before flipping SSA on cluster-wide.

The decision is about how many writers share an object. If one manifest is the only thing that ever changes an object, client-side apply is simple and adequate. But as soon as several actors co-manage it — GitOps reconciling the template, an HPA owning replicas, admission webhooks defaulting fields, operators writing status — you want per-field ownership and loud conflicts, which is exactly what SSA provides. Practically, teams standardize on stable field-manager names, remove autoscaled/defaulted fields from Git so the right owner holds them, prefer apply over imperative edits to keep managedFields clean, and plan the Helm 4 migration. The payoff is that shared objects stop silently reverting and every change is attributable.