Best Practices

Why It Matters

A blanket `minAvailable: 100%` feels safe until maintenance or autoscaler consolidation needs to evict a pod. At that point the cluster cannot move workloads, node repairs stall, and operators are tempted to delete safety controls during an incident. The failure gets worse at scale because many teams copy the same template into services with different replica counts and traffic tolerance.

How to Implement

For stateless tier-1 services, start with at least three replicas and allow one voluntary disruption. For small services, decide whether maintenance flexibility or strict availability matters more, then encode that decision explicitly. Verify with `kubectl get pdb` before maintenance and test node drains in staging.

Example

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
  namespace: payments
spec:
  minAvailable: 2 # Keeps two pods serving while one is voluntarily evicted.
  selector:
    matchLabels:
      app: checkout-api # Matches the Deployment pods protected by this budget.

Why It Matters

When requests and limits diverge, Kubernetes assigns the pod a Burstable QoS class. Under node CPU pressure, the kubelet throttles Burstable pods first using CFS bandwidth quotas, which introduces unpredictable latency spikes. In a payments-api deployment processing card authorizations, a 200ms throttle window translates directly into declined transactions and customer-facing errors. The team at a fintech company lost $47K in failed authorizations during a Black Friday spike because their checkout-worker pods were Burstable and got throttled when a noisy neighbor batch job consumed spare CPU. Guaranteed QoS pods are the last to be evicted during node pressure and receive stable scheduling priority. Without matching requests and limits, you are silently accepting that your most critical workloads will degrade first during resource contention events that happen exactly when traffic matters most.

How to Implement

For every Deployment that handles synchronous user-facing traffic, set spec.containers[].resources.requests.cpu equal to spec.containers[].resources.limits.cpu. Start by profiling steady-state CPU with kubectl top pods over a 7-day window. Take the p95 value and round up to the nearest 50m increment. Set both requests and limits to that value. For memory, follow the same pattern to achieve full Guaranteed QoS. Validate by running kubectl get pod -o jsonpath='{.status.qosClass}' and confirming it returns Guaranteed. Add an OPA Gatekeeper ConstraintTemplate that rejects deployments in the payments namespace where requests do not equal limits. Run this in audit mode for one sprint, then enforce. Update your Helm values.yaml to expose a single cpuAllocation field that populates both fields identically.

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments
  labels:
    app: payments-api
    team: payments-platform
    environment: production
spec:
  replicas: 4
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      containers:
      - name: payments-api
        image: registry.internal/payments-api:v2.14.3
        resources:
          requests:
            cpu: "500m"       # Equals limit - Guaranteed QoS
            memory: "512Mi"   # Equals limit - Guaranteed QoS
          limits:
            cpu: "500m"       # No burst allowed - predictable latency
            memory: "512Mi"   # OOMKill boundary matches allocation

Why It Matters

A single privileged container is a complete node compromise waiting to happen. When a container runs with privileged: true or hostPID: true, it can read all secrets from the kubelet, escape to the host filesystem, and pivot laterally to every pod on that node. In 2025, a supply-chain attack on a logging sidecar image gained code execution inside a container that happened to have hostNetwork: true. The attacker used the host network namespace to intercept traffic from 23 other pods, including one that held database credentials in environment variables. Pod Security Standards (PSS) at the restricted level prevent this entire class of attack by blocking privileged mode, host namespaces, hostPath mounts, and dangerous capabilities. Without PSS enforcement, you rely entirely on developers remembering not to add dangerous security contexts, which fails the moment someone copies a StackOverflow snippet that includes privileged: true to fix a permission error.

How to Implement

Apply the restricted Pod Security Standard at the namespace level using the built-in Pod Security Admission controller. Label each production namespace with pod-security.kubernetes.io/enforce: restricted, pod-security.kubernetes.io/audit: restricted, and pod-security.kubernetes.io/warn: restricted. Start by applying warn mode cluster-wide to identify violations without blocking deployments. Review violations with kubectl get events --field-selector reason=FailedCreate. Fix each violating workload by removing privileged, hostNetwork, hostPID, hostIPC settings and dropping ALL capabilities then adding only NET_BIND_SERVICE if needed. For system namespaces like kube-system and monitoring, use the baseline profile instead. Create an exception list for DaemonSets that genuinely need host access like CNI plugins and node-exporter, and isolate them in dedicated namespaces with baseline enforcement.

Example

# Namespace enforcement label
apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/enforce: restricted    # Block violations
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted       # Log violations
    pod-security.kubernetes.io/warn: restricted        # Warn in kubectl output
---
# Compliant pod spec
apiVersion: v1
kind: Pod
metadata:
  name: checkout-worker
  namespace: payments
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: checkout-worker
    image: registry.internal/checkout-worker:v3.2.1
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]       # Drop every Linux capability
      runAsUser: 10001      # Non-root UID

Why It Matters

Without NetworkPolicy, every pod can talk to every other pod in the cluster on every port. This means a compromised frontend pod can directly query the database, a hijacked logging agent can reach the secrets manager, and lateral movement after any breach is trivial. In a real incident at a SaaS company, an SSRF vulnerability in the user-profile-service allowed an attacker to scan the internal network and discover an unpatched Redis instance in the cache namespace that had no authentication because it was supposed to be internal-only. The attacker extracted session tokens for 140K users. A default-deny policy would have blocked the user-profile-service from reaching the cache namespace entirely, containing the blast radius to a single service. Kubernetes does not apply any NetworkPolicy by default, so the moment you create a cluster, all pod-to-pod traffic is permitted unless you explicitly restrict it.

How to Implement

Create a default-deny ingress and egress NetworkPolicy in every namespace as part of your namespace provisioning automation. Use a Helm chart or Kustomize overlay that includes the deny-all policy alongside namespace creation. Then for each service, create an explicit allow policy that permits only the specific ingress sources and egress destinations needed. Document communication paths in a service-mesh-independent way by labeling pods with app and tier labels that NetworkPolicies reference. For egress, allow DNS on port 53 to kube-dns, then add specific CIDR blocks for external services. Validate policies using kubectl exec to test connectivity between pods. Run a network policy audit tool like netpol-viewer weekly to detect services missing policies. In CI, use conftest to verify every Deployment has a corresponding NetworkPolicy in the same directory.

Example

# Step 1: Default deny all traffic in namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}          # Applies to ALL pods in namespace
  policyTypes:
  - Ingress
  - Egress
---
# Step 2: Allow payments-api to receive from ingress-nginx only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-payments-api
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: payments-api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx
      podSelector:
        matchLabels:
          app.kubernetes.io/component: controller
    ports:
    - protocol: TCP
      port: 8080
---
# Step 3: Allow DNS egress for all pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Why It Matters

Without a PodDisruptionBudget, Kubernetes treats voluntary disruptions like node drains, cluster upgrades, and autoscaler scale-downs as unrestricted operations that can terminate all replicas simultaneously. During a routine EKS upgrade, a platform team discovered that their order-processing-service lost all three replicas when the autoscaler drained a node for consolidation. The PDB was missing, so the eviction API returned success immediately for all three pods. Customers saw a 4-minute outage during which 2,300 orders failed with 503 errors. The team had assumed that rolling update strategy alone would protect them, but rolling updates only govern Deployment rollouts, not voluntary evictions triggered by external controllers. PDBs are the only mechanism that tells the eviction API to respect application availability during disruptions that originate outside the Deployment controller. Every production Deployment, StatefulSet, and ReplicaSet serving traffic needs one.

How to Implement

Calculate your PDB parameters from actual replica count and traffic tolerance. For a Deployment with 4 replicas that can serve full load on 3, set maxUnavailable: 1 or minAvailable: 3. Never set minAvailable equal to replicas because that makes the PDB unsatisfiable for any disruption. For StatefulSets like Kafka brokers where each pod holds unique partition data, use maxUnavailable: 1 to ensure only one broker is disrupted at a time. Create PDBs in the same Helm chart as the Deployment so they deploy together. Add a CI check that every Deployment in production namespaces has a corresponding PDB with matching selector labels. Monitor with kubectl get pdb -A and alert when disruptionsAllowed drops to zero for more than 10 minutes on tier-1 services. Test PDBs by running kubectl drain in staging and verifying pods are evicted one at a time.

Example

# PDB for a 4-replica stateless API
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: order-processing-pdb
  namespace: orders
  labels:
    app: order-processing-service
    team: commerce-platform
spec:
  maxUnavailable: 1          # Allow 1 pod disrupted at a time
  selector:
    matchLabels:
      app: order-processing-service
---
# PDB for a StatefulSet (Kafka brokers)
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-broker-pdb
  namespace: streaming
spec:
  maxUnavailable: 1          # Only 1 broker offline during maintenance
  selector:
    matchLabels:
      app: kafka-broker
---
# Corresponding Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-processing-service
  namespace: orders
spec:
  replicas: 4                 # PDB allows 3 minimum
  selector:
    matchLabels:
      app: order-processing-service

Why It Matters

Using only a liveness probe or using identical settings for all probes creates a dangerous failure mode where Kubernetes kills pods that are still initializing or restarts pods that are temporarily slow but recoverable. A Java Spring Boot application at a logistics company took 45 seconds to initialize its connection pool and warm its JIT compiler. The liveness probe had initialDelaySeconds: 30, which was not enough. During every rollout, pods were killed at second 31, restarted, killed again, creating a CrashLoopBackOff that cascaded across the rolling update. The team added more replicas thinking it was a capacity problem. The real issue was conflating startup health with runtime health. Startup probes exist specifically to handle slow-starting containers without compromising the fast failure detection that liveness probes provide during steady state. Without a startup probe, you must set a large initialDelaySeconds on liveness, which means genuinely crashed pods take that long to be detected.

How to Implement

Configure three distinct probes with different purposes and thresholds. The startup probe runs first and has generous timing: failureThreshold of 30 with periodSeconds of 10 gives the container 300 seconds to start. Once the startup probe succeeds once, Kubernetes activates the liveness and readiness probes. The liveness probe detects deadlocks and unrecoverable states with moderate timing: periodSeconds 10, failureThreshold 3, giving 30 seconds before restart. The readiness probe controls traffic routing and should be aggressive: periodSeconds 5, failureThreshold 2, so unhealthy pods are removed from the Service endpoint within 10 seconds. Use different endpoints if possible: /healthz/started for startup, /healthz/live for liveness checking thread deadlocks, and /healthz/ready for readiness checking downstream dependencies. Never use the same endpoint and thresholds for all three probes because their purposes are fundamentally different.

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-auth-service
  namespace: identity
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: user-auth-service
        image: registry.internal/user-auth-service:v4.7.0
        ports:
        - containerPort: 8080
        # STARTUP: generous - wait for JVM warmup + DB migration
        startupProbe:
          httpGet:
            path: /healthz/started     # Returns 200 once app is initialized
            port: 8080
          periodSeconds: 10
          failureThreshold: 30         # Up to 300s to start
          successThreshold: 1
        # LIVENESS: detect deadlocks, not slow responses
        livenessProbe:
          httpGet:
            path: /healthz/live        # Checks thread pool not deadlocked
            port: 8080
          periodSeconds: 10
          failureThreshold: 3          # Kill after 30s unresponsive
          successThreshold: 1
          timeoutSeconds: 3
        # READINESS: fast removal from service endpoints
        readinessProbe:
          httpGet:
            path: /healthz/ready       # Checks DB + Redis connections
            port: 8080
          periodSeconds: 5
          failureThreshold: 2          # Remove from LB after 10s
          successThreshold: 2          # Re-add after 2 consecutive passes
          timeoutSeconds: 3

Why It Matters

Without consistent labels, you cannot correlate metrics, logs, and traces to the owning team, cannot filter Prometheus queries by environment, cannot build Grafana dashboards that aggregate by service tier, and cannot enforce NetworkPolicies or PDBs that use label selectors. A platform team at an e-commerce company discovered during a P1 incident that 40% of their pods had no team label. When the payments service degraded, they could not quickly identify which team owned the upstream dependency causing the issue because kubectl get pods showed 200 pods with inconsistent naming. The mean time to resolution doubled from 12 minutes to 28 minutes because three teams had to be paged to determine ownership. Labels are the foundation of every Kubernetes selector mechanism: Services route traffic by label, HPA targets by label, PDBs protect by label, and monitoring tools aggregate by label. Inconsistent labeling makes every operational tool partially blind.

How to Implement

Define a labeling standard document that mandates these labels on every resource: app.kubernetes.io/name (service name), app.kubernetes.io/version (deployed version), app.kubernetes.io/component (api, worker, cache), team (owning team slug), environment (production, staging, dev), and tier (frontend, backend, data). Enforce labels using OPA Gatekeeper or Kyverno with a policy that rejects resources missing required labels. Add a ValidatingAdmissionPolicy in Kubernetes 1.28+ as a lightweight alternative. In Helm charts, template labels from values.yaml so developers cannot forget them. Configure Prometheus relabeling rules to extract team and environment labels from pod labels into metric labels, enabling per-team SLO dashboards. Run a weekly audit job that reports unlabeled resources to team Slack channels.

Example

# Kyverno policy enforcing required labels
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-standard-labels
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-required-labels
    match:
      any:
      - resources:
          kinds: ["Deployment", "StatefulSet", "DaemonSet"]
    validate:
      message: "Missing required labels: app, version, team, environment"
      pattern:
        metadata:
          labels:
            app.kubernetes.io/name: "?*"
            app.kubernetes.io/version: "?*"
            team: "?*"
            environment: "?*"
---
# Properly labeled Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inventory-service
  namespace: catalog
  labels:
    app.kubernetes.io/name: inventory-service
    app.kubernetes.io/version: v2.8.1
    app.kubernetes.io/component: api
    team: catalog-engineering
    environment: production
    tier: backend
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: inventory-service
  template:
    metadata:
      labels:
        app.kubernetes.io/name: inventory-service
        app.kubernetes.io/version: v2.8.1
        team: catalog-engineering
        environment: production

Why It Matters

Every package in a container image is attack surface. A standard ubuntu:22.04 base image contains 300+ packages including curl, wget, bash, apt, and network utilities that attackers use for reconnaissance and data exfiltration after gaining code execution. In 2025, a vulnerability in a libxml2 version shipped in a debian:bullseye base image gave attackers remote code execution in a notification-service that never used XML parsing directly. The library existed in the image purely because the base image included it. With a distroless image, that library would not have been present and the CVE would have been non-exploitable. Distroless images contain only your application binary and its runtime dependencies, reducing the CVE surface by 70-90% compared to full OS images. They also have no shell, so even if an attacker gains code execution, they cannot spawn an interactive session, install tools, or easily pivot. Additionally, smaller images mean faster pulls during scale-out events, reducing HPA response time from 45 seconds to 12 seconds in cold-start scenarios.

How to Implement

Standardize on gcr.io/distroless/static for Go binaries, gcr.io/distroless/java21-debian12 for Java services, and gcr.io/distroless/cc for C++ applications. For Python and Node.js services that need a minimal runtime, use chainguard images like cgr.dev/chainguard/python:latest-dev for building and cgr.dev/chainguard/python:latest for runtime. Configure your container registry (Harbor, ECR, or Artifact Registry) with a policy that only allows pulls from an approved base image allowlist. Set up Trivy or Grype scanning in CI that fails the build if the image has any critical or high CVEs. Use multi-stage Docker builds where the first stage uses a full image for compilation and the final stage copies only the binary into distroless. Add an OPA policy that rejects pods with images from unapproved registries like docker.io or quay.io unless explicitly exempted.

Example

# Multi-stage Dockerfile using distroless
# Stage 1: Build
FROM golang:1.22-bookworm AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /checkout-worker ./cmd/worker

# Stage 2: Runtime - distroless (no shell, no package manager)
FROM gcr.io/distroless/static:nonroot
COPY --from=builder /checkout-worker /checkout-worker
USER 65534:65534
ENTRYPOINT ["/checkout-worker"]
---
# Kyverno policy restricting image registries
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  rules:
  - name: validate-registries
    match:
      any:
      - resources:
          kinds: ["Pod"]
    validate:
      message: "Images must come from approved registries"
      pattern:
        spec:
          containers:
          - image: "registry.internal/* | gcr.io/distroless/*"

Why It Matters

In a multi-tenant cluster where multiple teams deploy to separate namespaces, a single team can accidentally or intentionally consume all available cluster resources if no quotas exist. A data engineering team deployed a Spark job that requested 64 CPU cores and 256Gi of memory across 40 pods in their analytics namespace. Within minutes, the cluster autoscaler had provisioned 12 new nodes. Meanwhile, the payments team's HPA triggered a scale-out that could not schedule new pods because the scheduler saw no available capacity on existing nodes and the autoscaler hit its maxNodes limit. The payments service degraded for 18 minutes during peak checkout traffic. ResourceQuota caps total resource consumption per namespace, ensuring no team can starve others. LimitRange sets defaults and maximums per container, catching the developer who forgets to set resource requests entirely and deploys a pod that either gets BestEffort QoS or requests absurd amounts. Together they implement resource isolation without requiring separate clusters per team, saving significant infrastructure cost.

How to Implement

For each namespace, create a ResourceQuota that limits total CPU requests, memory requests, CPU limits, memory limits, and object counts like pods and services. Base the quota on the team's actual capacity needs plus 30% headroom for scaling. Create a LimitRange that sets default requests and limits for containers that do not specify them, and caps the maximum any single container can request. This prevents a single pod from consuming the entire namespace quota. Apply quotas using a namespace provisioning controller or Helm chart that templates quota values from a central configuration. Monitor quota usage with kube_resourcequota metrics in Prometheus and alert at 80% consumption so teams can request increases before hitting hard limits. Review and adjust quotas quarterly based on actual usage patterns from metrics.

Example

# ResourceQuota for the analytics namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: analytics-team-quota
  namespace: analytics
spec:
  hard:
    requests.cpu: "32"          # Max 32 CPU cores requested across all pods
    requests.memory: "64Gi"     # Max 64Gi memory requested
    limits.cpu: "48"            # Max 48 CPU cores in limits
    limits.memory: "96Gi"       # Max 96Gi memory in limits
    pods: "100"                 # Max 100 pods in namespace
    services: "20"              # Max 20 services
    persistentvolumeclaims: "30"
---
# LimitRange for per-container defaults and caps
apiVersion: v1
kind: LimitRange
metadata:
  name: container-limits
  namespace: analytics
spec:
  limits:
  - type: Container
    default:                    # Applied when no limits specified
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:             # Applied when no requests specified
      cpu: "250m"
      memory: "256Mi"
    max:                        # Hard ceiling per container
      cpu: "4"
      memory: "8Gi"
    min:                        # Floor - prevents tiny containers
      cpu: "50m"
      memory: "64Mi"
  - type: PersistentVolumeClaim
    max:
      storage: "100Gi"          # No single PVC larger than 100Gi

Why It Matters

Resource requests and limits are the foundation of Kubernetes scheduling and stability. Requests guarantee a minimum resource allocation, determining the pod's Quality of Service (QoS) class. Limits prevent a single pod from consuming all node resources, which causes 'noisy neighbor' issues and node instability. Without requests, the scheduler places pods without knowing their needs, leading to over-saturated nodes. Without limits, a memory leak or CPU-intensive bug in one pod can trigger cascading failures across the entire node. For memory, the limit defines the boundary for OOM (Out-of-Memory) kills. For CPU, the limit defines a throttling ceiling (CFS quota). In production, defining these values is non-negotiable for achieving predictable performance, SLOs, and cost management.

How to Implement

Start by measuring actual usage with metrics-server, Prometheus, or a vendor APM over a representative traffic window (e.g., 7 days) to capture peaks. Set requests near your p95 usage and limits with a modest headroom, typically 1.5–2x requests. This places your pod in the 'Burstable' QoS class, offering a good balance of cost and performance. For critical, latency-sensitive services, set requests equal to limits to achieve the 'Guaranteed' QoS class, which gives the pod the highest scheduling priority and makes it the last to be evicted. Use Vertical Pod Autoscalers (VPA) in 'recommendation' mode to get data-driven suggestions for requests. Enforce defaults in each namespace with a `LimitRange` object and cap aggregate consumption per team with a `ResourceQuota`. Automate validation in CI using policy-as-code tools like OPA/Gatekeeper or Kyverno to reject deployments that omit resource definitions.

Example

# Deployment excerpt with requests and limits
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  template:
    spec:
      containers:
        - name: api
          image: myregistry/api:v2.4.1
          resources:
            requests:
              cpu: "250m"      # scheduler reserves this
              memory: "512Mi"
            limits:
              cpu: "500m"      # throttled beyond this (CFS quota)
              memory: "768Mi"  # OOMKill if exceeded
---
# Namespace default via LimitRange
apiVersion: v1
kind: LimitRange
metadata:
  name: default-resources
  namespace: production
spec:
  limits:
    - default:
        cpu: "500m"
        memory: "512Mi"
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      type: Container

Why It Matters

Readiness and liveness probes answer different questions. Readiness determines whether a pod should receive traffic; liveness determines whether the container should be restarted. Conflating them—or pointing both at the same shallow health endpoint—causes cascading failures during slow startups, dependency blips, or planned maintenance. A liveness probe that fires while the app is still warming caches will restart the container in a crash loop, never allowing it to become ready. A readiness probe that doubles as liveness may keep a legitimately unhealthy pod in the Service endpoints, sending user traffic to failing instances. During rolling updates, misconfigured probes stall rollouts or flip-flop pod readiness. Correct probe design protects user experience: readiness gates traffic until dependencies are satisfied, while liveness only intervenes when the process is truly deadlocked and cannot recover without restart.

How to Implement

Define a `readinessProbe` that checks all critical dependencies: database connectivity, cache warm-up, feature flag availability. This probe should fail fast if a dependency is lost, signaling the endpoint slice controller to remove the pod from service. Define a separate `livenessProbe` with a lightweight check that only verifies the process is running and responsive (e.g., a simple HTTP 200 on `/healthz`). This probe should be tolerant of transient dependency failures. For slow-starting applications (like Java services), always use a `startupProbe`. The startup probe disables the liveness probe until the application is fully initialized, preventing premature restarts. Configure generous `failureThreshold` and `periodSeconds` for the liveness probe to avoid restarting on temporary stalls, but use tighter timings for the readiness probe to react quickly to real issues. Since Kubernetes 1.23, you can use gRPC probes for gRPC services, which is more efficient than `exec`-ing `grpc-health-probe`.

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  template:
    spec:
      containers:
        - name: app
          image: myregistry/web:1.8.0
          ports:
            - containerPort: 8080
          # Slow JVM startup: protect liveness until ready
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            failureThreshold: 30
            periodSeconds: 10
          # Liveness: process alive only (no DB check)
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 15
            failureThreshold: 3
          # Readiness: full dependency check before traffic
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
            failureThreshold: 2

Why It Matters

Voluntary disruptions—node drains, cluster upgrades, spot instance reclaim, autoscaling scale-down—can remove multiple pods simultaneously unless you constrain them. Without a PodDisruptionBudget (PDB), Kubernetes may evict enough replicas to drop below your availability target during a routine maintenance window. A PDB that is too strict blocks node drains indefinitely, delaying security patches and causing operational toil. A PDB that is too loose allows quorum loss for stateful services or breaching n-1 redundancy for APIs. PDBs interact with Deployment maxUnavailable, replica count, and topology spread; misalignment produces surprising behavior during rolling updates versus evictions. For production services, PDBs translate business continuity requirements—"always keep at least two instances available"—into enforceable cluster policy. They are essential for HA services but must be calibrated to replica count and maintenance cadence.

How to Implement

Create a PDB for every production workload with more than one replica. Use `minAvailable` for services where you need a fixed number of available pods (e.g., a 3-node stateful set needing quorum should have `minAvailable: 2`). Use `maxUnavailable` for stateless services where you can tolerate a certain number of pods being down (e.g., a 10-replica API can tolerate `maxUnavailable: 2`). Remember that PDBs only protect against *voluntary* disruptions (like node drains), not *involuntary* ones (like node failure). Your total replica count must account for both. For example, to survive a zone failure (involuntary) and a node drain (voluntary) simultaneously, you need more replicas than your PDB alone would suggest. Since Kubernetes 1.26, you can set `unhealthyPodEvictionPolicy` to `AlwaysAllow` to allow evicting misbehaving pods (e.g., stuck in `Pending` or failed readiness) even if it violates the PDB, which can unblock stuck node drains.

Example

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-server-pdb
  namespace: production
spec:
  # Keep at least 3 of 5 replicas during voluntary disruption
  minAvailable: 3
  selector:
    matchLabels:
      app: api-server
      tier: backend
---
# Verify before node drain
# kubectl get pdb -n production
# kubectl drain worker-3 --ignore-daemonsets --delete-emptydir-data

Why It Matters

By default, every pod in a Kubernetes cluster can reach every other pod on any port—a flat network trust model that violates zero-trust principles. A compromised workload can scan the cluster, reach admin interfaces, exfiltrate data from databases, or pivot to the Kubernetes API via overly permissive service accounts. NetworkPolicies provide L3/L4 segmentation but are opt-in; without a default-deny baseline, adding permissive policies later does not reduce attack surface. Regulatory frameworks and internal security standards increasingly require micro-segmentation. Default-deny forces teams to document intended communication paths, surfacing hidden dependencies during migration. Combined with namespace isolation and egress controls, default-deny dramatically limits blast radius of RCE or supply-chain compromise in any single pod. Lateral movement after initial compromise is the most common path to catastrophic breach in container environments.

How to Implement

First, ensure your CNI (e.g., Calico, Cilium) supports and enforces NetworkPolicy. In each application namespace, apply a default-deny policy for both ingress and egress. Then, layer in explicit `allow` policies. A standard baseline includes allowing DNS egress to `kube-dns` and ingress from your monitoring namespace (for Prometheus scraping). For application traffic, use `podSelector` and `namespaceSelector` to allow ingress from specific upstream services or the ingress controller, and egress to specific downstream services or databases. Avoid using `ipBlock` with static IPs, as they are brittle; prefer selectors. For external traffic, use egress rules to whitelist specific CIDRs or FQDNs (if your CNI supports it, like Cilium) to prevent data exfiltration to arbitrary internet endpoints.

Example

# Step 1: default-deny all ingress and egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}   # all pods in namespace
  policyTypes:
    - Ingress
    - Egress
---
# Step 2: allow ingress from ingress-nginx only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080

Why It Matters

Mutable tags like latest or even semver tags can be repointed to different image content without changing the manifest string in your Deployment. A compromised registry, mistaken push, or supply-chain attack can replace v1.2.3 with malicious layers while your GitOps repo still shows the same tag—Kubernetes will pull the new content on next rollout or imagePullPolicy: Always refresh. Pinning by digest (@sha256:...) binds the pod spec to an immutable content address, ensuring what you tested in CI is exactly what runs in production. Digest pinning is a cornerstone of SLSA and reproducible deployments. It also prevents subtle drift between nodes during rolling updates when some nodes cached an older tag layer. For security and auditability, digest references belong in production manifests; tags remain useful in development for convenience.

How to Implement

Your CI/CD pipeline should be responsible for resolving tags to digests. After building and pushing an image with a tag (e.g., `v1.2.3`), the pipeline should use a tool like `crane digest` or `docker buildx imagetools inspect` to retrieve the immutable `sha256` digest. This digest should then be used to patch the Kubernetes manifest before applying it. For GitOps workflows, the pipeline should commit the updated digest back to the Git repository. This creates a closed loop where every deployment is explicitly tied to immutable image content. Augment this with an admission controller (like Kyverno or Gatekeeper) that enforces a policy requiring all images in production namespaces to be referenced by digest, rejecting any that use mutable tags.

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  template:
    spec:
      containers:
        - name: payment
          # Immutable reference — content cannot change silently
          image: registry.example.com/payment@sha256:abc123def456789...
          imagePullPolicy: IfNotPresent
---
# CI: resolve digest before deploy
# DIGEST=$(crane digest registry.example.com/payment:v2.1.0)
# kubectl set image deploy/payment-service payment=registry.example.com/payment@${DIGEST}

Why It Matters

In shared clusters, a single team can consume unbounded CPU, memory, object count, or persistent volume claims, starving others and forcing expensive emergency scale-out. Without ResourceQuotas, one runaway CI job or misconfigured HorizontalPodAutoscaler can create thousands of pods or exhaust cloud storage budgets. Quotas translate organizational budgets into hard Kubernetes limits at the namespace boundary, making capacity a first-class contract between platform and application teams. They complement LimitRange (per-pod defaults) by capping aggregate consumption. Quotas also encourage teams to right-size workloads and clean up orphaned resources. For platform operators, quotas simplify chargeback, prevent accidental cluster saturation, and provide predictable headroom for critical system namespaces. Finance and engineering leadership increasingly expect per-team cost visibility tied directly to Kubernetes resource consumption metrics.

How to Implement

Define a `ResourceQuota` for each namespace that sets hard limits on aggregate `requests.cpu`, `requests.memory`, `limits.cpu`, and `limits.memory`. Also, set quotas on object counts like `pods`, `persistentvolumeclaims`, and `services.loadbalancers`. It is crucial to also apply a `LimitRange` in the same namespace; otherwise, pods created without explicit requests/limits will not be constrained by the quota's CPU/memory limits (though they will count towards the pod limit). Monitor quota usage via `kube-state-metrics` and create alerts that fire when a namespace exceeds 80% of its quota, giving the platform and application teams time to react before deployments start failing. For chargeback, use tools like OpenCost or Kubecost, which can report on resource consumption per namespace, aligning technical quotas with financial budgets.

Example

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "100"
    persistentvolumeclaims: "10"
---
# Check utilization
# kubectl describe resourcequota team-alpha-quota -n team-alpha
# kubectl get resourcequota -n team-alpha

Why It Matters

Containers that run as UID 0 inherit root privileges inside the container namespace. If an attacker exploits an application vulnerability, root access simplifies escaping to the host—especially when combined with misconfigured capabilities, writable hostPath mounts, or privileged mode. Running as a non-root user follows the principle of least privilege and aligns with Pod Security Standards (restricted profile). Many compliance frameworks (PCI-DSS, SOC2) require demonstrable non-root execution. Even without full container breakout, root inside the container enables modifying system binaries, installing tooling, and tampering with process trees. Non-root execution limits damage from supply-chain compromised base images. Platform teams should treat runAsNonRoot: true as a default, not an exception for production workloads. CIS Kubernetes Benchmark and most enterprise security baselines flag root containers as a critical finding requiring immediate remediation.

How to Implement

In your Dockerfile, create a non-root user and switch to it with the `USER` instruction. In your Kubernetes Deployment manifest, set `securityContext.runAsNonRoot: true` and `securityContext.runAsUser` to a high-numbered UID (e.g., 65534). For containers that need to write to shared volumes, set `securityContext.fsGroup` to the same UID. At the container level, set `securityContext.allowPrivilegeEscalation: false` and `securityContext.capabilities.drop: ["ALL"]`. Use the built-in Pod Security Admission controller with the `restricted` profile at the namespace level to enforce these settings automatically. This prevents any new workloads from running as root. For existing workloads, use a policy tool like Kyverno in `audit` mode to find and remediate containers still running as root.

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-api
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: registry.example.com/secure-api@sha256:...
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

Why It Matters

A Deployment rollout can hang indefinitely—ImagePullBackOff, CrashLoopBackOff, failed readiness probes, or insufficient cluster capacity—while kubectl rollout status appears to wait forever and CI pipelines block without a clear failure signal. Kubernetes defaults progressDeadlineSeconds to 600 seconds, but many teams never tune it or disable implicit failure detection by setting it to zero. Without a deadline, stale ReplicaSets may retain partially updated state, leaving a mix of old and new versions without triggering rollback. Explicit deadlines convert silent stalls into actionable DeploymentProgressing=False conditions, enabling automated rollback in GitOps controllers and alerting in monitoring systems. For production, detecting a failed rollout within minutes—not hours—directly reduces MTTR and prevents shipping broken config to a growing percentage of traffic. Silent partial rollouts are especially dangerous during off-hours deploys when no engineer is watching the cluster dashboard.

How to Implement

Set `spec.progressDeadlineSeconds` on every production Deployment to a reasonable value that reflects your application's startup time and readiness probe configuration. A common starting point is 600 seconds (10 minutes). If a new ReplicaSet fails to make progress within this window, the Deployment controller will update its status with `reason: ProgressDeadlineExceeded`. This condition is a clear failure signal that can be used by CI/CD systems (like Argo Rollouts or Flux) to trigger an automatic rollback to the last known-good revision. Monitor for this condition using `kube-state-metrics` (`kube_deployment_status_condition{condition="Progressing", status="false"}`) to alert the on-call team that a deployment is stuck. Never set this value to zero, as that disables the feature entirely.

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-service
spec:
  replicas: 6
  progressDeadlineSeconds: 600  # fail rollout after 10 min stuck
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: checkout-service
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout@sha256:...
---
# Diagnose stuck rollout
# kubectl rollout status deployment/checkout-service --timeout=620s
# kubectl describe deployment checkout-service | grep -A5 Conditions

Why It Matters

Labels are the primary indexing mechanism in Kubernetes—Services route traffic via selectors, Deployments manage ReplicaSets via matchLabels, NetworkPolicies filter endpoints, Prometheus ServiceMonitors scrape targets, and cost allocation tools group spend by label. Inconsistent labeling—app versus app.kubernetes.io/name, missing version or component labels—breaks cross-cutting queries and causes subtle production bugs where Services select zero pods or select the wrong ones after a refactor. The recommended labels (app.kubernetes.io/name, instance, version, component, part-of, managed-by) enable interoperability with Helm, Argo CD, and ecosystem tools. Consistent labels also power effective kubectl -l filtering during incidents and allow topology spread constraints and affinity rules to target the right pod sets. Treat labels as API contracts, not afterthought annotations. During a Sev-1, inconsistent labels can add minutes to identifying affected pods across namespaces.

How to Implement

Standardize on the Kubernetes recommended labels (`app.kubernetes.io/name`, `.../instance`, `.../version`, `.../component`, `.../part-of`, `.../managed-by`). Use a tool like Kustomize's `commonLabels` feature or a Helm `_helpers.tpl` template to apply these labels consistently to every resource. Crucially, ensure that a Deployment's `spec.selector.matchLabels` exactly matches the labels in its `spec.template.metadata.labels`. A mismatch here is a common error that causes Deployments to lose control of their pods. Enforce your labeling standard using a policy-as-code tool like Kyverno or Gatekeeper, which can mutate objects to add required labels or validate that they exist before admission. This prevents configuration drift and ensures all resources are discoverable and governable.

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-v2
  labels:
    app.kubernetes.io/name: orders
    app.kubernetes.io/instance: orders-prod
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/component: api
    app.kubernetes.io/part-of: commerce
    app.kubernetes.io/managed-by: helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: orders
      app.kubernetes.io/instance: orders-prod
  template:
    metadata:
      labels:
        app.kubernetes.io/name: orders
        app.kubernetes.io/instance: orders-prod
        app.kubernetes.io/version: "2.4.0"
---
apiVersion: v1
kind: Service
metadata:
  name: orders
spec:
  selector:
    app.kubernetes.io/name: orders
    app.kubernetes.io/instance: orders-prod

Why It Matters

Without topology spread, the scheduler may place all replicas on a single node or in one availability zone—especially under resource pressure or when using pod affinity naively. A single node failure or zone outage then removes your entire service despite running multiple replicas. Topology Spread Constraints distribute pods across failure domains (topology.kubernetes.io/zone, kubernetes.io/hostname) according to skew limits you define. Unlike simple podAntiAffinity, spread constraints offer maxSkew semantics that gracefully degrade when capacity is uneven—ScheduleAnyway allows placement with warning rather than blocking entirely. For regional clusters, zone spread is essential for HA; for large nodes, hostname spread reduces noisy-neighbor resource contention. Spread constraints interact with PDBs and cluster autoscaler; together they define resilient placement. Cloud provider SLAs for individual zones do not cover your application availability—you own cross-zone resilience.

How to Implement

For any production service with three or more replicas, add a `topologySpreadConstraint` to the pod spec. Use `topologyKey: topology.kubernetes.io/zone` to ensure pods are balanced across availability zones, protecting against a zonal outage. Use `maxSkew: 1` to ensure the number of pods in any one zone does not exceed the number in any other zone by more than one. Set `whenUnsatisfiable: DoNotSchedule` to enforce this strictly. For very large deployments, you can add a second constraint with `topologyKey: kubernetes.io/hostname` and `whenUnsatisfiable: ScheduleAnyway` to encourage, but not require, spreading across individual nodes within a zone. This provides a good balance of high availability and scheduling flexibility.

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-api
spec:
  replicas: 6
  template:
    spec:
      topologySpreadConstraints:
        # Balance across availability zones
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: catalog-api
        # Soften node-level stacking
        - maxSkew: 2
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: catalog-api
      containers:
        - name: catalog
          image: registry.example.com/catalog@sha256:...
---
# Verify distribution
# kubectl get pods -o wide -l app.kubernetes.io/name=catalog-api

Why It Matters

Kubernetes v1.36 moves in-place vertical scaling for pod-level resources to beta and enables it by default through its feature gate. It lets operators update the aggregate `.spec.resources` budget for a running Pod, often without restarting containers. That is powerful during traffic spikes, but it still depends on node capacity, cgroup v2, CRI support, and each container's resize policy.

How to Implement

Use in-place pod-level resize for carefully bounded operational adjustments. Confirm cgroup v2 and runtime support, define container `resizePolicy`, and monitor `PodResizePending` and `PodResizeInProgress` conditions after patching. Build runbooks that explain when a resize is allowed, what signal justifies it, and when a rollout or horizontal scaling is safer.

Example

kubectl patch pod shared-pool-app --subresource resize --patch '{"spec":{"resources":{"limits":{"cpu":"4"}}}}'

kubectl get pod shared-pool-app -o jsonpath='{.status.conditions[?(@.type=="PodResizePending")].reason}'
# Check whether the node admitted the new pod-level resource envelope.

Why It Matters

Kubernetes v1.36 continues the maturation of Dynamic Resource Allocation, with key governance and selection capabilities reaching stable status. For GPU-heavy, accelerator-heavy, or multi-tenant AI platforms, stable DRA APIs make device allocation policy less dependent on ad hoc scheduler conventions or vendor-specific workarounds.

How to Implement

Model device access as a platform contract. Define who can request scarce hardware, how priorities are expressed, what administrative overrides are allowed, and how allocation failures surface to users. Pair DRA adoption with quota policy, workload identity, node isolation, and cost reporting so GPU access is observable and auditable.

Example

kubectl get resourceclaims -A
kubectl describe resourceclaim -n ml-training training-gpu-claim
# Review claim status, driver allocation, and why a workload is waiting for a device.