advancedstorage

What is a StatefulSet and how does it differ from a Deployment for stateful workloads?

⚡

Quick Answer

A StatefulSet provides stable network identities, ordered deployment and scaling, and persistent storage binding for Pods that need consistent identity across restarts. Unlike Deployments which treat Pods as interchangeable, StatefulSets guarantee predictable Pod names, stable DNS entries, and one-to-one PVC binding.

Detailed Answer

Think of the difference like hotel rooms versus assigned apartments. A Deployment is like a hotel: guests (Pods) get any available room, rooms are interchangeable, and if you check out and check back in, you might get a different room with a fresh set of towels. A StatefulSet is like an apartment building: each tenant (Pod) has a specific unit number, their own mailbox with their name on it, and their personal belongings (data) stay in the apartment even if they temporarily leave. When they return, they get the exact same apartment with all their things.

A StatefulSet is a workload controller designed for applications that require one or more of: stable unique network identifiers (pod-name is predictable like db-0, db-1, db-2), stable persistent storage (each Pod gets its own PVC that survives rescheduling), ordered graceful deployment and scaling (Pods are created sequentially from index 0 to N-1 and terminated in reverse), and ordered automated rolling updates. The StatefulSet controller assigns each Pod a sticky ordinal index and a predictable (deterministic) name: {statefulset-name}-{ordinal}. Combined with a headless Service, each Pod gets a DNS record: {pod-name}.{service-name}.{namespace}.svc.cluster.local.

Internally, the StatefulSet controller reconciliation loop differs significantly from the Deployment controller. When scaling up, it creates Pods one at a time in ordinal order, waiting for each Pod to be Running and Ready before creating the next. This makes sure a database primary (index 0) is fully initialized before replicas (index 1, 2) try to connect to it. When scaling down, Pods are removed in reverse ordinal order. The volumeClaimTemplates field defines PVC templates; the controller creates a unique PVC for each Pod (named {pvc-name}-{statefulset-name}-{ordinal}) and binds it to the Pod. Critically, PVCs are NOT deleted when the StatefulSet is scaled down or deleted -- this is a safety mechanism to prevent accidental data loss. You must manually delete PVCs to reclaim storage.

At production scale, StatefulSets are used for databases (PostgreSQL, MySQL), distributed systems (Kafka, ZooKeeper, Elasticsearch), and any workload where Pod identity matters. The podManagementPolicy field controls parallelism: OrderedReady (default) enforces sequential operations, while Parallel allows all Pods to launch or terminate simultaneously -- useful for workloads like Kafka where brokers are independent. Rolling updates use the updateStrategy field: RollingUpdate (default) updates Pods from highest to lowest ordinal, and you can set a partition value to stage updates to only a subset of Pods (canary updates for stateful workloads). The OnDelete strategy requires manual Pod deletion to trigger updates, giving operators full control.

A non-obvious gotcha is that StatefulSet Pods stuck in a non-Ready state block the entire scaling and update pipeline when using OrderedReady policy. If Pod db-1 cannot become Ready, Pod db-2 will never be created. This can be catastrophic during a database cluster bootstrap if the readiness probe is misconfigured. Another trap: StatefulSets require a headless Service but the Service must exist before the StatefulSet is created -- the controller does not create it automatically. If the headless Service is accidentally deleted, DNS resolution for individual Pods breaks but the Pods continue running, creating a subtle split-brain scenario where Pods lose the ability to discover peers. Finally, the persistent PVC retention behavior changed in Kubernetes 1.27 with the persistentVolumeClaimRetentionPolicy field, which can be set to Delete to auto-cleanup PVCs on StatefulSet deletion, but it is still in beta and disabled by default.

# Headless Service required for StatefulSet DNS records apiVersion: v1 kind: Service metadata: name: inventory-db # Service name used in DNS records namespace: inventory # Inventory namespace spec: clusterIP: None # Headless service - required for StatefulSet selector: app: inventory-db # Select StatefulSet Pods ports: - port: 5432 # PostgreSQL port name: postgres # Named port for SRV records --- # StatefulSet for PostgreSQL with replication apiVersion: apps/v1 kind: StatefulSet metadata: name: inventory-db # StatefulSet name - Pods will be inventory-db-0, inventory-db-1, etc. namespace: inventory # Inventory namespace spec: serviceName: inventory-db # Must match the headless Service name replicas: 3 # Primary + 2 replicas podManagementPolicy: OrderedReady # Sequential creation (primary first) updateStrategy: type: RollingUpdate # Rolling update from highest ordinal down rollingUpdate: partition: 0 # Update all Pods (set to 2 for canary on only Pod 2) persistentVolumeClaimRetentionPolicy: # K8s 1.27+ beta feature whenDeleted: Retain # Keep PVCs when StatefulSet is deleted whenScaled: Retain # Keep PVCs when scaling down selector: matchLabels: app: inventory-db # Pod selector template: metadata: labels: app: inventory-db # Pod label matching selector spec: terminationGracePeriodSeconds: 60 # Allow PostgreSQL time for clean shutdown containers: - name: postgres # PostgreSQL container image: postgres:16.2-alpine # PostgreSQL 16 Alpine image ports: - containerPort: 5432 # PostgreSQL listening port name: postgres # Named port env: - name: POSTGRES_PASSWORD # Database password valueFrom: secretKeyRef: # From Kubernetes Secret name: inventory-db-credentials # Secret name key: password # Key in Secret - name: POD_NAME # Inject Pod name for replication setup valueFrom: fieldRef: # From Pod metadata fieldPath: metadata.name # inventory-db-0, inventory-db-1, etc. volumeMounts: - name: pgdata # Mount persistent volume mountPath: /var/lib/postgresql/data # PostgreSQL data directory readinessProbe: # Check if PostgreSQL is accepting connections exec: command: # Run pg_isready utility - pg_isready # PostgreSQL readiness check - -U # Username flag - postgres # Default user initialDelaySeconds: 10 # Wait for initial startup periodSeconds: 5 # Check every 5 seconds resources: requests: # Scheduling resources cpu: "1" # 1 CPU core memory: 2Gi # 2GB memory volumeClaimTemplates: # PVC template - creates pgdata-inventory-db-0, pgdata-inventory-db-1, etc. - metadata: name: pgdata # PVC name prefix spec: accessModes: [ReadWriteOnce] # Single node read-write storageClassName: gp3-encrypted # AWS EBS gp3 with encryption resources: requests: storage: 100Gi # 100GB per replica

What is a StatefulSet and how does it differ from a Deployment for stateful workloads?

More Kubernetes questions