How do you run ClickHouse on Kubernetes using the Altinity Operator, and what are the operational challenges?

Think of the Altinity Operator as a specialized property manager for a high-rise building (Kubernetes cluster) who knows exactly how to build and maintain database apartments (ClickHouse nodes). Instead of you manually wiring electricity, plumbing, and internet to each apartment, you describe what you want (3 shards, 2 replicas, 500GB each) and the property manager handles construction, maintenance, and repairs according to database-specific best practices.

The Altinity ClickHouse Operator (also known as clickhouse-operator) watches for ClickHouseInstallation custom resources and reconciles them into Kubernetes primitives. A single ClickHouseInstallation CR creates: StatefulSets (one per shard-replica combination), PersistentVolumeClaims (for data and logs), Services (per-replica for direct access, per-shard for load balancing, cluster-wide for Distributed table queries), ConfigMaps (for ClickHouse server configuration, users, and cluster topology), and optionally a ClickHouse Keeper StatefulSet for coordination.

Internally, the Operator handles the complex orchestration that makes running a stateful distributed database on Kubernetes viable. During rolling upgrades, it restarts one replica at a time, waits for the replication queue to drain (ensuring the restarted replica has caught up), and only proceeds to the next replica when the cluster is healthy. It injects macros (shard number, replica name) into each pod's configuration so ReplicatedMergeTree tables automatically know their identity. The Operator also reconciles cluster topology changes: adding a shard creates new StatefulSets and updates the remote_servers configuration on all existing nodes.

At production scale, the operational challenges are significant. Persistent storage must use high-performance storage classes (gp3, io2 on AWS, pd-ssd on GCP) because ClickHouse is I/O intensive during merges. Pod anti-affinity rules must spread replicas of the same shard across different nodes and availability zones to survive node or zone failures. Resource requests and limits must account for merge operations that temporarily spike CPU and memory usage (set memory limits 30% above normal to prevent OOM during large merges). Schema changes (ALTER TABLE) must be coordinated across all replicas via ON CLUSTER syntax.

The non-obvious gotcha is PersistentVolume expansion. When a ClickHouse node fills its disk, you need to resize the PVC, which requires the storage class to support volume expansion (allowVolumeExpansion: true) and may require a pod restart depending on the CSI driver. If the disk fills completely before you resize, ClickHouse enters read-only mode and cannot even perform background merges, creating a deadlock where the table is both full and has too many parts. Production clusters should alert at 70% disk usage and automate PVC expansion or have a TTL policy that prevents unbounded growth.