How do you upgrade a Kubernetes operator in-place without disrupting running workloads?
Quick Answer
Back up existing CRs, check CRD compatibility between versions, dry-run the Helm upgrade, then upgrade the controller. Verify existing workloads continue reconciling without restarts.
Detailed Answer
Upgrading an operator in production is risky because operators manage custom resources representing running workloads. A new version may change CRD schemas or reconciliation logic.
Step 1: Check release notes for breaking CRD changes between versions. Step 2: Back up all existing custom resources: kubectl get sparkapplications -A -o yaml > backup.yaml Step 3: Dry-run the upgrade: helm upgrade --dry-run to preview changes. Step 4: Upgrade the controller deployment. The new controller picks up existing CRs and reconciles. Step 5: Watch controller logs for errors. Verify existing pods are not restarted.
For multi-version jumps (e.g., Spark Operator 3.1 to 3.5), check webhook configs, RBAC changes, and deprecated CRD fields. Always test on a staging namespace first.
Code Example
kubectl get sparkapplications -n data-pipelines -o yaml > spark-backup.yaml helm upgrade spark-operator spark-operator/spark-operator --version 3.5.0 --namespace spark-operator --dry-run helm upgrade spark-operator spark-operator/spark-operator --version 3.5.0 --namespace spark-operator kubectl logs -n spark-operator deployment/spark-operator -f
Interview Tip
Show you think about CRD backward compatibility, backup, dry-run, and staged rollout. Mention that operators manage state through CRDs and a bad upgrade can orphan workloads.