Why doesn't a helm rollback always fully undo the effects of a previous upgrade, and what should you check before relying on it during an incident?
Quick Answer
helm rollback only reverts the Kubernetes resources Helm manages back to a prior revision's manifest — it does not undo side effects from hooks (like a pre-upgrade database migration Job) and does not delete PersistentVolumeClaims or other resources a chart's uninstall/rollback policy explicitly preserves. If the failed upgrade already ran a destructive or irreversible hook, rolling back the Deployment/Service manifests won't undo that hook's real-world effect.
Detailed Answer
Helm rollback works by taking the manifest recorded for a target prior revision and applying it, much like a targeted upgrade to an old state — for stateless resources (Deployments, ConfigMaps, Services) this is usually clean and sufficient. But two categories of side effects commonly surprise teams during an incident: (1) lifecycle hooks — if the failed upgrade's pre-upgrade or post-upgrade hook already ran a schema migration, seeded data, or called an external API, rolling back the application code doesn't reverse that migration; you may now have new code rolled back but a database schema that only the new code understood, causing a fresh set of errors; (2) PVCs and other resources with helm.sh/resource-policy: keep (or created by StatefulSets) are deliberately not deleted or reverted by Helm, since losing persistent data on a rollback would usually be worse than the problem being fixed.
Before relying on rollback during a live incident: check whether the failed upgrade included any hooks that could have made irreversible changes (migrations, external calls), confirm what data-bearing resources exist and whether the chart's resource-policy would leave them untouched by the rollback, and have a plan for what state the database/external systems will be in relative to the code version you're rolling back to. In practice, many teams pair application rollbacks with a corresponding, separately-tested migration rollback script rather than assuming Helm handles the whole picture.
Interview Tip
The phrase to use is 'rollback reverts the manifest, not the world' — hooks and stateful data are the two things that don't automatically follow.