How do you implement disaster recovery and migration strategies for Flux-managed clusters, including backup, restore, and cross-cluster failover?

Disaster recovery for Flux-managed clusters benefits from the fundamental GitOps principle that Git is the single source of truth. Since all Kubernetes manifests, Kustomizations, HelmReleases, and configuration are stored in version-controlled repositories, recovering a cluster is theoretically as simple as bootstrapping Flux on a new cluster and pointing it at the same Git repositories. In practice, however, several complexities make disaster recovery more nuanced: encrypted secrets must be recoverable with the correct decryption keys, stateful workloads need data recovery beyond just manifest reapplication, Flux internal resources like suspend states and last-applied revisions need consideration, and the ordering of resource creation during recovery must respect dependency chains.

The primary recovery procedure begins with provisioning a new Kubernetes cluster, either in the same region for single-region failures or in a different region for regional outages. Flux is then bootstrapped using flux bootstrap with the same Git repository URL, branch, and path that the original cluster used. This bootstrap process installs Flux controllers, creates the GitRepository source, and applies the root Kustomization that triggers the entire reconciliation tree. Resources are applied in dependency order based on the dependsOn chains defined in Kustomization resources. The entire application stack can be recreated from Git within minutes, assuming the underlying infrastructure like databases, message queues, and storage is available or can be provisioned through Crossplane or Terraform.

Secret management is the most critical aspect of Flux disaster recovery. If the cluster used SOPS for secret encryption, the age private key or KMS key ARN must be available in the new cluster before Flux can decrypt secrets. This means the decryption keys themselves must be backed up outside the cluster, typically in a secure vault service or an offline backup. For clusters using Sealed Secrets, the controller's private key must be backed up and restored before any SealedSecret resources are applied, otherwise the controller generates a new key pair and cannot decrypt existing sealed secrets. Teams should document and regularly test the key recovery process, as a lost decryption key means all encrypted secrets in Git are unrecoverable. Some organizations maintain a break-glass procedure where plaintext secrets are stored in an out-of-band vault specifically for disaster recovery.

Migration between clusters, whether for Kubernetes version upgrades, cloud provider changes, or architecture improvements, follows a similar pattern but with the added complexity of maintaining service continuity. The blue-green cluster migration pattern involves bootstrapping Flux on the new cluster while the old cluster continues serving traffic, verifying all resources reconcile successfully, migrating stateful data, updating DNS or load balancer configurations to point to the new cluster, and finally decommissioning the old cluster. During the migration window, Git repositories should be frozen to prevent changes that might apply to both clusters simultaneously. Flux's suspend feature is useful here: suspending reconciliation on the old cluster prevents it from reverting changes that are being migrated, while the new cluster actively reconciles.

Production disaster recovery planning must include regular testing, runbook documentation, and recovery time objective measurements. Teams should perform quarterly disaster recovery drills where they bootstrap a complete cluster from Git and measure how long it takes to reach full operational status. The recovery time depends on factors like the number of resources, external dependency availability, image pull times, and health check durations. Monitoring should track the last successful reconciliation time for all resources, as a cluster that has not reconciled recently may have accumulated drift that complicates recovery. Velero complements Git-based recovery by backing up persistent volume data, custom resource states, and cluster-level resources that may not be fully represented in Git. The combination of Git as the declarative state backup and Velero as the runtime state backup provides comprehensive disaster recovery coverage.