architectcicd

What is the recommended ArgoCD disaster recovery strategy, and how does git as single source of truth enable control-plane recovery without restoring etcd backups?

⚡

Quick Answer

ArgoCD disaster recovery leverages git as the authoritative source by exporting Application definitions, AppProjects, and cluster Secrets to a backup repository. Recovery involves reinstalling ArgoCD, restoring these declarative resources, and letting ArgoCD reconcile desired state from git. This avoids complex etcd restore procedures because the actual workload state is defined in git, not in ArgoCD's internal database.

Detailed Answer

Think of rebuilding a house after a fire. If you have the blueprints (git) and the building permits (Application definitions), you can reconstruct everything without needing the original furniture (ArgoCD's internal state). The key insight is that ArgoCD is itself a reconciliation engine, so its own configuration should follow the same GitOps principles it enforces on workloads.

ArgoCD's disaster recovery philosophy is grounded in the principle that git contains the truth about what should be deployed. The ArgoCD control plane stores Application resources, AppProject definitions, repository credentials, cluster connection Secrets, and RBAC policies. If the ArgoCD installation is lost, these resources are what need to be reconstructed. The actual workload manifests, Helm charts, and Kustomize overlays remain safely in git and do not need separate backup.

The recommended backup strategy has three layers. First, all ArgoCD Application and AppProject resources should themselves be managed via GitOps, typically using an app-of-apps pattern or ApplicationSets stored in a dedicated bootstrap repository. Second, Secrets containing cluster credentials, repository SSH keys, and OIDC configurations should be backed up to a secure store like Vault or AWS Secrets Manager, since these cannot be safely stored in git. Third, the argocd-export command produces a YAML dump of all ArgoCD resources from its internal state, which serves as a point-in-time snapshot for comparison and audit.

During recovery, the team installs a fresh ArgoCD instance, restores Secrets from the secure store, applies the bootstrap repository containing Application and AppProject definitions, and ArgoCD automatically begins reconciling all managed workloads from git. The first sync cycle may take significant time as ArgoCD clones all repositories, renders manifests, and diffs against live cluster state, but no workload changes are applied because the clusters already contain the correct resources. Monitoring during recovery should track sync status, repository clone success rates, and cluster connection health.

The non-obvious gotcha is that argocd-export does not capture everything. Custom resource health checks defined in argocd-cm ConfigMap, notification templates in argocd-notifications-cm, and RBAC policies in argocd-rbac-cm are ConfigMaps that must be version-controlled separately. Teams that rely solely on argocd-export discover during recovery that their custom health assessments, notification routing, and RBAC rules are missing, leaving them with a functional but misconfigured ArgoCD instance that approves resources it should block or fails to notify teams about sync failures.

# Export all ArgoCD resources for point-in-time backup argocd admin export -n argocd > argocd-backup-2026-06-20.yaml # Store the backup in a version-controlled disaster recovery repository cp argocd-backup-2026-06-20.yaml /opt/dr-repo/argocd/ cd /opt/dr-repo && git add argocd/ && git commit -m 'ArgoCD backup 2026-06-20' # Bootstrap Application that manages all other Applications (app-of-apps) apiVersion: argoproj.io/v1alpha1 # ArgoCD Application API kind: Application # Top-level bootstrap Application metadata: name: platform-bootstrap # The root Application that creates all others namespace: argocd # Must be in the ArgoCD namespace spec: project: default # Uses the default project for bootstrap source: repoURL: https://git.company.com/platform/argocd-bootstrap.git # DR-safe bootstrap repo targetRevision: main # Tracks the main branch path: applications/ # Directory containing all Application YAML files destination: server: https://kubernetes.default.svc # Applies to the local cluster (ArgoCD's own cluster) namespace: argocd # Creates Application resources in the argocd namespace syncPolicy: automated: # Bootstrap auto-syncs to restore all Applications prune: true # Removes Applications deleted from the bootstrap repo selfHeal: true # Re-creates Applications if manually deleted --- # Recovery procedure script after ArgoCD control-plane loss # Step 1: Install fresh ArgoCD kubectl create namespace argocd # Create the ArgoCD namespace on the recovery cluster kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.14.0/manifests/ha/install.yaml # Install HA ArgoCD # Step 2: Restore Secrets from Vault (cluster creds, repo keys, OIDC config) vault kv get -format=json secret/argocd/cluster-secrets | jq -r '.data.data.manifests' | kubectl apply -n argocd -f - # Step 3: Restore ConfigMaps with custom health checks, RBAC, and notifications kubectl apply -n argocd -f /opt/dr-repo/argocd/configmaps/argocd-cm.yaml # Custom resource health checks kubectl apply -n argocd -f /opt/dr-repo/argocd/configmaps/argocd-rbac-cm.yaml # RBAC policies kubectl apply -n argocd -f /opt/dr-repo/argocd/configmaps/argocd-notifications-cm.yaml # Notification templates # Step 4: Apply the bootstrap Application to trigger full reconciliation kubectl apply -n argocd -f /opt/dr-repo/argocd/applications/platform-bootstrap.yaml # Step 5: Monitor recovery progress across all Applications argocd app list --server https://argocd.platform.company.com -o wide | grep -v Synced

What is the recommended ArgoCD disaster recovery strategy, and how does git as single source of truth enable control-plane recovery without restoring etcd backups?

More Argo CD questions