Argo CD

Think of a theater production. The stage crew must set the floor before actors enter, lights must be ready before the curtain opens, and cleanup happens after the show. If the lighting check fails, the performance should stop before customers see chaos. Argo CD sync phases and waves give Kubernetes deployments that same ordered choreography, but bad choreography can also stop the whole show.

Argo CD continuously compares desired state in Git with live state in clusters. During sync, hooks can run before, during, after, or on failure of the main operation. Sync waves add finer ordering with numeric annotations, where lower waves apply before higher waves. Health checks determine whether resources are ready enough for reconciliation to continue. This matters when namespaces, CRDs, RBAC, migrations, Deployments, and smoke tests depend on one another.

Internally, Argo CD orders work by phase, then wave, then kind, then name. PreSync hooks run first and can block everything if they fail. Normal resources default to wave zero unless annotated. PostSync hooks run after resources are synced and healthy. SyncFail hooks can perform cleanup. Argo CD also waits briefly between waves so other controllers have time to observe changes and update status.

In production, this is useful for database migrations, CRDs, admission policies, and smoke tests. Teams put CRDs and namespaces in early waves, workloads later, and validation jobs after workloads report healthy. They monitor application sync status, health status, operation duration, hook pods, repo-server errors, and controller queue depth. Progressive syncs and sync windows can reduce blast radius for fleets.

The gotcha is that hooks do not run during selective sync, and a resource that is unhealthy in an early wave can prevent later waves from applying. Another subtle failure is a migration hook that succeeds once but is not idempotent, then blocks rollback or retry. Senior engineers keep hooks small, observable, and retry-safe; avoid hiding critical dependencies in random waves; and document when manual intervention is required.

Think of ArgoCD like a thermostat in your house. You set the desired temperature (your Git repository declares the desired cluster state), and the thermostat continuously monitors the current temperature (cluster state) and turns on heating or cooling (applies manifests) whenever there is a deviation. Just as the thermostat does not care how the temperature drifted — whether someone opened a window or the sun came out — ArgoCD does not care whether drift was caused by a kubectl edit, a rogue operator, or a failed rollout. It simply reconciles back to the declared truth in Git.

At its core, ArgoCD implements the GitOps pattern where Git serves as the single source of truth for declarative infrastructure and application definitions. When you register an Application resource with ArgoCD, you tell it which Git repository, branch, and path contain your Kubernetes manifests. ArgoCD then compares the rendered manifests from Git against the live objects in the target Kubernetes cluster. If they match, the application is considered Synced. If they differ, ArgoCD marks it as OutOfSync and can either alert you or automatically apply the changes depending on your sync policy configuration.

Internally, ArgoCD consists of three main components that power the reconciliation loop. The Repository Server clones Git repositories, caches them, and renders manifests using the appropriate tool such as Helm, Kustomize, or plain YAML. The Application Controller is the brain that runs the reconciliation loop on a configurable interval, defaulting to every three minutes. It fetches the desired state from the Repository Server, fetches the live state from the Kubernetes API server, performs a three-way diff using the last-applied-configuration annotation, and determines if the application is synced or drifted. The API Server exposes the gRPC and REST API consumed by the web UI and CLI, handles authentication via OIDC or DEX, and enforces RBAC policies for multi-tenant environments.

In production, you typically deploy ArgoCD in a dedicated namespace like argocd on a management cluster. Teams configure webhook notifications from GitHub or GitLab so that ArgoCD reacts to pushes within seconds rather than waiting for the next polling interval. You also integrate ArgoCD with notification controllers to send alerts to Slack or PagerDuty when an application goes OutOfSync or a sync operation fails. Resource tracking is configured using either the label-based or annotation-based method, and health checks are customized for CRDs that ArgoCD does not natively understand, ensuring the reconciliation loop accurately reflects the true health of your workloads.

A critical gotcha engineers overlook is the difference between the refresh interval and the sync interval. Refreshing means ArgoCD re-reads Git and re-computes the diff, while syncing means it actually applies changes to the cluster. Even with auto-sync enabled, ArgoCD will not sync if the previous sync failed — it enters a degraded state requiring manual intervention or a retry policy. Another pitfall is ignoring resource exclusions; by default ArgoCD tracks every resource it manages, and if an operator or admission controller mutates fields like timestamps or status subresources, ArgoCD may report perpetual OutOfSync status. You must configure ignoreDifferences in the Application spec to exclude those volatile fields from the diff computation.

Imagine an ArgoCD Application as a contract between a landlord and a tenant. The contract specifies exactly what the apartment should look like (Git source), which building and unit it applies to (destination cluster and namespace), and the rules for maintenance (sync policy). The sync policy is like the maintenance clause — it determines whether the landlord can enter and fix things automatically when the tenant makes unauthorized modifications, or whether the landlord must first get approval before making any changes. Just as different tenants may have different maintenance agreements, different applications can have different sync policies depending on their criticality.

The Application custom resource is the fundamental building block of ArgoCD. It encapsulates four key pieces of information: the source (a Git repository URL, revision, and path), the destination (a cluster API server endpoint and namespace), the project (which governs RBAC and allowed resources), and the sync policy (which defines automation behavior). When ArgoCD processes an Application, it renders the manifests from the source using the detected or configured tool — Helm, Kustomize, or directory of plain YAML — and compares the rendered output against the live state in the destination cluster. The comparison result is the sync status, which can be Synced, OutOfSync, or Unknown.

Sync policies control the automation level through three primary mechanisms. First, automated sync enables ArgoCD to apply changes without human intervention whenever it detects that Git has diverged from the cluster state. Second, the prune option within automated sync controls whether ArgoCD deletes Kubernetes resources that were removed from Git; without prune enabled, ArgoCD will only create and update resources but never delete them, leading to orphaned objects. Third, selfHeal instructs ArgoCD to revert any manual changes made directly to the cluster, effectively enforcing Git as the immutable source of truth. Additionally, syncOptions provide granular control such as CreateNamespace, ApplyOutOfSyncOnly, ServerSideApply, and Replace, each addressing specific deployment scenarios.

In production environments, teams typically use automated sync with prune and selfHeal for non-critical services like internal tools and staging environments, while keeping manual sync for production-critical services like the payments-api or order-service to maintain a human approval gate. Sync windows add another layer of control by restricting when automatic syncs can occur — for example, blocking deployments during peak traffic hours or maintenance windows. ArgoCD also supports sync retries with configurable backoff strategies to handle transient failures like API server throttling or temporary network issues, preventing a single failed sync from requiring manual intervention during off-hours.

A common gotcha is enabling automated sync with prune without understanding its implications. If someone accidentally pushes a commit that removes a critical manifest file — say the production database StatefulSet — ArgoCD will dutifully delete that StatefulSet from the cluster within seconds. This is why many teams use the argocd.argoproj.io/sync-options annotation with Prune=false on critical resources as a safety net, preventing accidental deletion even when the global prune policy is enabled. Another frequently overlooked detail is that automated sync will not trigger if the Application is in a degraded or error health state from a previous failed sync. Engineers must configure proper alerting on sync failures and understand that selfHeal only applies to resources ArgoCD manages — it cannot detect or fix drift in resources outside its tracking scope.

Think of ApplicationSets like a mail merge in a word processor. You write one template letter (the Application template) and provide a spreadsheet of recipients with their unique details (the generator producing parameters). The mail merge engine produces one personalized letter for each row in the spreadsheet. Similarly, the ApplicationSet controller takes one Application template and one or more generators, then produces a unique ArgoCD Application for each set of parameters the generator yields — one per cluster, one per environment, or one per microservice, depending on your strategy.

ApplicationSets are a Kubernetes custom resource managed by the ApplicationSet controller, which runs alongside the core ArgoCD components. The resource specification consists of two main sections: generators and a template. Generators produce a list of parameter sets — key-value pairs that get substituted into the template. ArgoCD ships with several built-in generators including the list generator for explicit enumeration, the cluster generator that discovers clusters registered in ArgoCD, the Git directory generator that scans a repository for subdirectories, the Git file generator that reads JSON or YAML parameter files from a repository, the pull request generator for ephemeral preview environments, and the matrix and merge generators for combining multiple generators together. Each generator yields rows of parameters, and for each row, the controller renders the template with those parameters substituted in, creating a distinct Application resource.

Internally, the ApplicationSet controller watches for ApplicationSet resources and runs a reconciliation loop separate from the main ArgoCD Application Controller. When it detects a new or modified ApplicationSet, it evaluates the generators to produce the parameter matrix, renders the template for each parameter set, and then creates, updates, or deletes the corresponding Application resources. The controller uses owner references to track which Applications belong to which ApplicationSet, enabling garbage collection when parameters are removed from a generator. The matrix generator deserves special attention because it computes the Cartesian product of two child generators — for example, combining a cluster generator with a Git directory generator to deploy every microservice to every cluster, producing N times M Applications from just two generators.

In production, ApplicationSets shine for platform teams managing dozens or hundreds of clusters. A common pattern is to use the cluster generator combined with labels to target specific cluster groups — for example, deploying the payments-api to all clusters labeled region=us-east and tier=production. Another powerful pattern uses the Git file generator where each cluster has a configuration file in a dedicated repository, and the file contains cluster-specific parameters like replica counts, resource limits, and feature flags. When a new cluster is provisioned and registered with ArgoCD, the cluster generator automatically picks it up and creates Applications for it without any manual intervention, achieving true zero-touch multi-cluster deployment at scale.

A significant gotcha with ApplicationSets is the deletion policy. By default, when an ApplicationSet is deleted, all generated Applications and their managed resources are also deleted — this cascading behavior can be catastrophic if someone accidentally deletes the ApplicationSet in Git. The preserveResourcesOnDeletion policy prevents this by keeping the Applications alive even after the ApplicationSet is removed. Another common pitfall is generator parameter conflicts when using the matrix generator; if both child generators produce a parameter with the same key, the second generator's value silently overwrites the first, leading to subtle deployment errors. Teams should also be cautious with the pull request generator in production clusters — without proper RBAC scoping the ApplicationSet to a sandboxed project, a malicious pull request could deploy arbitrary workloads to production namespaces.

Think of sync waves and hooks like staging a theater production. The stage crew sets up the backdrop first (wave -1 for config maps and secrets), then the orchestra pit is prepared (wave 0 for database migrations via a PreSync hook), the actors take their positions (wave 1 for the main deployments), and finally the curtain call sequence runs (wave 2 PostSync hooks for smoke tests and notifications). Each phase must complete successfully before the next begins, and if anything fails — say a prop malfunctions — there is a SyncFail hook that triggers the stage manager's emergency protocol. This ordered execution ensures that dependencies are satisfied before dependents are deployed.

Sync waves work through the argocd.argoproj.io/sync-wave annotation, which accepts an integer value. During a sync operation, ArgoCD groups all resources by their wave number and processes them in ascending order. Resources without a wave annotation default to wave 0. Within each wave, ArgoCD applies all resources simultaneously and waits for every resource in that wave to reach a healthy state before progressing to the next wave. If any resource in a wave fails to become healthy within the configured timeout, the sync operation is marked as failed and subsequent waves are not executed. Hooks, on the other hand, are resources annotated with argocd.argoproj.io/hook and are executed at specific phases: PreSync runs before any wave begins, Sync runs alongside wave 0, PostSync runs after all waves complete successfully, and SyncFail runs only when a sync operation fails.

Internally, the sync engine within the Application Controller orchestrates this process. When a sync is triggered, the controller first identifies all hook and wave annotations across the managed resources. It constructs an execution plan that orders PreSync hooks first, followed by each wave in numerical order with their contained resources, and finally PostSync hooks. For each phase, the controller applies the resources to the cluster and then enters a polling loop, checking the health status of each resource against ArgoCD's built-in health assessment logic. For Deployments, health means all replicas are available. For Jobs used as hooks, health means the Job completed successfully. The controller uses the argocd.argoproj.io/hook-delete-policy annotation to determine whether hook resources should be cleaned up before each sync, after successful completion, or after failure, preventing accumulation of completed Job pods.

In production, a typical deployment of the order-service might use wave -2 for namespace and RBAC resources, wave -1 for ConfigMaps and Secrets, a PreSync hook Job for database schema migrations, wave 0 for the core Deployment and Service, wave 1 for HorizontalPodAutoscaler and PodDisruptionBudget, and a PostSync hook for integration smoke tests that verify the API is responding correctly. This ensures the database schema is ready before the application starts, and the application is running before autoscaling policies are applied. Teams often combine waves with health checks on custom resources to gate progression — for example, waiting for a CertificateRequest to be issued before deploying the Ingress that references it.

The most dangerous gotcha with sync waves is circular health dependencies. If a resource in wave 0 depends on a resource in wave 1 to become healthy — for example, a Service in wave 0 that needs endpoints from a Deployment in wave 1 — the sync will deadlock because wave 0 will never become healthy and wave 1 will never execute. Engineers must carefully map dependency graphs before assigning wave numbers. Another common mistake is forgetting the hook-delete-policy annotation on PreSync Jobs; without it, the Job from the previous sync still exists and ArgoCD will not re-run it because Kubernetes prevents creating a Job with the same name. Setting HookSucceeded as the delete policy ensures the previous Job is cleaned up before the next sync creates a new one.

Imagine deploying a new version of software like opening a new lane on a highway. With a standard Kubernetes Deployment, you instantly redirect all traffic to the new lane — if there is a pothole, every car hits it. A canary release is like opening the new lane to only five percent of traffic first, monitoring for incidents, then gradually increasing to twenty-five, fifty, and finally one hundred percent. A blue-green deployment is like building an entirely parallel highway, verifying it with test traffic, and then switching the exit ramp so all cars seamlessly move to the new road. Argo Rollouts provides the traffic controller that orchestrates these patterns with automated analysis and instant rollback capabilities.

Argo Rollouts extends Kubernetes with a Rollout custom resource that replaces the standard Deployment. The Rollout spec includes a strategy section where you define either a canary or blueGreen configuration. For canary deployments, you specify steps that control the rollout progression — setWeight to adjust traffic percentage, pause to wait for a duration or manual approval, and analysis to run automated metric checks. The Rollout controller manages two ReplicaSets: the stable version serving production traffic and the canary version receiving the configured percentage. Traffic splitting is achieved through integration with service mesh or ingress controllers including Istio, Linkerd, Nginx, ALB, SMI, Traefik, and Ambassador, each using their native traffic management primitives.

Internally, the Argo Rollouts controller watches Rollout resources and manages the rollout state machine. When a new image or spec change is detected, the controller creates a canary ReplicaSet and begins executing the defined steps sequentially. At each analysis step, the controller creates an AnalysisRun resource that queries configured metrics providers — Prometheus, Datadog, NewRelic, CloudWatch, or custom webhook providers — and evaluates the results against success criteria. If an AnalysisRun determines the canary is unhealthy based on error rate thresholds or latency percentiles, the controller automatically aborts the rollout and scales the canary ReplicaSet to zero, restoring all traffic to the stable version. For blue-green deployments, the controller maintains an active and preview Service, each pointing to different ReplicaSets. The preview Service receives the new version for testing, and upon promotion, the active Service selector is switched to point to the new ReplicaSet.

In production, teams typically integrate Argo Rollouts with ArgoCD by simply replacing the Deployment kind with Rollout in their Git manifests. ArgoCD recognizes the Rollout CRD and provides native health status tracking in its dashboard. A common pattern for the checkout-worker service is a canary strategy with five steps: set weight to ten percent, run a five-minute Prometheus analysis checking error rates and p99 latency, set weight to thirty percent, run another analysis for ten minutes, then set weight to one hundred percent. The AnalysisTemplate queries like rate(http_requests_total{status=~"5.*",service="checkout-worker"}[5m]) / rate(http_requests_total{service="checkout-worker"}[5m]) ensure the canary's error rate stays below the configured threshold before each promotion step.

The most critical gotcha is that Argo Rollouts traffic splitting only works if you have a supported traffic router configured. Without Istio VirtualServices, Nginx annotations, or another supported ingress controller, the weight-based traffic splitting has no effect and the canary just runs alongside the stable version without controlled traffic distribution — it falls back to basic replica-ratio-based splitting which is imprecise. Another common mistake is setting analysis intervals too short; a two-minute analysis window may not capture enough data points for statistically significant error rate calculations, leading to false positives that promote a bad canary. Engineers should also understand that argo rollouts differ from ArgoCD — they are separate projects with separate controllers, and you need both installed to get GitOps-driven progressive delivery.

Think of ArgoCD's manifest rendering like a universal translator at the United Nations. Delegates speak different languages (Helm charts with values, Kustomize with overlays, plain YAML), but the translator converts everything into one common language (rendered Kubernetes manifests) before the assembly (the cluster) acts on them. The translator does not care which language was spoken — it detects the language automatically and produces the same standard output. This means teams across an organization can each use their preferred templating tool while ArgoCD provides a consistent GitOps workflow for all of them.

ArgoCD detects the manifest source type automatically based on the repository contents. If it finds a Chart.yaml file, it treats the source as a Helm chart. If it finds a kustomization.yaml or kustomization.yml file, it uses Kustomize. If neither is present, it treats the directory as plain YAML manifests and applies all YAML and JSON files found in the specified path. You can also explicitly set the source type in the Application spec to override auto-detection. For Helm, you can specify values files, inline value overrides, release name, and chart version. For Kustomize, you can override images, add name prefixes and suffixes, and set common labels and annotations. For plain manifests, ArgoCD supports recursive directory traversal and file include or exclude patterns using the directory.recurse and directory.exclude options.

Internally, the Repository Server is the component responsible for all manifest rendering. When the Application Controller requests the desired state for an application, the Repository Server clones the Git repository (or fetches from a Helm chart repository), caches the result, and invokes the appropriate rendering tool. For Helm, it runs the equivalent of helm template with the configured values, producing raw Kubernetes manifests without installing a release into the cluster — this is a critical distinction because ArgoCD does not use Tiller or the Helm release secret mechanism. For Kustomize, it runs kustomize build on the specified path. The rendered output is then returned to the Application Controller for diffing against the live state. The Repository Server maintains a cache of cloned repositories and rendered manifests with configurable TTL, reducing Git API calls and rendering overhead for frequently polled applications.

In production, organizations commonly use a hybrid approach. Infrastructure components like cert-manager, ingress-nginx, and monitoring stacks are deployed as Helm charts sourced directly from public or private Helm repositories using the helm source type with a repoURL pointing to the chart repository. Application teams use Kustomize with a base-and-overlay pattern where the base directory contains the core Deployment, Service, and ConfigMap definitions, and overlays for staging and production add environment-specific patches like replica counts, resource limits, and ingress hostnames. Some teams use Helm charts as Kustomize bases by running helm template in a CI pipeline and committing the rendered output, giving them the parameterization power of Helm with the patching flexibility of Kustomize. This multi-tool strategy allows each team to use the approach that best fits their workflow while ArgoCD provides a unified deployment and observability layer.

A major gotcha with Helm integration is that ArgoCD runs helm template, not helm install, which means Helm hooks and lookup functions behave differently. The lookup function always returns empty results because there is no live cluster context during template rendering, causing templates that depend on runtime lookups to produce incorrect manifests. Helm release lifecycle hooks like pre-install and post-install are not natively supported — you must use ArgoCD sync hooks instead. Another common pitfall with Kustomize is version mismatches; ArgoCD bundles a specific version of Kustomize, and if your kustomization.yaml uses features from a newer version, rendering will fail silently or produce unexpected output. Teams should pin the Kustomize version in ArgoCD's configmap or use a custom plugin to ensure compatibility. For plain manifests, engineers frequently forget that ArgoCD applies all YAML files in the directory and accidentally include test fixtures or documentation YAML files in the deployment path.

Think of a mail merge in a word processor. You have one template letter and a spreadsheet of recipients, and the system generates a personalized letter for each row. ApplicationSet generators work the same way: you write one Application template, and the generator produces a concrete ArgoCD Application for every entry it discovers, whether that entry is a cluster name, a git directory, or a computed combination of both.

ApplicationSet exists because manually creating and maintaining hundreds of Application resources across dozens of clusters is operationally unsustainable. The list generator is the simplest: you hardcode a JSON array of clusters and parameters, and each entry produces one Application. The git generator scans a repository for directories or files matching a pattern, so adding a new microservice directory automatically creates its Application. The matrix generator takes two child generators and computes every combination, like deploying every service to every cluster. The merge generator combines outputs from multiple generators by matching on a key field, allowing base parameters to be enriched with cluster-specific overrides.

Internally, the ApplicationSet controller runs inside the argocd-applicationset-controller pod and periodically evaluates each generator. For git generators, it clones or fetches the target repository at a configurable interval, walks the directory tree or parses files, and emits parameter sets. The matrix generator performs a Cartesian product of its child generator outputs. The merge generator performs a left-join on a specified key. Each resulting parameter set is rendered into the Application template, and the controller creates, updates, or deletes Application resources to match the computed set.

At production scale, teams running 200-plus clusters with 50-plus microservices can easily generate 10,000 Applications from a single ApplicationSet using matrix generators. This requires monitoring the ApplicationSet controller's memory consumption, git clone frequency (to avoid hammering the git server), reconciliation loop duration, and Application sync status across the fleet. The progressive sync feature with maxUpdate controls how many Applications sync simultaneously during a rollout, preventing a blast-radius scenario where every cluster updates at once.

The non-obvious gotcha is the delete behavior. By default, when a generator stops producing a parameter set, the corresponding Application is deleted, which triggers a cascade delete of the deployed resources. If a git directory is accidentally renamed or a list entry removed, the ApplicationSet controller can delete production workloads across multiple clusters in a single reconciliation loop. Architects must configure the preserveResourcesOnDeletion policy and use SCM-protected branches to prevent accidental mass deletion from a bad git commit.

Think of an air traffic control center managing flights across multiple airports. Each airport is a separate cluster, the control center is the ArgoCD installation, and the radar screens are the cached views of cluster state. If one radar screen handles all airports, it becomes overwhelmed. Sharding distributes airports across multiple controllers, and a shared communication backbone (Redis) keeps everyone coordinated.

ArgoCD was originally designed to manage applications on the cluster where it runs, but production enterprises routinely manage 50 to 300 clusters from a single ArgoCD instance. Each external cluster is registered as a Kubernetes Secret in the argocd namespace with the label argocd.argoproj.io/secret-type=cluster. This Secret contains the cluster API server URL, a bearer token or client certificate, TLS CA data, and optional configuration like connection limits and namespace restrictions. The application-controller watches these secrets and treats each registered cluster as a sync target.

Internally, three components must scale independently. The application-controller reconciles Application resources by comparing desired state (from git) with live state (from clusters). At scale, a single controller instance becomes CPU-bound when diffing thousands of resources across dozens of clusters. ArgoCD supports sharding the controller so that each shard handles a subset of clusters, assigned by round-robin or explicit annotation. The repo-server clones git repositories, renders Helm charts, runs Kustomize, and returns manifests. Multiple repo-server replicas share the load, and the parallelism limit per server controls concurrent rendering operations. Redis caches rendered manifests, application state, and RBAC data. Without proper Redis availability, every cache miss triggers a full git clone and render cycle.

At production scale with 100-plus clusters, architects must monitor application-controller queue depth, reconciliation duration per cluster, repo-server clone and render times, Redis memory usage and eviction rates, and git server request rates. The controller's ARGOCD_CONTROLLER_REPLICAS environment variable sets shard count, and each replica processes only its assigned clusters. Redis should run in sentinel mode or as a Redis cluster for high availability, because a Redis failure causes all repo-server cache lookups to fall back to git, which can overwhelm the git server and spike reconciliation times by an order of magnitude.

The non-obvious gotcha is cluster Secret rotation. When a service account token or client certificate in a cluster Secret expires, ArgoCD loses access to that cluster silently unless health checks are monitored. Applications appear OutOfSync or Unknown, but the root cause is a credential failure, not a drift issue. Architects should automate Secret rotation with external-secrets-operator or a custom controller and set up alerts on cluster connection errors in the application-controller logs before tokens approach expiry.

Think of a theater production with scene changes. The stagehands must set up the backdrop (wave 0) before the props (wave 1) before the actors enter (wave 2). If the backdrop collapses during setup, the director stops everything rather than sending actors onto a broken stage. ArgoCD sync waves work the same way: they enforce ordering so that infrastructure dependencies are ready before the services that depend on them.

Sync waves and hooks solve a fundamental problem in GitOps: not all resources should be applied simultaneously. A database migration must complete before the application deployment that depends on the new schema. A ConfigMap or Secret must exist before the Deployment that mounts it. A namespace must be created before resources are placed in it. Without ordering, ArgoCD would apply everything at once, leading to race conditions and transient failures.

Internally, ArgoCD processes sync waves by sorting all resources by their argocd.argoproj.io/sync-wave annotation (defaulting to 0). Within each wave, it applies resources and waits for them to reach a healthy state according to their resource health check. Hooks are special resources annotated with argocd.argoproj.io/hook, which can be PreSync (runs before any wave), Sync (runs during the sync phase), PostSync (runs after all waves complete), or SyncFail (runs when sync fails). Hook resources are typically Jobs or Pods. The hook-delete-policy annotation controls whether hooks are deleted before creation, after successful completion, or after failure.

At production scale, teams use wave 0 for namespaces and CRDs, wave 1 for ConfigMaps and Secrets, wave 2 for databases and stateful services, wave 3 for application Deployments, and wave 4 for ingress routes and monitoring. PreSync hooks commonly run database migrations, schema validators, or pre-deployment smoke tests. PostSync hooks run integration tests, cache warmers, or notification jobs. Monitoring should track sync duration per wave, hook execution time, hook pod resource consumption, and retry counts.

The non-obvious gotcha is partial sync state after a hook failure. If a PreSync migration Job fails at wave 1, ArgoCD stops the sync operation entirely. Wave 2 and beyond are never attempted, but wave 0 resources are already applied. The Application is now in a partially synced state that does not match either the previous git commit or the current one. Retrying the sync re-runs all hooks from the beginning, which means PreSync hooks must be idempotent. If a database migration is not idempotent and partially applied, retrying the sync can corrupt data. Architects must design every hook to be safely re-runnable and implement SyncFail hooks for cleanup or rollback notifications.

Think of rebuilding a house after a fire. If you have the blueprints (git) and the building permits (Application definitions), you can reconstruct everything without needing the original furniture (ArgoCD's internal state). The key insight is that ArgoCD is itself a reconciliation engine, so its own configuration should follow the same GitOps principles it enforces on workloads.

ArgoCD's disaster recovery philosophy is grounded in the principle that git contains the truth about what should be deployed. The ArgoCD control plane stores Application resources, AppProject definitions, repository credentials, cluster connection Secrets, and RBAC policies. If the ArgoCD installation is lost, these resources are what need to be reconstructed. The actual workload manifests, Helm charts, and Kustomize overlays remain safely in git and do not need separate backup.

The recommended backup strategy has three layers. First, all ArgoCD Application and AppProject resources should themselves be managed via GitOps, typically using an app-of-apps pattern or ApplicationSets stored in a dedicated bootstrap repository. Second, Secrets containing cluster credentials, repository SSH keys, and OIDC configurations should be backed up to a secure store like Vault or AWS Secrets Manager, since these cannot be safely stored in git. Third, the argocd-export command produces a YAML dump of all ArgoCD resources from its internal state, which serves as a point-in-time snapshot for comparison and audit.

During recovery, the team installs a fresh ArgoCD instance, restores Secrets from the secure store, applies the bootstrap repository containing Application and AppProject definitions, and ArgoCD automatically begins reconciling all managed workloads from git. The first sync cycle may take significant time as ArgoCD clones all repositories, renders manifests, and diffs against live cluster state, but no workload changes are applied because the clusters already contain the correct resources. Monitoring during recovery should track sync status, repository clone success rates, and cluster connection health.

The non-obvious gotcha is that argocd-export does not capture everything. Custom resource health checks defined in argocd-cm ConfigMap, notification templates in argocd-notifications-cm, and RBAC policies in argocd-rbac-cm are ConfigMaps that must be version-controlled separately. Teams that rely solely on argocd-export discover during recovery that their custom health assessments, notification routing, and RBAC rules are missing, leaving them with a functional but misconfigured ArgoCD instance that approves resources it should block or fails to notify teams about sync failures.

Think of two different approaches to managing a fleet of retail stores. ArgoCD is like a central headquarters with a control room, video feeds from every store, and managers who issue directives from HQ. Flux is like giving each store manager an autonomous playbook and letting them follow it independently, reporting status back when asked. Both keep stores running correctly, but the management overhead, visibility, and failure patterns differ fundamentally.

ArgoCD and Flux are both CNCF graduated GitOps tools, but they make different architectural choices. ArgoCD centralizes control: a single ArgoCD installation can manage hundreds of clusters through registered cluster connections, and operators see everything in a unified web UI with role-based access control, SSO integration, and audit logging. Flux decentralizes control: each cluster runs its own set of Flux controllers (source-controller, kustomize-controller, helm-controller, notification-controller), and each cluster independently reconciles from git without needing a central management plane.

The reconciliation models differ in important ways. ArgoCD's application-controller actively compares live cluster state against rendered manifests and reports drift. Sync can be automatic or manual, and webhooks from git providers can trigger immediate reconciliation. Flux's source-controller polls git repositories or OCI registries at a configurable interval, downloads the latest artifacts, and the kustomize-controller or helm-controller applies them. Flux's model is strictly pull-based: the cluster reaches out to git, not the other way around. This means Flux clusters behind restrictive firewalls need only egress access to git, while ArgoCD needs both egress to git and ingress credentials to target clusters.

At production scale, the UI and RBAC differences become decisive. ArgoCD provides a full web dashboard showing application topology, sync status, resource health, logs, and diff views. Its RBAC system supports project-based access control with OIDC/SAML integration, allowing platform teams to grant developers read-only access to specific projects. Flux has no built-in UI; teams use Weave GitOps (now a commercial product), Capacitor, or custom dashboards. Flux's RBAC relies on Kubernetes-native RBAC applied to the service accounts used by Flux controllers, which is simpler but less granular for multi-tenant developer self-service.

The non-obvious gotcha is that neither tool is universally superior. ArgoCD's centralized model creates a single point of failure and a scaling challenge: the control plane must handle all clusters, and an ArgoCD outage blocks deployments everywhere. Flux's decentralized model means no single point of failure for deployments, but troubleshooting requires accessing each cluster individually, and ensuring consistent configuration across hundreds of Flux installations requires its own automation layer (often Terraform or Crossplane). Architects should choose ArgoCD when developer self-service UI, centralized audit logging, and granular RBAC are requirements, and Flux when cluster autonomy, firewall restrictions, or minimal central infrastructure are priorities.

Common Causes of False OutOfSync

1. Server-side defaults: Kubernetes API server adds default fields (e.g., strategy.rollingUpdate, resources: {}, terminationGracePeriodSeconds: 30) that don't exist in your Git manifests. ArgoCD sees these as differences.

2. Helm template non-determinism: Helm functions like randAlphaNum, now, or lookup produce different output each time, causing perpetual drift.

3. Mutating admission webhooks: Webhooks (Istio sidecar injector, Vault agent injector) modify resources after apply, adding fields not in Git.

4. Resource hooks: ArgoCD resource hooks (argocd.argoproj.io/hook) may leave resources in unexpected states.

5. CRD schema changes: CRD upgrade changes field defaults or removes deprecated fields.

6. Last-applied-configuration annotation: kubectl apply sets this annotation; ArgoCD uses server-side diff, so the annotation itself can cause drift.

Debugging Steps

1. argocd app diff my-app --local ./manifests — shows exact diff 2. Check ArgoCD UI diff view — highlights which fields differ 3. kubectl get <resource> -o yaml vs your Git manifest — compare manually 4. Check for mutating webhooks: kubectl get mutatingwebhookconfigurations

Fixes

1. Ignore known diffs in ArgoCD Application spec: ``yaml spec: ignoreDifferences: - group: apps kind: Deployment jsonPointers: - /spec/template/metadata/annotations/kubectl.kubernetes.io~1last-applied-configuration ``

2. Use server-side apply: argocd app set my-app --server-side-diff=true (ArgoCD 2.10+) 3. Pin Helm values explicitly: Don't rely on Helm chart defaults 4. Normalize manifests: Use argocd app set --self-heal to auto-sync

Argo CD's core reconciliation loop continuously compares the live manifests in the cluster against what's rendered from the target Git revision, and any difference is drift — this could come from someone running kubectl edit directly against a resource, another controller mutating a field Argo CD also manages, or simply Git having moved forward since the last sync.

By default (manual sync), Argo CD only reports drift as an 'OutOfSync' status in the UI/CLI and does nothing further until an operator explicitly runs argocd app sync. With automated sync policy enabled, Argo CD will automatically apply a sync whenever it detects the live state differs from Git — but note this only triggers when Argo CD's own reconciliation notices drift on its polling interval, not instantaneously. Adding selfHeal: true to the automated sync policy makes Argo CD actively revert manual changes back to the Git-defined state as soon as it detects them, treating any out-of-band kubectl edit as something to be corrected rather than tolerated — this is what makes Git genuinely the single source of truth rather than just a convenient deployment starting point.

A common gotcha: without selfHeal, a well-meaning but manual kubectl scale during an incident will silently persist until the next sync (which might reset it, surprising the responder), and with selfHeal it might get reverted almost immediately, which is exactly the desired GitOps behavior but can confuse an engineer who doesn't realize self-healing is enabled and expects their manual fix to stick.

An ordinary Argo CD Application points at a Git path containing Kubernetes manifests for a real workload (Deployments, Services, etc.). In App of Apps, the parent Application's Git path instead contains a set of Argo CD Application custom resource manifests themselves — so when Argo CD syncs the parent, what it 'applies' is the creation of several child Application objects in the argocd namespace, and each of THOSE then independently reconciles against its own target Git path/cluster/namespace, syncing the actual workload manifests.

This is commonly combined with ApplicationSets for even more dynamic generation (e.g. templating one child Application per entry in a list of clusters or a Git directory structure), but the basic App of Apps pattern alone already solves a real operational problem: onboarding a new microservice or a new cluster becomes 'add one more Application manifest to the parent's Git repo' rather than manually running argocd app create by hand for every new service, which doesn't scale past a handful of applications and isn't itself tracked in Git.

A nuance worth knowing: because each child Application syncs independently once created, ordering/dependencies between them (e.g. a shared CRD needing to exist before a dependent Application's resources can be applied) require explicit handling — commonly via sync waves (argocd.argoproj.io/sync-wave annotations) rather than assuming the App of Apps parent enforces any particular child ordering by default.

argocd app get <app> and argocd app diff <app> are the first stops — diff shows a live-vs-target manifest comparison so you can see exactly which fields Argo CD considers out of sync, rather than guessing from the UI's summary badge alone.

Common root causes, roughly in likelihood order for a persistent/confusing OutOfSync: (1) a mutating admission webhook or another controller (e.g. a HorizontalPodAutoscaler adjusting replica count, or a service mesh sidecar injector adding fields) modifies the resource after Argo CD applies it, and Argo CD's next reconciliation sees that as drift from Git — the fix is usually adding an ignoreDifferences rule for that specific field path in the Application spec, rather than fighting the other controller; (2) templating differences — what helm template or kustomize build renders locally doesn't exactly match what Argo CD renders server-side (different chart/kustomize version, different values precedence), which argocd app diff combined with running the same render command locally can confirm; (3) RBAC — the Argo CD Application controller's service account lacking permission on a specific resource type/namespace will cause sync failures on just those resources, visible in the Application's conditions/events rather than the diff itself; (4) resources explicitly excluded via syncOptions or the Application's ignoreDifferences block that were configured for a different reason and are now masking something real.

The systematic approach: reproduce the render locally (helm template/kustomize build) to rule out a templating mismatch first, since that's usually fastest to confirm or rule out, then check argocd app diff for the specific field-level differences, then check RBAC/events only if the diff shows a resource that never even got attempted.

Argo CD's default behavior (prune: false, or not set) is conservative: it applies whatever's in Git, but if a manifest is removed from the tracked Git path, Argo CD doesn't infer that means 'delete this from the cluster too' — the corresponding resource is simply no longer managed/tracked but stays running, becoming what's often called an orphaned resource. This avoids accidental deletions but means cluster cleanliness degrades over time unless someone manually cleans up orphaned resources.

prune: true in the sync policy makes Argo CD's reconciliation symmetric: anything present in the live cluster that Argo CD manages but that's NOT present in the current target Git revision gets deleted on the next sync. This keeps the cluster's actual state exactly matching Git, which is the more 'complete' GitOps promise, but it also means a mistake in Git — someone accidentally deleting a manifest file, a bad rebase/merge that drops a resource, a misconfigured Kustomize overlay that stops including something — now automatically deletes real running resources on the very next automated sync, potentially including things like PersistentVolumeClaims if not explicitly protected.

Because of this, teams often combine prune with additional safety nets: PrunePropagationPolicy to control cascade behavior, explicit preserveResourcesOnDeletion for stateful resources, requiring manual sync (not fully automated) for particularly sensitive Applications even if prune is enabled, and strong Git branch protection/review requirements specifically because a bad merge is now a production deletion risk, not just a drift risk.

GitOps's core premise — Git as the single source of truth, fully readable and diffable — is fundamentally in tension with secrets, which must never be plaintext-readable in Git. Argo CD doesn't natively solve this; it just applies whatever manifests are in Git, so teams need a complementary mechanism.

Sealed Secrets (Bitnami/community project) lets you encrypt a Secret client-side using a public key, producing a SealedSecret custom resource that's safe to commit to Git in plaintext-looking-but-actually-encrypted form; a cluster-side controller holding the matching private key decrypts it into a real Secret only inside the cluster. Argo CD just syncs the SealedSecret manifest like anything else — the encryption/decryption is transparent to it.

External Secrets Operator (and similar tools) take a different approach: Git only contains a reference/policy manifest (which secret path in Vault/AWS Secrets Manager/etc. to pull, and which Kubernetes Secret name to populate), and a controller running in-cluster fetches the actual secret value from the external secret store and creates/updates the real Kubernetes Secret — Argo CD tracks and syncs the reference manifest, never the actual secret value, which never touches Git at all.

Both patterns keep Argo CD's job unchanged (sync manifests from Git) while solving the secret-material problem underneath it; the choice between them usually comes down to whether the org already has a central secrets manager (favoring External Secrets Operator) or wants a simpler, self-contained encrypt-and-commit workflow (favoring Sealed Secrets).

Instead of pushing changes with kubectl from CI, you commit desired state to Git and Argo CD pulls and applies it, detecting and optionally correcting drift. Benefits: every change is reviewed and auditable, rollback is git revert, and the cluster self-documents. It's declarative and continuous rather than imperative and one-shot.

It specifies repoURL, targetRevision, and path for the source, the destination cluster and namespace, and a syncPolicy. Argo CD renders the manifests (plain YAML, Kustomize, or Helm) and keeps the destination matching. ApplicationSets can generate many Applications automatically.

The two are independent: an app can be Synced but Degraded (matches Git but pods crashloop) or OutOfSync but Healthy (someone changed the cluster manually). Argo CD surfaces both per resource in a live tree, which is key to diagnosing incidents.

Automated sync applies Git changes without a manual click. prune: true removes orphans when you delete a manifest — without it, deleted resources linger. selfHeal: true undoes drift like a manual kubectl scale within seconds, enforcing that Git always wins. Together they make the cluster fully declarative.

An ApplicationSet with a git directory generator creates an Application for every services/* folder automatically — add a folder, get an app. Cluster and list generators fan out across environments. App-of-Apps is a simpler variant where one root Application deploys child Applications.

Because Git is the source of truth, reverting the offending commit flows through review and Argo CD applies it. argocd app history and app rollback <rev> revert to a previously synced revision instantly, but bypass Git, so treat it as emergency-only to avoid drift between Git and cluster.

The default project allows everything, which is unsafe for multi-team clusters. Custom projects scope source repos, destination clusters/namespaces, and permitted resource kinds, and integrate with RBAC/SSO. They're the primary multi-tenancy guardrail in Argo CD.

argocd.argoproj.io/sync-wave orders resources (lower waves first), so a database migration or CRD can apply before the workload that needs it. Sync hooks (PreSync/PostSync) run Jobs at defined phases — e.g. a PreSync migration or a PostSync smoke test — enabling safe, ordered rollouts.