How do Argo CD sync phases, waves, hooks, and health checks prevent or cause production deployment failures?
Quick Answer
Argo CD uses sync phases and waves to order resources, hooks to run lifecycle jobs, and health checks to decide whether reconciliation can progress. They prevent unsafe ordering when designed carefully, but failed hooks, unhealthy early waves, or selective sync behavior can leave applications stuck or partially deployed.
Detailed Answer
Think of a theater production. The stage crew must set the floor before actors enter, lights must be ready before the curtain opens, and cleanup happens after the show. If the lighting check fails, the performance should stop before customers see chaos. Argo CD sync phases and waves give Kubernetes deployments that same ordered choreography, but bad choreography can also stop the whole show.
Argo CD continuously compares desired state in Git with live state in clusters. During sync, hooks can run before, during, after, or on failure of the main operation. Sync waves add finer ordering with numeric annotations, where lower waves apply before higher waves. Health checks determine whether resources are ready enough for reconciliation to continue. This matters when namespaces, CRDs, RBAC, migrations, Deployments, and smoke tests depend on one another.
Internally, Argo CD orders work by phase, then wave, then kind, then name. PreSync hooks run first and can block everything if they fail. Normal resources default to wave zero unless annotated. PostSync hooks run after resources are synced and healthy. SyncFail hooks can perform cleanup. Argo CD also waits briefly between waves so other controllers have time to observe changes and update status.
In production, this is useful for database migrations, CRDs, admission policies, and smoke tests. Teams put CRDs and namespaces in early waves, workloads later, and validation jobs after workloads report healthy. They monitor application sync status, health status, operation duration, hook pods, repo-server errors, and controller queue depth. Progressive syncs and sync windows can reduce blast radius for fleets.
The gotcha is that hooks do not run during selective sync, and a resource that is unhealthy in an early wave can prevent later waves from applying. Another subtle failure is a migration hook that succeeds once but is not idempotent, then blocks rollback or retry. Senior engineers keep hooks small, observable, and retry-safe; avoid hiding critical dependencies in random waves; and document when manual intervention is required.