Your ArgoCD application is stuck in 'OutOfSync' state but the manifests in Git match what's deployed. What are the possible causes and how do you debug?
Quick Answer
Common causes: resource hooks, server-side field defaults, helm template differences, last-applied-annotation drift, or CRD schema changes. Debug with `argocd app diff` and check for ignored differences.
Detailed Answer
Common Causes of False OutOfSync
1. Server-side defaults: Kubernetes API server adds default fields (e.g., strategy.rollingUpdate, resources: {}, terminationGracePeriodSeconds: 30) that don't exist in your Git manifests. ArgoCD sees these as differences.
2. Helm template non-determinism: Helm functions like randAlphaNum, now, or lookup produce different output each time, causing perpetual drift.
3. Mutating admission webhooks: Webhooks (Istio sidecar injector, Vault agent injector) modify resources after apply, adding fields not in Git.
4. Resource hooks: ArgoCD resource hooks (argocd.argoproj.io/hook) may leave resources in unexpected states.
5. CRD schema changes: CRD upgrade changes field defaults or removes deprecated fields.
6. Last-applied-configuration annotation: kubectl apply sets this annotation; ArgoCD uses server-side diff, so the annotation itself can cause drift.
Debugging Steps
1. argocd app diff my-app --local ./manifests — shows exact diff 2. Check ArgoCD UI diff view — highlights which fields differ 3. kubectl get <resource> -o yaml vs your Git manifest — compare manually 4. Check for mutating webhooks: kubectl get mutatingwebhookconfigurations
Fixes
1. Ignore known diffs in ArgoCD Application spec: ``yaml spec: ignoreDifferences: - group: apps kind: Deployment jsonPointers: - /spec/template/metadata/annotations/kubectl.kubernetes.io~1last-applied-configuration ``
2. Use server-side apply: argocd app set my-app --server-side-diff=true (ArgoCD 2.10+) 3. Pin Helm values explicitly: Don't rely on Helm chart defaults 4. Normalize manifests: Use argocd app set --self-heal to auto-sync
Code Example
# Debug: see exact diff
argocd app diff checkout-service
# Ignore known diffs in Application spec
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas # Managed by HPA
- group: ""
kind: Service
jqPathExpressions:
- .spec.clusterIP # Assigned by k8s
# Enable server-side diff (ArgoCD 2.10+)
argocd app set checkout-service --server-side-diff=true
# Check mutating webhooks
kubectl get mutatingwebhookconfigurations -o nameInterview Tip
This is a very common ArgoCD interview question because it happens constantly in production. Mention server-side defaults and mutating webhooks as the top two causes. The ignoreDifferences config and server-side diff are the production solutions.