How does ArgoCD manage multiple clusters using cluster secrets, sharded repo-servers, and Redis clustering, and what architectural decisions determine whether the control plane scales or collapses?

Think of an air traffic control center managing flights across multiple airports. Each airport is a separate cluster, the control center is the ArgoCD installation, and the radar screens are the cached views of cluster state. If one radar screen handles all airports, it becomes overwhelmed. Sharding distributes airports across multiple controllers, and a shared communication backbone (Redis) keeps everyone coordinated.

ArgoCD was originally designed to manage applications on the cluster where it runs, but production enterprises routinely manage 50 to 300 clusters from a single ArgoCD instance. Each external cluster is registered as a Kubernetes Secret in the argocd namespace with the label argocd.argoproj.io/secret-type=cluster. This Secret contains the cluster API server URL, a bearer token or client certificate, TLS CA data, and optional configuration like connection limits and namespace restrictions. The application-controller watches these secrets and treats each registered cluster as a sync target.

Internally, three components must scale independently. The application-controller reconciles Application resources by comparing desired state (from git) with live state (from clusters). At scale, a single controller instance becomes CPU-bound when diffing thousands of resources across dozens of clusters. ArgoCD supports sharding the controller so that each shard handles a subset of clusters, assigned by round-robin or explicit annotation. The repo-server clones git repositories, renders Helm charts, runs Kustomize, and returns manifests. Multiple repo-server replicas share the load, and the parallelism limit per server controls concurrent rendering operations. Redis caches rendered manifests, application state, and RBAC data. Without proper Redis availability, every cache miss triggers a full git clone and render cycle.

At production scale with 100-plus clusters, architects must monitor application-controller queue depth, reconciliation duration per cluster, repo-server clone and render times, Redis memory usage and eviction rates, and git server request rates. The controller's ARGOCD_CONTROLLER_REPLICAS environment variable sets shard count, and each replica processes only its assigned clusters. Redis should run in sentinel mode or as a Redis cluster for high availability, because a Redis failure causes all repo-server cache lookups to fall back to git, which can overwhelm the git server and spike reconciliation times by an order of magnitude.

The non-obvious gotcha is cluster Secret rotation. When a service account token or client certificate in a cluster Secret expires, ArgoCD loses access to that cluster silently unless health checks are monitored. Applications appear OutOfSync or Unknown, but the root cause is a credential failure, not a drift issue. Architects should automate Secret rotation with external-secrets-operator or a custom controller and set up alerts on cluster connection errors in the application-controller logs before tokens approach expiry.