How do you implement Crossplane in a multi-cluster, multi-cloud architecture?

Implementing Crossplane in a multi-cluster, multi-cloud architecture requires designing a control plane topology that balances centralized governance with operational isolation. The most common production pattern is a dedicated control plane cluster that runs only Crossplane and its providers, separate from workload clusters where applications like payments-api, user-auth-service, and checkout-service run. This separation ensures that control plane failures do not affect running applications and that infrastructure reconciliation loops do not compete with application workloads for cluster resources.

The control plane cluster hosts multiple Crossplane providers simultaneously, typically provider-aws, provider-gcp, and provider-azure, each configured with ProviderConfigs that hold cloud credentials. For a multi-account setup, you create multiple ProviderConfigs per cloud: one for the production AWS account, one for staging, one for the GCP project hosting analytics workloads, and so on. Each managed resource or composition references a specific ProviderConfig via providerConfigRef, directing Crossplane to use the correct credentials when reconciling. The order-processing-service might reference providerConfigRef: aws-production while inventory-sync references providerConfigRef: gcp-analytics, all managed from the same control plane.

Composition selection enables multi-cloud abstraction. A single XRD like XDatabaseInstance can have multiple Compositions: one for AWS Aurora, one for GCP Cloud SQL, and one for Azure Database. The Claim includes a compositionSelector with matchLabels to pick the right implementation. This allows the platform team to define a consistent API across clouds while the underlying provisioning logic differs per provider. When the payments-api team moves from AWS to GCP, they change one label in their Claim rather than rewriting their infrastructure manifests. Composition functions can add runtime logic like checking cloud-specific quotas before provisioning.

Multi-cluster resource distribution requires connecting the Crossplane control plane to workload clusters. Crossplane itself provisions the workload clusters using managed resources like EKS Cluster, GKE Cluster, or AKS Cluster. After cluster creation, Crossplane can deploy applications and configurations into those clusters using provider-kubernetes or provider-helm, which connect to the workload cluster API servers using kubeconfig credentials stored as Secrets. This creates a hierarchy: the control plane cluster provisions workload clusters across clouds, then deploys applications into them. The checkout-service runs on EKS in AWS while user-auth-service runs on GKE in GCP, all orchestrated from the central Crossplane control plane.

Production resilience for multi-cloud Crossplane requires careful credential management, failure domain isolation, and observability. Store cloud credentials in external secret managers like HashiCorp Vault using External Secrets Operator, rotate them automatically, and audit access. Implement circuit breakers for cloud API rate limiting by configuring provider poll intervals and concurrency limits. Monitor Crossplane's reconciliation metrics per provider to detect when a cloud API is degraded. Use Velero to back up the control plane cluster's etcd, because losing the Crossplane state means losing the mapping between desired and actual infrastructure across all clouds. For disaster recovery, maintain a standby control plane cluster that can be promoted if the primary fails, with shared state stored in an external database or replicated etcd cluster.