How do you optimize Crossplane performance for managing thousands of resources?

Optimizing Crossplane performance for managing thousands of resources requires understanding the reconciliation mechanics of both the Crossplane core and individual provider controllers. Each managed resource in Crossplane is watched by a controller that periodically polls the cloud API to detect drift and reconcile toward the desired state. When you scale to thousands of resources across services like payments-api, order-processing-service, checkout-service, user-auth-service, and inventory-sync, the default controller settings become bottlenecks: cloud API rate limits are exhausted, controller pods consume excessive memory caching resource state, and reconciliation latency increases as the controller work queue grows.

The first optimization is tuning provider controller concurrency. Each provider controller has a maxReconcileRate parameter that controls how many resources can be reconciled simultaneously. The default is typically 10, which is conservative for production. Increasing this to 50 or 100 allows the controller to process more resources in parallel, but you must balance this against cloud API rate limits. AWS, for example, has per-service rate limits that vary by API: EC2 DescribeInstances allows 100 requests per second, but RDS DescribeDBClusters allows only 25. If the payments-api team has 200 RDS-related managed resources and the controller tries to reconcile them all at maxReconcileRate: 100, it will hit rate limits and trigger exponential backoff, actually slowing down reconciliation.

The second optimization involves using Crossplane's provider family packages instead of monolithic providers. The monolithic provider-aws installs CRDs for all AWS services (500+ CRDs), even if you only use EC2, RDS, and S3. Each CRD consumes API server memory and increases watch establishment time. Provider family packages like provider-aws-ec2, provider-aws-rds, and provider-aws-s3 install only the CRDs for the services you use. For a deployment managing infrastructure for checkout-service (EC2, ALB), order-processing-service (SQS, RDS), and inventory-sync (S3, DynamoDB), you install only four provider family packages instead of the monolithic provider, reducing CRD count from 500+ to under 50.

The third optimization is poll interval management. Crossplane's default poll interval is 1 minute, meaning every managed resource makes a cloud API call every 60 seconds. For stable, long-lived resources like VPCs, subnets, and IAM roles that rarely change, this is wasteful. Use the crossplane.io/paused annotation to completely stop reconciliation on resources that should not change, or configure longer poll intervals at the provider level. Some providers support resource-level poll interval overrides via annotations, allowing you to set a 10-second poll for actively changing resources (like a database being restored) and a 10-minute poll for stable resources.

The fourth optimization is etcd and API server tuning. Thousands of managed resources mean thousands of objects in etcd, each with status subresources that update on every reconciliation cycle. This generates significant write I/O in etcd and watch event traffic through the API server. Tune etcd's --quota-backend-bytes to accommodate the larger dataset, increase the API server's --max-requests-inflight for higher concurrent watch connections, and consider using an etcd cluster with SSD-backed storage. Monitor etcd's database size and compaction frequency to prevent performance degradation.

The fifth optimization is sharding and resource distribution. For very large deployments, split managed resources across multiple ProviderConfigs that point to different cloud API endpoints or accounts. This distributes API rate limit consumption across multiple credential sets. Additionally, consider running multiple instances of the same provider with different ProviderConfig assignments, using label selectors to partition which resources each instance reconciles. This horizontal scaling approach allows the platform team to manage 10,000+ resources across the order-processing-service, payments-api, and user-auth-service without any single controller instance becoming a bottleneck.