advancedoperations

How do you handle Crossplane upgrades and provider version management in production?

⚡

Quick Answer

Manage Crossplane upgrades using a staged rollout strategy: test in a non-production control plane first, upgrade the Crossplane core via Helm with careful CRD migration, then upgrade providers individually using revision activation policies. Use provider version pinning, revision history limits, and automated health checks to enable safe rollbacks if a provider upgrade introduces regressions.

Detailed Answer

Handling Crossplane upgrades and provider version management in production requires a systematic approach because the Crossplane ecosystem has three independent versioning axes: the Crossplane core runtime, individual providers, and the Compositions that define infrastructure blueprints. A careless upgrade can break reconciliation for thousands of managed resources across services like payments-api, checkout-service, and inventory-sync, so organizations must treat Crossplane upgrades with the same rigor as Kubernetes control plane upgrades.

Crossplane core upgrades are managed through Helm. The Crossplane Helm chart deploys the core controllers, RBAC resources, and webhook configurations. Before upgrading, review the release notes for breaking changes in the API extensions, package manager, or composition engine. CRD schema changes are particularly sensitive because they affect how existing XRDs and Compositions are validated. The upgrade process starts with applying the new CRDs before upgrading the controllers, which the Helm chart handles automatically. However, if you have custom webhook configurations or admission controllers that validate Crossplane resources, these must be compatible with the new CRD schemas before the upgrade proceeds.

Provider version management leverages Crossplane's package revision system. Each provider version is a separate PackageRevision object in the cluster. The Provider resource's revisionActivationPolicy controls how new versions are activated. With the Automatic policy, installing a new provider version immediately activates it and deactivates the old revision. With the Manual policy, the new revision is installed but remains inactive until explicitly activated, allowing the platform team to test compatibility before switching. For production environments managing order-processing-service and user-auth-service infrastructure, the Manual activation policy provides a safety net.

The staged upgrade process follows a pipeline pattern. First, upgrade in a development control plane cluster that mirrors production's provider configurations and compositions but manages sandbox cloud accounts. Run comprehensive tests that provision and destroy sample resources for each composition to verify the new provider version handles the full lifecycle correctly. Second, upgrade in a staging control plane cluster that manages pre-production infrastructure. Monitor for reconciliation errors over a burn-in period of at least 24 hours. Third, upgrade in the production control plane using the Manual activation policy. Activate the new provider revision, monitor reconciliation metrics, and keep the previous revision available for instant rollback.

Provider revision rollback is straightforward but must be tested. Each provider maintains a revision history controlled by revisionHistoryLimit. If a new provider-aws version introduces a regression that breaks RDS cluster reconciliation for the payments-api database, you can revert by changing the Provider resource to reference the previous version. Crossplane deactivates the current revision and reactivates the previous one. However, if the new provider version modified the CRD schema, rolling back might require manual CRD patching because Kubernetes does not automatically downgrade CRDs.

Automated version management integrates with GitOps. Store Provider resources in a Git repository and use Renovate Bot or Dependabot to create pull requests when new provider versions are released. The CI pipeline runs integration tests against the proposed provider version before the PR is merged. Use Kyverno or OPA Gatekeeper policies to enforce that all Provider resources in the production control plane must pin exact versions rather than floating tags, preventing unexpected provider updates. Monitor the crossplane_provider_revision_health metric to alert when a provider revision enters an unhealthy state after an upgrade.

# Provider resource with version pinning and manual activation apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: # AWS provider for managing production infrastructure name: provider-aws spec: # Pin to an exact version for reproducible deployments package: xpkg.upbound.io/upbound/provider-aws-ec2:v1.2.1 # Manual activation requires explicit approval after install revisionActivationPolicy: Manual # Keep 3 revisions for rollback capability revisionHistoryLimit: 3 # Controller configuration for resource limits runtimeConfigRef: name: provider-aws-runtime --- # Runtime config to tune provider controller resources apiVersion: pkg.crossplane.io/v1beta1 kind: DeploymentRuntimeConfig metadata: # Runtime config for the AWS provider controller name: provider-aws-runtime spec: deploymentTemplate: spec: # Set replica count for high availability replicas: 2 template: spec: containers: - name: package-runtime resources: limits: # Memory limit to prevent OOM during large reconciliation cycles memory: 1Gi # CPU limit for predictable scheduling cpu: 500m requests: # Memory request for baseline operation memory: 512Mi # CPU request for guaranteed scheduling cpu: 250m --- # Helm values for Crossplane core upgrade # helm upgrade crossplane crossplane-stable/crossplane \ # --namespace crossplane-system \ # --version 1.15.0 \ # --set args='{--enable-composition-functions}' \ # --set metrics.enabled=true \ # --wait --timeout 5m # Check provider revision status after upgrade kubectl get providerrevisions -o wide # Activate a specific provider revision manually kubectl patch providerrevision provider-aws-abc123 \ --type merge -p '{"spec":{"desiredState":"Active"}}' # Roll back to previous revision if issues detected kubectl patch providerrevision provider-aws-xyz789 \ --type merge -p '{"spec":{"desiredState":"Active"}}' # Monitor reconciliation health after provider upgrade kubectl get managed -o wide | grep -v 'True' # Check for resources stuck in a non-ready state kubectl get managed -o json | jq '.items[] | select(.status.conditions[]? | select(.type=="Ready" and .status!="True")) | .metadata.name' # Verify no CRD schema conflicts after upgrade kubectl get crds -l pkg.crossplane.io/package=provider-aws -o name | wc -l # Kyverno policy to enforce exact version pinning on Providers apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: # Policy name describing its enforcement purpose name: require-provider-version-pin spec: # Block non-compliant Provider resources from being created validationFailureAction: Enforce rules: # Rule to validate provider package version format - name: check-version-pin match: resources: # Apply to all Provider resources kinds: - Provider validate: message: "Provider package must use an exact version tag (e.g., v1.2.1)" pattern: spec: # Require the package reference to contain a version tag package: "*:v*"

How do you handle Crossplane upgrades and provider version management in production?

More Crossplane questions