How do you troubleshoot Crossplane resource provisioning failures?

Crossplane resource provisioning failures are inevitable in production, and having a systematic troubleshooting methodology is essential for platform engineers. When the payments-api team reports that their database claim has been pending for thirty minutes, the investigation follows a layered approach starting from the user-facing claim and drilling down through the composite resource, composed managed resources, provider logs, and cloud API responses. Each layer reveals different types of failures, from schema validation errors at the claim level to IAM permission denials at the cloud API level.

The first diagnostic step is examining the claim and composite resource conditions. Running kubectl describe on the PostgreSQLInstance claim shows conditions including Ready, Synced, and any custom conditions set by the Composition. A Synced=False condition indicates that Crossplane cannot reconcile the desired state with the external resource, and the condition message provides the first clue. Common messages include cannot resolve reference meaning a dependency is not ready, cannot assume role indicating credential issues, or InvalidParameterCombination revealing an incompatible set of cloud resource configurations. The composite resource sits above the claim and aggregates the readiness of all composed resources, so describing the XPostgreSQLInstance resource shows which specific composed resource is blocking overall readiness. If four out of five composed resources show Ready=True and one shows Ready=False, the problem is isolated to that single resource.

The second layer involves inspecting the managed resources individually. The kubectl get managed command lists all Crossplane-managed resources with their Synced and Ready status, providing a cluster-wide view of provisioning health. For the failing order-processing-service database, kubectl describe on the specific RDS Instance managed resource reveals detailed events. Crossplane emits events like cannot create external resource with an error message from the cloud API, such as DBSubnetGroupNotFoundFault indicating that the subnet group referenced in the RDS configuration does not exist or is not ready yet. The events section also shows timestamps for each reconciliation attempt, helping correlate failures with other cluster or cloud events. Field-level validation errors appear as events with messages like spec.forProvider.instanceClass is invalid, pointing directly to the misconfigured field.

Provider pod logs contain the most granular debugging information. The AWS provider pod runs in the crossplane-system namespace and logs every API call it makes to AWS, including the request parameters, response status codes, and error messages. For the checkout-service provisioning failure, filtering logs by the resource name reveals the exact AWS API call that failed and the error response. Common discoveries include throttling errors when the provider is managing hundreds of resources and hitting AWS API rate limits, access denied errors when the IAM role in the ProviderConfig lacks the required permissions, and resource limit exceeded errors when the AWS account has hit a service quota. Provider logs also show the reconciliation timing, helping identify whether the issue is intermittent due to rate limiting or persistent due to misconfiguration. Setting the provider log level to debug through the ControllerConfig resource increases verbosity for troubleshooting complex issues.

Credential and permission validation is the third critical troubleshooting domain. The ProviderConfig might reference a Kubernetes Secret that has been deleted or rotated, an IRSA service account annotation that does not match the IAM role trust policy, or an AssumeRole chain that fails at an intermediate step. The kubectl describe providerconfig command shows whether the ProviderConfig itself is healthy. For IRSA-based authentication, verifying the service account annotation, the IAM role trust policy, and the OIDC provider configuration requires cross-checking between Kubernetes, IAM, and EKS resources. A common failure mode for the user-auth-service is that the ProviderConfig works fine for existing resources but fails for new resource types because the IAM policy attached to the assumed role lacks permissions for the new AWS service. Testing permissions with aws sts assume-role followed by the specific API call from outside the cluster helps isolate whether the issue is in the Crossplane authentication chain or in the IAM policy itself.

Preventive troubleshooting practices include setting up Prometheus alerts on the crossplane_managed_resource_ready metric that fires when resources remain not-ready beyond a threshold, configuring PagerDuty integration for Synced=False conditions on production resources, and running crossplane beta validate in CI pipelines to catch schema validation errors before they reach the cluster. The crossplane beta trace command is particularly valuable because it shows the complete resource tree from claim through composite to all managed resources with their status in a single output, eliminating the need to manually describe each resource in the chain when the inventory-sync team reports a failure.