How does Crossplane reconciliation work and how do you handle drift detection?

Crossplane's reconciliation engine is built on the Kubernetes controller pattern, where every managed resource has a dedicated controller running a continuous observe-compare-act loop. When the platform team defines an RDS instance for the payments-api, the AWS provider controller watches for that resource, calls the cloud API to observe the current state, compares it against the desired spec, and takes corrective action if differences exist. This loop runs on a configurable interval, defaulting to one minute for most providers, and ensures that infrastructure configuration remains consistent with what is declared in the Kubernetes API server regardless of manual changes, automated policies, or external drift.

The reconciliation loop has four distinct phases that execute in sequence. The Observe phase calls the cloud provider API to read the current state of the external resource, such as querying the AWS DescribeDBInstances API for the payments-api RDS instance. The controller maps the cloud API response back into the Crossplane resource status fields, populating conditions like Ready, Synced, and LastAsyncOperation. The Compare phase checks whether the observed external state matches the desired spec fields. The provider performs a field-by-field diff, identifying properties that have diverged such as an instance class that was manually changed from db.r5.xlarge to db.r5.2xlarge through the AWS console. The Act phase issues update API calls to bring the external resource back into compliance. For the RDS instance, this might call ModifyDBInstance to revert the instance class. Finally, the Status Update phase writes the reconciliation result back to the managed resource status, setting the Synced condition to True if everything matches or False with a descriptive message if reconciliation failed.

Drift detection is automatic and continuous, but teams need to understand its nuances to operate Crossplane effectively. Not all fields on a managed resource are reconciled equally. Some fields are immutable after creation, meaning Crossplane detects drift but cannot correct it without destroying and recreating the resource. For example, changing the engine field on an RDS instance from postgres to mysql would require recreation. The checkout-service team might see a Synced=False condition with a message indicating the field cannot be updated in place. Late-initialized fields present another subtlety. Many cloud resources have properties that the provider sets at creation time and then writes back into the Crossplane spec through a process called late initialization. For instance, the VPC ID or the assigned availability zone for an order-processing-service ElastiCache cluster may not be specified in the claim but gets populated after creation. These late-initialized fields become part of the desired state going forward.

Tuning reconciliation behavior is essential for production stability and cost management. The pollInterval on the provider controls how frequently the observe loop runs. Setting it too low generates excessive API calls that can trigger rate limiting on the cloud provider, especially when managing hundreds of resources for services like inventory-sync and user-auth-service across multiple accounts. Setting it too high delays drift detection, allowing manual changes to persist longer. Most production teams settle on intervals between one and five minutes as a balance. The managementPolicy field on managed resources controls whether Crossplane should only observe without taking corrective action (ObserveOnly), create and observe but never update or delete (CreateAndObserve), or fully manage the lifecycle (Default). ObserveOnly mode is particularly useful for importing existing infrastructure that the checkout-service team created manually before adopting Crossplane, allowing visibility without risking unintended changes.

Monitoring drift events is critical for understanding operational patterns. Teams should configure alerts on the Synced condition transitioning to False, which indicates that either drift was detected and correction failed, or that the cloud API returned an error. Crossplane emits Kubernetes events on managed resources during reconciliation, and these events can be forwarded to observability platforms through tools like kube-state-metrics or Falco. A dashboard showing drift frequency per resource type helps platform teams identify systemic issues, such as an automation script in the payments-api account that repeatedly modifies security group rules, triggering constant reconciliation cycles. The events also provide timestamps that help correlate infrastructure changes with application incidents during post-mortems.