Loading...
Crossplane
Everything for Crossplane in one place — pick a section below. 36 reviewed items across 3 content types.
📝Interview Questions24Q&A with detailed answers and code examples→
🏭Use Cases6Real-world production patterns and scenarios→
⭐Best Practices6Security, scaling, and day-2 operations→
🗺️Learning PathA guided path from fundamentals to advanced→
24 / 24
⚡
Quick Answer
Crossplane is an open-source Kubernetes add-on that transforms your Kubernetes cluster into a universal control plane for managing cloud infrastructure. It extends the Kubernetes API with custom resources representing cloud services, allowing you to provision and manage databases, networks, and storage using the same kubectl and YAML workflows you already use for application deployments.
Detailed Answer
Imagine you have a central command center that already runs your applications. Instead of having a separate room with different controls for your cloud infrastructure, what if you could manage everything from that same command center using the same language and processes? That is exactly what Crossplane does. It turns your existing Kubernetes cluster into a unified control plane that can provision and manage infrastructure across AWS, GCP, Azure, and any other cloud provider, all through the familiar Kubernetes API.
Crossplane installs as a set of controllers inside your Kubernetes cluster. Once installed, it extends the Kubernetes API by registering Custom Resource Definitions (CRDs) for cloud resources like RDS databases, S3 buckets, VPCs, and Cloud SQL instances. When you apply a YAML manifest describing an AWS RDS instance, the Crossplane controller picks it up from the API server, calls the AWS API to create the actual database, and then continuously reconciles the desired state in your YAML with the real state in AWS. If someone manually changes the database configuration through the AWS console, Crossplane detects the drift and corrects it automatically, just like how a Kubernetes Deployment controller replaces a crashed Pod.
For a team running the payments-api service, this means the same GitOps pipeline that deploys application code can also provision the PostgreSQL database that payments-api depends on. The infrastructure YAML lives in the same repository, goes through the same pull request review, and gets applied by the same ArgoCD or Flux pipeline. A developer on the order-processing-service team does not need AWS console access or Terraform CLI installed locally. They simply write a Kubernetes manifest, push it to Git, and the infrastructure appears. This eliminates the traditional handoff between development and infrastructure teams, where a developer submits a Jira ticket and waits days for a database to be provisioned.
The reconciliation loop is what makes Crossplane fundamentally different from imperative infrastructure tools. Terraform runs once and produces a state file, but between runs, your infrastructure can drift without detection. Crossplane controllers run continuously inside the cluster, watching for changes every few seconds. If the inventory-sync service needs a Redis cache and someone accidentally deletes it from the cloud console, Crossplane recreates it automatically. This continuous reconciliation model means your infrastructure is self-healing, just like your application workloads already are under Kubernetes.
Crossplane also introduces a powerful abstraction layer through Compositions and Composite Resources, which let platform teams define opinionated infrastructure templates. Instead of exposing raw cloud primitives to every developer, the platform team can create a custom DatabaseClaim resource that bundles an RDS instance, a security group, a subnet group, and a Kubernetes Secret with connection details. The checkout-service team just creates a DatabaseClaim and gets a fully configured, production-ready database without knowing the underlying cloud specifics. This separation of concerns between platform engineers and application developers is one of the core design principles of Crossplane.
Code Example
# Install Crossplane into the crossplane-system namespace
helm repo add crossplane-stable https://charts.crossplane.io/stable
# Update the Helm repo index to get latest chart versions
helm repo update
# Install Crossplane with default configuration
helm install crossplane crossplane-stable/crossplane \
--namespace crossplane-system --create-namespace
# Verify Crossplane pods are running
kubectl get pods -n crossplane-system
# Check that Crossplane CRDs are registered
kubectl get crds | grep crossplane
# Example: Provision an AWS S3 bucket for payments-api assets
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
name: payments-api-assets # Bucket for storing payment receipts
spec:
forProvider:
region: us-east-1 # Deploy in US East region
tags:
Team: payments # Tag for cost allocation
Environment: production # Production workload
providerConfigRef:
name: aws-provider-config # Reference to AWS credentials💡
Interview Tip
A junior engineer typically says Crossplane is like Terraform but for Kubernetes. While that comparison is a reasonable starting point, the interviewer wants you to articulate the fundamental architectural difference: Crossplane uses the Kubernetes reconciliation loop to continuously ensure your infrastructure matches your desired state, whereas Terraform is a run-once CLI tool that creates a point-in-time snapshot. Explain that Crossplane extends the Kubernetes API with CRDs, meaning you get all of Kubernetes' built-in capabilities for free, including RBAC, namespaces, audit logging, and the watch mechanism. Mention that this enables GitOps workflows where infrastructure and application manifests live side by side in the same repository and are deployed by the same pipeline.
◈ Architecture Diagram
┌──────────────── Kubernetes Cluster ─────────────────┐
│ │
│ ┌──────────────────────────────────────────────┐ │
│ │ Crossplane Controllers │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ AWS │ │ GCP │ │ Azure │ │ │
│ │ │ Provider │ │ Provider │ │ Provider │ │ │
│ │ └────┬─────┘ └────┬─────┘ └──────┬───────┘ │ │
│ └───────┼────────────┼──────────────┼──────────┘ │
│ │ │ │ │
│ ┌───────┴────────────┴──────────────┴──────────┐ │
│ │ Kubernetes API Server │ │
│ │ ┌────────┐ ┌────────┐ ┌────────────────┐ │ │
│ │ │ Bucket │ │ RDS │ │ VPC │ │ │
│ │ │ CRD │ │ CRD │ │ CRD │ │ │
│ │ └────────┘ └────────┘ └────────────────┘ │ │
│ └──────────────────────────────────────────────┘ │
│ │ │ │ │
└──────────┼────────────┼──────────────┼───────────────┘
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ AWS │ │ GCP │ │ Azure │
│ Cloud │ │ Cloud │ │ Cloud │
└──────────┘ └──────────┘ └──────────┘💬 Comments
⚡
Quick Answer
Crossplane Providers are plugin packages that install cloud-specific controllers and CRDs into your cluster, enabling Crossplane to manage resources on a particular cloud platform. Each Provider handles authentication, API communication, and reconciliation for its target platform, such as provider-aws for Amazon Web Services or provider-gcp for Google Cloud Platform.
Detailed Answer
Think of Crossplane Providers like adapter plugs you use when traveling internationally. Your laptop charger (your Kubernetes YAML) stays the same, but the adapter plug (Provider) translates the connection to work with the specific power outlet (cloud API) in each country. You need a different adapter for each country, and each adapter knows the exact pin configuration and voltage requirements. Similarly, each Crossplane Provider knows how to authenticate with and call the APIs of a specific cloud platform.
A Provider is a Crossplane package that bundles two things: a set of Custom Resource Definitions (CRDs) representing the cloud resources it can manage, and a controller that watches those CRDs and makes the corresponding API calls to the cloud platform. When you install provider-aws, it registers hundreds of CRDs in your cluster, including Bucket for S3, Instance for RDS, VPC for networking, and many more. The provider-aws controller then watches the Kubernetes API server for any resources of these types and translates create, update, and delete operations into AWS API calls. For the user-auth-service team that needs a DynamoDB table, they just create a Table resource in Kubernetes, and the AWS provider handles everything else.
Authentication is configured through a ProviderConfig resource, which tells the Provider how to authenticate with the cloud platform. For AWS, this could be static credentials stored in a Kubernetes Secret, an IAM Roles for Service Accounts (IRSA) configuration on EKS, or a cross-account assume role chain. For GCP, it might be a service account JSON key or Workload Identity Federation. For Azure, it could be a service principal or managed identity. The ProviderConfig is separate from the Provider installation itself, which means you can have multiple ProviderConfigs for the same Provider, each pointing to a different account or project. This is how platform teams manage multi-account setups where the payments-api infrastructure lives in a production AWS account and the checkout-service staging environment lives in a development account.
The Upbound marketplace hosts the official Providers maintained by the Crossplane community and Upbound. The most commonly used Providers are provider-aws (also called upbound/provider-aws), provider-gcp, and provider-azure, but there are also Providers for Helm, Kubernetes itself, SQL databases, and many other systems. Each Provider follows semantic versioning, and you can pin your Provider to a specific version to avoid unexpected changes. When you upgrade a Provider, it may introduce new CRDs or modify existing ones, so platform teams typically test Provider upgrades in a staging cluster before rolling them out to production where the order-processing-service and inventory-sync service depend on them.
A common beginner mistake is installing a monolithic Provider package that includes every CRD for a cloud platform. For AWS alone, this can mean over 900 CRDs, which consumes significant memory and CPU in the cluster. Modern best practice is to use Provider families, where you install only the specific resource groups you need. Instead of installing the full provider-aws, you install provider-aws-s3, provider-aws-rds, and provider-aws-ec2 separately. This reduces the resource footprint dramatically and speeds up cluster startup times, which is especially important when running Crossplane in smaller clusters that also host application workloads.
Code Example
# Install the AWS Provider using a Crossplane Provider resource
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-aws-s3 # Install only the S3 resource group
spec:
package: xpkg.upbound.io/upbound/provider-aws-s3:v1.2.0 # Pinned version
# Wait for the Provider to become healthy
---
# Create a Kubernetes Secret with AWS credentials
apiVersion: v1
kind: Secret
metadata:
name: aws-credentials # Secret containing AWS access keys
namespace: crossplane-system # Must be in Crossplane namespace
type: Opaque
stringData:
credentials: | # AWS credentials in shared config format
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
---
# Configure the Provider with authentication details
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
name: aws-provider-config # Referenced by managed resources
spec:
credentials:
source: Secret # Read credentials from Kubernetes Secret
secretRef:
name: aws-credentials # Name of the Secret created above
namespace: crossplane-system # Namespace of the Secret
key: credentials # Key within the Secret data
# Verify the Provider is installed and healthy
# kubectl get providers
# kubectl get providerconfigs💡
Interview Tip
A junior engineer typically says Providers let Crossplane talk to cloud platforms. To impress the interviewer, go deeper and explain the three-layer architecture: the Provider package installs CRDs and controllers, the ProviderConfig handles authentication separately so you can have multiple credentials for multi-account setups, and the managed resources are the actual infrastructure objects. Mention the Provider family pattern where you install granular packages like provider-aws-s3 instead of the monolithic provider-aws to reduce CRD count and memory usage. This shows you understand the operational considerations of running Crossplane at scale, not just the conceptual overview.
◈ Architecture Diagram
┌────────────── Crossplane Cluster ──────────────────┐
│ │
│ ┌─── Provider Installation ──────────────────┐ │
│ │ │ │
│ │ Provider: provider-aws-s3 │ │
│ │ → Installs CRDs: Bucket, BucketPolicy │ │
│ │ → Deploys controller pod │ │
│ │ │ │
│ │ Provider: provider-aws-rds │ │
│ │ → Installs CRDs: Instance, SubnetGroup │ │
│ │ → Deploys controller pod │ │
│ └────────────────────────────────────────────┘ │
│ │
│ ┌─── ProviderConfig ────────────────────────┐ │
│ │ │ │
│ │ name: aws-prod-account │ │
│ │ credentials → Secret (aws-creds) │ │
│ │ │ │
│ │ name: aws-dev-account │ │
│ │ credentials → IRSA (EKS role) │ │
│ └─────────────┬─────────────────────────────┘ │
│ │ │
└────────────────┼────────────────────────────────────┘
│ API calls
▼
┌──────────────┐
│ AWS Cloud │
│ ┌────────┐ │
│ │ S3 │ │
│ │ RDS │ │
│ │ EC2 │ │
│ └────────┘ │
└──────────────┘💬 Comments
⚡
Quick Answer
Managed Resources are Crossplane's representation of individual cloud infrastructure resources as Kubernetes custom resources. Each Managed Resource maps one-to-one to an external cloud resource like an S3 bucket, an RDS database, or a GCP Cloud SQL instance, and the Crossplane controller continuously reconciles its desired state with the actual state in the cloud.
Detailed Answer
Think of a Managed Resource like a reservation card at a restaurant. You fill out the card with your preferences: table for four, by the window, at 7 PM. You hand the card to the host (Kubernetes API server), and the host passes it to the restaurant manager (Crossplane controller) who actually arranges the table. If someone moves your table, the manager notices and puts it back. The card is your declaration of what you want, and the manager ensures reality matches.
A Managed Resource (MR) is the most granular unit of infrastructure in Crossplane. Each MR corresponds to exactly one external resource in a cloud provider. An S3 Bucket MR maps to one S3 bucket in AWS, an RDS Instance MR maps to one RDS database instance, a CloudSQL Instance MR maps to one Cloud SQL database in GCP. When you create a Managed Resource by applying its YAML manifest, the Crossplane provider controller calls the cloud API to create the corresponding resource. The MR's status section is then updated with the external resource's actual attributes, such as its ARN, endpoint URL, and current state. For example, when the payments-api team creates an RDS Instance MR, the status will eventually show the database endpoint that the application can connect to.
The spec of a Managed Resource is divided into two main sections. The forProvider section contains the configuration that gets sent to the cloud API, things like region, instance class, storage size, and engine version. The providerConfigRef section tells the controller which ProviderConfig to use for authentication, allowing different MRs to target different cloud accounts. There is also a deletionPolicy field that controls what happens when you delete the MR from Kubernetes. Setting it to Delete means the external resource is destroyed when the MR is deleted, which is the default. Setting it to Orphan means the external resource is left untouched in the cloud even after the MR is removed from the cluster, which is useful during migrations or when you want to decommission the Crossplane management of a resource without destroying the actual infrastructure that the order-processing-service depends on.
Crossplane continuously reconciles each Managed Resource with its external counterpart. The controller checks the external resource's state at regular intervals, typically every few minutes, and compares it against the desired state in the MR's spec. If there is drift, for example if someone manually changed the instance class of the user-auth-service database through the AWS console, Crossplane detects this and reverts the change to match the spec. This drift detection and correction is one of the most powerful features of Crossplane, as it ensures that infrastructure configuration defined in Git remains the source of truth even when humans or other tools make ad-hoc changes.
Managed Resources also support cross-resource references through the ref and selector patterns. For instance, an RDS Instance can reference a SubnetGroup and SecurityGroup by name, and Crossplane resolves these references to the actual cloud resource IDs before making the API call. This is analogous to how Kubernetes Services reference Pods through label selectors. For the checkout-service team provisioning a full database stack, they create a SubnetGroup MR, a SecurityGroup MR, and an Instance MR that references the other two. Crossplane handles the dependency ordering automatically, creating the subnet group and security group before attempting to create the database instance.
Code Example
# Managed Resource: AWS RDS Instance for the payments-api service
apiVersion: rds.aws.upbound.io/v1beta1
kind: Instance
metadata:
name: payments-api-db # Name of the Managed Resource in Kubernetes
spec:
forProvider:
region: us-east-1 # AWS region for the database
instanceClass: db.t3.medium # Instance size for production workload
engine: postgres # PostgreSQL engine type
engineVersion: "15.4" # Specific engine version
allocatedStorage: 50 # 50 GB of storage
dbName: payments # Name of the default database
masterUsername: payments_admin # Admin username for the database
masterPasswordSecretRef: # Reference to a Secret for the password
name: payments-db-password # Kubernetes Secret name
namespace: crossplane-system # Namespace of the Secret
key: password # Key within the Secret
publiclyAccessible: false # Not accessible from the internet
skipFinalSnapshot: false # Take a snapshot before deletion
tags:
Service: payments-api # Tag for identifying the owning service
ManagedBy: crossplane # Tag indicating Crossplane management
providerConfigRef:
name: aws-provider-config # Which AWS account to use
deletionPolicy: Delete # Destroy the RDS instance when MR is deleted
# Check the status of the Managed Resource
# kubectl get instance.rds.aws.upbound.io payments-api-db
# View detailed status including the endpoint
# kubectl describe instance.rds.aws.upbound.io payments-api-db💡
Interview Tip
A junior engineer typically describes Managed Resources as YAML files that create cloud resources. The interviewer wants to hear about the lifecycle and operational characteristics. Explain the three key behaviors: continuous reconciliation that detects and corrects drift, the deletionPolicy that controls whether external resources survive MR deletion, and cross-resource references that let Crossplane resolve dependency ordering automatically. Mention the status section and how it exposes the external resource's attributes like endpoint URLs and ARNs, which downstream services like payments-api use for connectivity. Also highlight that each MR maps one-to-one to an external resource, which is why Compositions exist to bundle multiple MRs into higher-level abstractions.
◈ Architecture Diagram
┌─── Managed Resource (Kubernetes) ────────────────┐
│ │
│ kind: Instance (RDS) │
│ name: payments-api-db │
│ │
│ spec: │
│ forProvider: │
│ engine: postgres │
│ instanceClass: db.t3.medium │
│ │
│ status: │
│ atProvider: │
│ endpoint: payments-api-db.xxx.rds.amazonaws │
│ status: available │
│ conditions: │
│ - type: Ready → ✓ │
│ - type: Synced → ✓ │
└──────────────────┬────────────────────────────────┘
│
Crossplane Controller
(continuous reconciliation)
│
▼
┌─── External Cloud Resource ──────────────────────┐
│ │
│ AWS RDS Instance │
│ ┌───────────────────────────────┐ │
│ │ payments-api-db │ │
│ │ engine: postgres 15.4 │ │
│ │ class: db.t3.medium │ │
│ │ storage: 50 GB │ │
│ │ status: available ✓ │ │
│ └───────────────────────────────┘ │
│ │
│ Drift detected? → Auto-corrected ← │
└───────────────────────────────────────────────────┘💬 Comments
⚡
Quick Answer
A Composite Resource (XR) is a custom, higher-level API resource defined by a platform team that bundles multiple Managed Resources into a single abstraction. It allows application developers to request complex infrastructure setups, like a database with networking and security groups, through a single simplified resource rather than managing each cloud component individually.
Detailed Answer
Imagine ordering a combo meal at a restaurant instead of ordering each item separately. You say 'I want combo number three,' and the kitchen knows that means a burger, fries, and a drink prepared in a specific way. You do not need to know the recipe for each item, the cooking temperature, or which supplier provided the ingredients. The combo meal is the abstraction that hides all the complexity behind a simple choice. In Crossplane, a Composite Resource works the same way: it is a single API object that, when created, triggers the provisioning of multiple underlying cloud resources according to a template defined by the platform team.
A Composite Resource is an instance of a custom API type defined by a CompositeResourceDefinition (XRD). The platform team defines the XRD to specify what parameters developers can configure, such as database size (small, medium, large), region, and engine type, and what parameters are hidden because they are standardized across the organization. When a developer on the order-processing-service team creates an XR, Crossplane looks up the corresponding Composition, which maps the XR's parameters to the specific Managed Resources that need to be created. A single DatabaseInstance XR might create an RDS instance, a security group, a subnet group, a parameter group, and a Kubernetes Secret containing the connection string, all from one simple YAML manifest.
The real power of Composite Resources is the separation of concerns they enable between platform engineers and application developers. Platform engineers define the Compositions, encoding organizational best practices, compliance requirements, and security policies into the infrastructure templates. They decide that every database gets encryption at rest, automated backups, multi-AZ deployment in production, and appropriate tagging for cost allocation. Application developers on the payments-api or checkout-service teams never see these details. They interact with the simplified XR API that exposes only the parameters relevant to their application, like size and region. This is the concept of a platform API or Internal Developer Platform (IDP) built directly into Kubernetes.
Composite Resources also solve the problem of infrastructure consistency across teams. Without XRs, each team would write their own Managed Resource manifests, leading to inconsistent configurations. One team might forget to enable encryption, another might use a deprecated instance class, and a third might skip setting up proper backup retention. With Composite Resources, the platform team controls the template, ensuring that every database provisioned by every team meets the same production standards. If a security audit requires changing the minimum TLS version for all databases, the platform team updates the Composition once, and all new XRs automatically pick up the change.
Composite Resources can be exposed to developers through Claims (XRCs), which are namespace-scoped versions of the cluster-scoped XR. While XRs are cluster-scoped and typically managed by platform teams, Claims are namespaced and designed for application teams. A developer in the inventory-sync namespace creates a DatabaseClaim, which Crossplane translates into a cluster-scoped XR, which in turn creates all the underlying Managed Resources. This namespace scoping integrates naturally with Kubernetes RBAC, letting platform teams control who can provision what infrastructure through standard Kubernetes role bindings rather than building a separate access control system.
Code Example
# A developer creates a Composite Resource Claim
# This single resource triggers creation of multiple cloud resources
apiVersion: database.example.org/v1alpha1
kind: PostgreSQLClaim # Claim is the namespace-scoped version of the XR
metadata:
name: payments-api-db # Name chosen by the developer
namespace: payments-team # Developer's namespace
spec:
parameters:
size: medium # Platform team maps this to db.t3.medium
region: us-east-1 # AWS region for the database
storageGB: 100 # Storage size in gigabytes
version: "15" # Major PostgreSQL version
compositionSelector:
matchLabels:
provider: aws # Select the AWS-specific Composition
writeConnectionSecretToRef:
name: payments-db-conn # Secret with connection details
# The platform team's Composition creates these resources automatically:
# 1. RDS Instance (db.t3.medium, encrypted, multi-AZ)
# 2. Security Group (port 5432 open only to cluster CIDR)
# 3. Subnet Group (private subnets only)
# 4. Parameter Group (tuned for production workloads)
# 5. Kubernetes Secret (host, port, username, password)
# Check what was created from the Claim
# kubectl get composite -l crossplane.io/claim-name=payments-api-db
# kubectl get managed -l crossplane.io/claim-name=payments-api-db💡
Interview Tip
A junior engineer typically says Composite Resources group cloud resources together. The interviewer wants you to articulate why this abstraction layer matters from an organizational perspective. Explain the separation of concerns: platform engineers encode security, compliance, and best practices into Compositions, while application developers interact with a simplified API that hides cloud-specific complexity. Mention the distinction between cluster-scoped XRs and namespace-scoped Claims, because Claims integrate with Kubernetes RBAC for access control. Highlight that this pattern creates an Internal Developer Platform where the platform team controls the golden path and developers get self-service infrastructure without needing cloud console access or deep cloud expertise.
◈ Architecture Diagram
┌─── Developer Creates ────────────────────────┐
│ │
│ PostgreSQLClaim (namespace: payments-team) │
│ size: medium, region: us-east-1 │
│ │
└──────────────────┬────────────────────────────┘
│
▼
┌─── Crossplane Creates XR ────────────────────┐
│ │
│ PostgreSQLInstance (cluster-scoped XR) │
│ │
└──────────────────┬────────────────────────────┘
│
Composition Template
│
┌─────────────┼─────────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌────────────────┐
│ RDS │ │ Security │ │ Subnet │
│ Instance │ │ Group │ │ Group │
│ │ │ │ │ │
│ db.t3. │ │ port: │ │ private │
│ medium │ │ 5432 │ │ subnets │
└──────────┘ └──────────┘ └────────────────┘
│
▼
┌──────────────────────────────────────────┐
│ Kubernetes Secret: payments-db-conn │
│ host: xxx.rds.amazonaws.com │
│ port: 5432 │
│ username: payments_admin │
│ password: ●●●●●●●● │
└──────────────────────────────────────────┘💬 Comments
⚡
Quick Answer
Crossplane is installed on a Kubernetes cluster using Helm, the Kubernetes package manager. You add the Crossplane Helm repository, install the chart into a dedicated namespace called crossplane-system, and then verify that the Crossplane pods and CRDs are running. After installation, you install Providers and configure credentials to start managing cloud infrastructure.
Detailed Answer
Installing Crossplane is straightforward, but understanding what the installation actually does helps you troubleshoot issues and plan capacity. The process has three phases: installing the Crossplane core, installing one or more Providers, and configuring authentication. Each phase builds on the previous one, and skipping or misconfiguring any step results in a non-functional setup that cannot provision infrastructure.
The first phase installs the Crossplane core using Helm. Helm deploys two main components into the crossplane-system namespace: the Crossplane pod itself, which runs the core controllers responsible for package management and Composite Resource reconciliation, and the RBAC manager pod, which automatically creates and manages the Kubernetes RBAC roles needed by Providers. The Crossplane core also registers several foundational CRDs, including Provider, ProviderConfig, Composition, CompositeResourceDefinition, and others. These CRDs are the building blocks that everything else depends on. For a team running the checkout-service and payments-api in the same cluster, the core installation is a one-time operation shared by all teams.
The second phase installs Providers by creating Provider resources that reference OCI images from a registry, typically xpkg.upbound.io. When you create a Provider resource, the Crossplane package manager downloads the Provider image, extracts the CRDs and controller binary, registers the CRDs with the Kubernetes API server, and deploys the controller as a new pod in the crossplane-system namespace. For example, installing provider-aws-rds adds CRDs like Instance, SubnetGroup, and ParameterGroup, and starts a controller pod that knows how to reconcile these resources with the AWS RDS API. Each Provider runs as its own pod, so you can install provider-aws-s3 and provider-aws-rds independently, each with its own resource allocation and lifecycle.
The third phase configures authentication by creating ProviderConfig resources. Without a ProviderConfig, Providers have no credentials and cannot make API calls to the cloud. The ProviderConfig references a Kubernetes Secret containing credentials, or on managed Kubernetes services like EKS, it can use IAM Roles for Service Accounts (IRSA) for a more secure, keyless authentication approach. Platform teams managing infrastructure for the order-processing-service and inventory-sync typically create separate ProviderConfigs for production and development accounts, ensuring that development testing never accidentally modifies production resources.
There are important prerequisites and sizing considerations before installation. Crossplane requires Kubernetes 1.16 or later, and the cluster needs sufficient resources to run the Crossplane pods and Provider controllers. Each Provider consumes memory proportional to the number of CRDs it registers and the number of Managed Resources it watches. A provider-aws with all resource groups can use over 1 GB of RAM, which is why the Provider family pattern of installing granular packages is recommended. The cluster also needs outbound internet access to pull Provider images and make cloud API calls. For air-gapped environments where the user-auth-service runs in a restricted network, you can mirror Provider images to a private registry and configure Crossplane to pull from there instead.
After installation, verification is critical. Check that the Crossplane pods are Running and Ready, that the Providers show Installed and Healthy status, and that the expected CRDs are registered. A common installation failure occurs when the Helm release succeeds but the Provider pod crashes due to insufficient memory or missing RBAC permissions. Running kubectl describe on the Provider resource and checking the pod logs are the first debugging steps when things go wrong.
Code Example
# Prerequisites: Ensure Helm 3 and kubectl are installed
helm version --short # Verify Helm is available
kubectl cluster-info # Verify cluster connectivity
# Step 1: Add the Crossplane Helm repository
helm repo add crossplane-stable https://charts.crossplane.io/stable
# Step 2: Update Helm repos to get the latest chart version
helm repo update
# Step 3: Install Crossplane into its dedicated namespace
helm install crossplane crossplane-stable/crossplane \
--namespace crossplane-system \
--create-namespace \
--set args='{--enable-usages}' # Enable the Usages feature
# Step 4: Verify Crossplane pods are running
kubectl get pods -n crossplane-system
# Expected: crossplane-xxx Running, crossplane-rbac-manager-xxx Running
# Step 5: Verify core CRDs are registered
kubectl get crds | grep crossplane.io
# Step 6: Install an AWS Provider
kubectl apply -f - <<EOF
apiVersion: pkg.crossplane.io/v1 # Crossplane package API
kind: Provider # Resource type for installing providers
metadata:
name: provider-aws-s3 # Name for this provider installation
spec:
package: xpkg.upbound.io/upbound/provider-aws-s3:v1.2.0 # OCI image
EOF
# Step 7: Wait for the Provider to become healthy
kubectl get providers -w # Watch until HEALTHY shows True
# Step 8: Configure AWS credentials
kubectl create secret generic aws-creds \
-n crossplane-system \
--from-file=credentials=$HOME/.aws/credentials # Use local AWS creds💡
Interview Tip
A junior engineer typically recites the Helm install commands without explaining what they do. The interviewer wants to hear about the three-phase installation process: core Crossplane, Providers, and ProviderConfig authentication. Explain that the core installs the package manager and Composition controllers, Providers install cloud-specific CRDs and controllers as separate pods, and ProviderConfigs handle authentication separately so you can target multiple cloud accounts. Mention the resource sizing consideration that each Provider consumes memory proportional to its CRD count, which is why Provider families exist. This demonstrates that you have planned and executed Crossplane installations in real environments, not just followed a quickstart tutorial.
◈ Architecture Diagram
┌─── Installation Flow ────────────────────────────┐ │ │ │ Step 1: Helm Install Core │ │ ┌─────────────────────────────────────────┐ │ │ │ crossplane-system namespace │ │ │ │ ┌──────────────┐ ┌──────────────────┐ │ │ │ │ │ Crossplane │ │ RBAC Manager │ │ │ │ │ │ Controller │ │ Controller │ │ │ │ │ └──────────────┘ └──────────────────┘ │ │ │ │ Core CRDs: Provider, Composition, XRD │ │ │ └─────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ Step 2: Install Providers │ │ ┌─────────────────────────────────────────┐ │ │ │ ┌───────────┐ ┌───────────┐ │ │ │ │ │provider │ │provider │ │ │ │ │ │-aws-s3 │ │-aws-rds │ │ │ │ │ │ │ │ │ │ │ │ │ │ + CRDs │ │ + CRDs │ │ │ │ │ │ + ctrl pod│ │ + ctrl pod│ │ │ │ │ └───────────┘ └───────────┘ │ │ │ └─────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ Step 3: Configure Authentication │ │ ┌─────────────────────────────────────────┐ │ │ │ ProviderConfig → Secret (credentials) │ │ │ │ or IRSA / Workload Identity │ │ │ └─────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ✓ Ready to provision infrastructure │ └───────────────────────────────────────────────────┘
💬 Comments
⚡
Quick Answer
A Crossplane Composition is a template that maps a Composite Resource (XR) to a specific set of Managed Resources. It defines exactly which cloud resources to create, how to configure them, and how to wire parameters from the XR down to each underlying resource. Platform teams use Compositions to encode infrastructure best practices and organizational standards into reusable templates.
Detailed Answer
Think of a Composition like a blueprint filed at an architecture firm. When a client (developer) requests a standard two-bedroom apartment (Composite Resource), the architect does not design from scratch every time. Instead, they pull the standard blueprint (Composition) that specifies exactly where the walls go, what materials to use, how the plumbing connects, and which electrical codes to follow. The client chooses the apartment size and floor, but the structural decisions are already made by the firm's experts. The blueprint is the single source of truth that guarantees every apartment meets building codes and the firm's quality standards.
A Composition resource in Crossplane connects a Composite Resource Definition (XRD) to the actual cloud resources that should be created when someone instantiates an XR. It contains a list of resource templates, each defining a Managed Resource with its full configuration. The Composition uses patches to map values from the XR's spec into the correct fields of each Managed Resource. For instance, when the payments-api team creates a PostgreSQLInstance XR with size set to large, the Composition patches that value into the RDS Instance's instanceClass as db.r5.xlarge, sets the allocatedStorage to 500 GB, enables multi-AZ deployment, and configures performance insights. The XR's region parameter gets patched into every Managed Resource that needs a region field.
Patches are the mechanism that makes Compositions dynamic rather than static. Crossplane supports several patch types. The FromCompositeFieldPath patch copies a value from the XR down to a Managed Resource, like mapping spec.parameters.region to spec.forProvider.region. The ToCompositeFieldPath patch copies a value from a Managed Resource back up to the XR's status, like surfacing the database endpoint. The Combine patch merges multiple fields, for example concatenating the service name and environment to generate a unique resource identifier. Transform patches apply functions to values, such as mapping the string large to the integer 500 for storage size. These patches give platform teams fine-grained control over how developer inputs translate to cloud configurations.
Compositions also handle the connection details pipeline, which automatically propagates secrets from Managed Resources up to the XR and eventually to the Claim. When an RDS Instance is created, AWS generates an endpoint hostname and the password is already stored in a Secret. The Composition defines connectionDetails entries that extract these values and publish them as a Kubernetes Secret that the checkout-service or user-auth-service application can mount as environment variables. This eliminates the manual step of copying connection strings between teams and systems, which is error-prone and a security risk.
A key design decision is that Compositions support selection through labels. A single XRD can have multiple Compositions, each targeting a different cloud provider or environment tier. The PostgreSQLInstance XRD might have an aws-production Composition that creates multi-AZ RDS with encryption and a gcp-development Composition that creates a basic Cloud SQL instance. The XR selects which Composition to use through a compositionSelector with label matching. This enables true multi-cloud support where the inventory-sync team can provision identical database abstractions on different clouds simply by changing a label. Platform teams can also set a default Composition on the XRD, so developers who do not specify a preference automatically get the production-ready configuration.
Code Example
# Composition: Maps a PostgreSQLInstance XR to AWS resources
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
name: postgresql-aws-production # Name of this Composition template
labels:
provider: aws # Label for Composition selection
tier: production # Production-grade template
spec:
compositeTypeRef:
apiVersion: database.example.org/v1alpha1 # XR API group
kind: PostgreSQLInstance # XR kind this Composition satisfies
resources:
- name: rds-instance # First resource: the RDS database
base:
apiVersion: rds.aws.upbound.io/v1beta1 # AWS RDS API
kind: Instance # Managed Resource type
spec:
forProvider:
engine: postgres # Always PostgreSQL
publiclyAccessible: false # Never public in production
storageEncrypted: true # Encryption at rest enforced
multiAZ: true # High availability enabled
deletionPolicy: Delete # Clean up on removal
patches:
- type: FromCompositeFieldPath # Map XR region to MR region
fromFieldPath: spec.parameters.region
toFieldPath: spec.forProvider.region
- type: FromCompositeFieldPath # Map XR size to instance class
fromFieldPath: spec.parameters.size
toFieldPath: spec.forProvider.instanceClass
transforms:
- type: map # Transform size label to AWS instance type
map:
small: db.t3.small # Small maps to t3.small
medium: db.t3.medium # Medium maps to t3.medium
large: db.r5.xlarge # Large maps to r5.xlarge
connectionDetails:
- name: host # Expose the RDS endpoint as a connection detail
fromFieldPath: status.atProvider.endpoint
type: FromFieldPath
- name: security-group # Second resource: network security
base:
apiVersion: ec2.aws.upbound.io/v1beta1 # AWS EC2 API
kind: SecurityGroup # Security group for the database
spec:
forProvider:
description: "Database security group" # Human-readable label💡
Interview Tip
A junior engineer typically says a Composition defines what resources to create. The interviewer is looking for a deeper understanding of the patch system and why it matters. Explain that patches are the bridge between the developer-facing XR API and the cloud-specific Managed Resources, using FromCompositeFieldPath to push parameters down and ToCompositeFieldPath to surface status information back up. Mention the transform capability that maps human-friendly values like small, medium, and large to cloud-specific instance types. Highlight that multiple Compositions can satisfy the same XRD, enabling multi-cloud and multi-tier strategies where the same developer API provisions infrastructure on different providers or with different production-readiness levels based on label selection.
◈ Architecture Diagram
┌─── XR: PostgreSQLInstance ──────────────┐
│ spec: │
│ parameters: │
│ size: medium │
│ region: us-east-1 │
│ compositionSelector: │
│ matchLabels: │
│ provider: aws │
└──────────────┬──────────────────────────┘
│
▼
┌─── Composition: postgresql-aws ─────────┐
│ │
│ Patches: │
│ size: medium → db.t3.medium │
│ region → us-east-1 │
│ │
│ Resources: │
│ ┌─────────┐ ┌─────────┐ ┌──────────┐ │
│ │ RDS │ │ SecGrp │ │ Subnet │ │
│ │Instance │ │ │ │ Group │ │
│ │ │ │ port: │ │ │ │
│ │db.t3. │ │ 5432 │ │ private │ │
│ │medium │ │ │ │ subnets │ │
│ └────┬────┘ └─────────┘ └──────────┘ │
│ │ │
│ ▼ connectionDetails │
│ ┌──────────────────────────────┐ │
│ │ Secret: host, port, password │ │
│ └──────────────────────────────┘ │
└─────────────────────────────────────────┘💬 Comments
⚡
Quick Answer
A CompositeResourceDefinition (XRD) is the schema definition that creates a new custom API type in Crossplane. It defines the structure, parameters, and validation rules for both the cluster-scoped Composite Resource (XR) and optionally its namespace-scoped Claim (XRC), essentially creating the contract between platform engineers and application developers.
Detailed Answer
Think of an XRD like a form template that an HR department creates for employee requests. The template defines which fields exist (employee name, department, request type), which fields are required versus optional, what valid values look like (department must be one of engineering, marketing, or finance), and what the form is called (Time Off Request). The HR team designs the form once, and then every employee fills in their own copy. The form structure never changes between employees; only the values they enter differ. In Crossplane, the XRD is that form template, defining the API structure that developers use to request infrastructure.
An XRD registers two Kubernetes Custom Resource Definitions (CRDs) with the API server. The first is the Composite Resource (XR), which is cluster-scoped and typically used by platform teams or automation. The second is the Claim (XRC), which is namespace-scoped and designed for application developers. The XRD specifies the API group, kind names for both the XR and Claim, the supported versions, and an OpenAPI v3 schema that defines all the configurable parameters. When the payments-api team needs a database, they create a PostgreSQLClaim in their namespace, and Crossplane validates it against the schema defined in the XRD before accepting it.
The schema section of the XRD is where platform engineers make critical design decisions about what to expose to developers and what to hide. For a database XRD, the platform team might expose parameters like size with an enum of small, medium, and large, region with a list of allowed regions, and engineVersion with supported PostgreSQL versions. They would not expose parameters like subnet IDs, security group rules, or encryption algorithms, because those are standardized across the organization and encoded in the Composition. This schema acts as the contract between the platform team and the application teams. The checkout-service developers know exactly what they can configure, and the platform team knows exactly what guardrails are in place.
The XRD also defines which connection detail keys the Composite Resource publishes. When a database XR is created, it generates connection information like hostname, port, username, and password. The XRD's connectionSecretKeys field declares which of these keys are part of the published API. This is important because it determines what information ends up in the Kubernetes Secret that the application Pod reads for database connectivity. If the platform team adds a new connection detail, like a read replica endpoint for the order-processing-service to use for read-heavy queries, they update both the XRD and the Composition to expose it.
XRDs support multiple versions with conversion, allowing platform teams to evolve their APIs over time without breaking existing consumers. If the platform team decides to rename the size parameter to tier in version v1alpha2, they can define both v1alpha1 and v1alpha2 schemas in the XRD with appropriate defaults and conversion logic. Existing Claims from the inventory-sync team that use v1alpha1 continue to work, while new Claims can use the updated v1alpha2 schema. This versioning strategy mirrors how Kubernetes itself evolves its built-in APIs, moving from alpha to beta to stable, and ensures that platform APIs are treated with the same rigor as any other production API.
Code Example
# CompositeResourceDefinition (XRD) for a PostgreSQL database
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: postgresqlinstances.database.example.org # Must match group+plural
spec:
group: database.example.org # API group for the custom resource
names:
kind: PostgreSQLInstance # Cluster-scoped XR kind name
plural: postgresqlinstances # Plural name for kubectl
claimNames:
kind: PostgreSQLClaim # Namespace-scoped Claim kind name
plural: postgresqlclaims # Plural name for Claims
connectionSecretKeys: # Keys published in the connection Secret
- host # Database hostname
- port # Database port number
- username # Database admin username
- password # Database admin password
versions:
- name: v1alpha1 # Version of this API
served: true # This version is available for use
referenceable: true # Compositions can reference this version
schema:
openAPIV3Schema: # Validation schema for the XR
type: object
properties:
spec:
type: object # Top-level spec object
properties:
parameters:
type: object # Parameters developers can set
required: # These fields are mandatory
- size
- region
properties:
size: # Database size parameter
type: string
enum: [small, medium, large] # Allowed values
description: "Database instance size" # Help text
region: # AWS region parameter
type: string
enum: [us-east-1, us-west-2, eu-west-1] # Allowed regions
storageGB: # Storage parameter with default
type: integer
default: 50 # Default to 50 GB if not specified
version: # PostgreSQL version parameter
type: string
default: "15" # Default PostgreSQL version
# Verify the XRD created both CRDs
# kubectl get crds | grep database.example.org
# Expected: postgresqlinstances.database.example.org
# Expected: postgresqlclaims.database.example.org💡
Interview Tip
A junior engineer typically says an XRD defines a custom resource. The interviewer wants you to explain the XRD as an API contract between platform engineers and application developers. Emphasize three things: the schema section controls what parameters developers can configure with validation and enums acting as guardrails, the claimNames field creates the namespace-scoped Claim that integrates with Kubernetes RBAC for access control, and the connectionSecretKeys field defines the interface for passing credentials back to applications. Mention that XRDs support versioning, which lets platform teams evolve their APIs without breaking existing consumers, just like Kubernetes evolves its own built-in API versions from alpha to beta to stable.
◈ Architecture Diagram
┌─── XRD: postgresqlinstances.database.example.org ──┐ │ │ │ Registers two CRDs: │ │ │ │ ┌─ Cluster-scoped ─────────────────────────────┐ │ │ │ kind: PostgreSQLInstance (XR) │ │ │ │ Used by: platform teams, automation │ │ │ └──────────────────────────────────────────────┘ │ │ │ │ ┌─ Namespace-scoped ───────────────────────────┐ │ │ │ kind: PostgreSQLClaim (XRC) │ │ │ │ Used by: application developers │ │ │ └──────────────────────────────────────────────┘ │ │ │ │ Schema (API Contract): │ │ ┌──────────────────────────────────────────────┐ │ │ │ size: enum [small, medium, large] ✓ │ │ │ │ region: enum [us-east-1, us-west-2] ✓ │ │ │ │ storageGB: integer (default: 50) │ │ │ │ version: string (default: "15") │ │ │ └──────────────────────────────────────────────┘ │ │ │ │ Connection Secret Keys: │ │ ┌──────────────────────────────────────────────┐ │ │ │ host │ port │ username │ password │ │ │ └──────────────────────────────────────────────┘ │ │ │ │ ┌──────────────┐ │ │ │ Compositions │ ← satisfy this XRD │ │ │ aws-prod │ │ │ │ gcp-dev │ │ │ └──────────────┘ │ └──────────────────────────────────────────────────────┘
💬 Comments
⚡
Quick Answer
Crossplane and Terraform both manage cloud infrastructure as code, but they differ fundamentally in architecture. Terraform is a CLI tool that runs plan-and-apply cycles with a state file, while Crossplane runs as a Kubernetes controller with continuous reconciliation. Crossplane integrates natively with the Kubernetes ecosystem, whereas Terraform operates as an external tool with its own state management and workflow.
Detailed Answer
Think of the difference like two approaches to keeping your garden healthy. Terraform is like a gardener who visits once a week, takes a photo of the garden, compares it to the blueprint, makes all the changes needed, and leaves until next week. Between visits, weeds can grow, plants can die, and animals can dig holes without anyone noticing. Crossplane is like an automated sprinkler and monitoring system that runs twenty-four hours a day, continuously sensing soil moisture, detecting weeds, and making adjustments in real time. Both approaches result in a healthy garden, but they handle drift and real-time changes very differently.
The most fundamental architectural difference is the reconciliation model. Terraform uses an imperative workflow: you run terraform plan to see what would change, then terraform apply to make those changes. Between runs, Terraform has no awareness of your infrastructure. If someone modifies an RDS instance through the AWS console, Terraform will not notice until the next plan-and-apply cycle, which might not happen for hours or days. Crossplane controllers run continuously inside Kubernetes, checking the state of every Managed Resource every few minutes. If someone manually changes the security group for the payments-api database, Crossplane detects the drift and reverts it within minutes. This continuous reconciliation means your Git repository remains the true source of truth at all times, not just at the moment of the last apply.
State management is another major differentiator. Terraform maintains a state file that records the mapping between your configuration and real cloud resources. This state file is critical and fragile. If it becomes corrupted, gets out of sync, or is lost, you face a painful recovery process involving terraform import for every resource. Teams have spent entire weekends reconstructing state files after S3 backend issues or accidental deletions. Crossplane stores its state in the Kubernetes API server, backed by etcd, which is the same battle-tested system that stores all your application workloads. There is no separate state file to manage, back up, or worry about. The Managed Resources in the cluster are the state, and etcd handles persistence, replication, and backup through the same mechanisms that protect your Deployments and Services.
The developer experience and access model differ significantly. With Terraform, developers need the Terraform CLI installed, access to the state backend, cloud provider credentials on their workstation, and knowledge of HCL, the HashiCorp Configuration Language. With Crossplane, developers interact through kubectl and YAML, tools they already use daily for application deployments. RBAC controls who can create which resources in which namespaces, and credentials are managed centrally through ProviderConfigs rather than distributed to individual developer laptops. For the order-processing-service team, requesting a database is as simple as creating a Claim in their namespace, the same workflow they use to create a Deployment or ConfigMap.
However, Terraform has significant advantages that should not be dismissed. Its ecosystem is massive and battle-tested, with thousands of providers and modules covering virtually every cloud service and SaaS platform. Terraform's plan output gives operators a clear preview of changes before they are applied, which is a safety mechanism that Crossplane's continuous reconciliation does not natively provide in the same way. Terraform is also simpler to get started with because it does not require a Kubernetes cluster as a prerequisite. For teams without existing Kubernetes expertise, or for managing the Kubernetes clusters themselves, Terraform often makes more sense. Many organizations use both: Terraform for foundational infrastructure like VPCs, Kubernetes clusters, and IAM roles, and Crossplane for application-level infrastructure like databases, caches, and queues that the checkout-service and inventory-sync teams provision through self-service.
The choice between Crossplane and Terraform often comes down to organizational context rather than technical superiority. If your organization is Kubernetes-native, has a platform engineering team, and wants to build an Internal Developer Platform with self-service infrastructure, Crossplane fits naturally into that architecture. If your infrastructure team has deep Terraform expertise, a mature CI/CD pipeline for Terraform runs, and manages infrastructure that spans beyond Kubernetes, continuing with Terraform while evaluating Crossplane for specific use cases is a pragmatic approach.
Code Example
# Terraform approach: Provision an RDS instance
# File: main.tf (requires Terraform CLI and state backend)
resource "aws_db_instance" "payments_db" { # Terraform resource block
identifier = "payments-api-db" # RDS instance identifier
engine = "postgres" # Database engine type
engine_version = "15.4" # PostgreSQL version
instance_class = "db.t3.medium" # Instance size
allocated_storage = 50 # Storage in GB
db_name = "payments" # Default database name
username = "payments_admin" # Master username
password = var.db_password # Password from variable
skip_final_snapshot = false # Take snapshot on deletion
}
# terraform plan # Preview changes (manual step)
# terraform apply # Apply changes (manual step)
# State stored in terraform.tfstate or remote backend
# Crossplane approach: Same RDS instance as a Managed Resource
apiVersion: rds.aws.upbound.io/v1beta1 # Crossplane AWS RDS API
kind: Instance # Crossplane Managed Resource kind
metadata:
name: payments-api-db # Kubernetes resource name
spec:
forProvider:
region: us-east-1 # AWS region
engine: postgres # Database engine type
engineVersion: "15.4" # PostgreSQL version
instanceClass: db.t3.medium # Instance size
allocatedStorage: 50 # Storage in GB
dbName: payments # Default database name
masterUsername: payments_admin # Master username
providerConfigRef:
name: aws-provider-config # Centralized credentials
# kubectl apply -f database.yaml # Apply like any K8s resource
# State stored in Kubernetes API server (etcd)
# Continuous reconciliation detects and corrects drift automatically💡
Interview Tip
A junior engineer typically picks one tool as better than the other. The interviewer wants a nuanced comparison that shows you understand the tradeoffs and can recommend the right tool for the right context. Cover four comparison axes: reconciliation model (continuous versus run-once), state management (etcd versus state files), developer experience (kubectl versus Terraform CLI), and ecosystem maturity (Terraform's massive provider ecosystem versus Crossplane's growing but smaller one). Then give your recommendation based on organizational context: Crossplane for Kubernetes-native platform engineering teams, Terraform for infrastructure teams managing foundational resources or organizations without deep Kubernetes expertise. Many production environments use both tools together, with Terraform managing foundational infrastructure and Crossplane managing application-level resources through self-service Claims.
◈ Architecture Diagram
┌─── Terraform ────────────────────┐ ┌─── Crossplane ──────────────────┐ │ │ │ │ │ Developer Workstation │ │ Kubernetes Cluster │ │ ┌──────────────────┐ │ │ ┌──────────────────┐ │ │ │ terraform plan │ │ │ │ kubectl apply │ │ │ │ terraform apply │ │ │ │ (YAML manifest) │ │ │ └────────┬─────────┘ │ │ └────────┬─────────┘ │ │ │ │ │ │ │ │ ▼ │ │ ▼ │ │ ┌──────────────────┐ │ │ ┌──────────────────┐ │ │ │ State File │ │ │ │ API Server │ │ │ │ (S3 backend) │ │ │ │ (etcd state) │ │ │ │ │ │ │ │ │ │ │ │ ✗ Can get │ │ │ │ ✓ Built-in │ │ │ │ corrupted │ │ │ │ replication │ │ │ └────────┬─────────┘ │ │ └────────┬─────────┘ │ │ │ │ │ │ │ │ ▼ │ │ ▼ │ │ ┌──────────────────┐ │ │ ┌──────────────────┐ │ │ │ Run-once apply │ │ │ │ Continuous │ │ │ │ No drift detect │ │ │ │ Reconciliation │ │ │ │ between runs │ │ │ │ Drift corrected │ │ │ └────────┬─────────┘ │ │ └────────┬─────────┘ │ │ │ │ │ │ │ │ ▼ │ │ ▼ │ │ Cloud APIs │ │ Cloud APIs │ └──────────────────────────────────┘ └──────────────────────────────────┘
💬 Comments
⚡
Quick Answer
Crossplane Compositions define a template of managed resources that are created when a user submits a Composite Resource claim, and patches dynamically map values from the claim into the underlying infrastructure resources, enabling self-service provisioning with guardrails.
Detailed Answer
Crossplane Compositions are the central abstraction that transforms a Kubernetes cluster into a universal control plane for infrastructure. A Composition defines which managed resources should be created and how values from a user-facing Composite Resource (XR) or Claim (XRC) flow into those resources through patches. When a platform team at a company creates a Composition for the payments-api database, they are encoding organizational best practices such as encryption at rest, backup windows, and network isolation into a reusable blueprint that application developers consume without needing cloud console access.
The architecture involves three core components working together. First, the CompositeResourceDefinition (XRD) defines the schema of the custom API that application teams interact with. This is analogous to creating a custom Kubernetes resource type, such as XPostgreSQLInstance, with fields like storageGB, region, and environment. Second, the Composition maps that XRD to a specific set of managed resources. For example, the payments-api team might request an XPostgreSQLInstance, and the Composition creates an RDS instance, a subnet group, a security group, and a parameter group in AWS. Third, patches are the glue that transfers values between the composite resource and the composed resources. The FromCompositeFieldPath patch type copies a field from the XR spec into a managed resource spec, while ToCompositeFieldPath copies status values back up to the XR for consumers to read, such as the database endpoint or connection secret.
Patches support powerful transformations that make dynamic configuration possible. The transform block can apply string formatting using the fmt type, converting a simple environment name like staging into a full resource name like payments-api-db-staging. Math transforms can multiply a user-requested storage value by a factor for different environments. Map transforms convert enumerated values, for instance mapping small to db.t3.micro and large to db.r5.2xlarge for the order-processing-service database. Convert transforms handle type coercion between strings and integers. The combine transform merges multiple fields, such as joining region and environment into a unique identifier for the inventory-sync database subnet group. Each patch also supports a policy block with fromFieldPath set to Required or Optional, controlling whether a missing source field should block provisioning or use a default.
In production environments, Compositions typically create five to fifteen managed resources per claim. The checkout-service database Composition at a typical e-commerce company might provision an RDS instance, a KMS key for encryption, an IAM role for enhanced monitoring, a CloudWatch alarm for CPU utilization, a subnet group spanning three availability zones, and a security group with ingress rules scoped to the application namespace CIDR. Each of these resources receives values from the claim through patches, and the connection details such as host, port, username, and password are propagated back through connectionDetails blocks in the Composition, ultimately stored as a Kubernetes Secret in the claiming namespace where the application pod mounts them.
A critical best practice is versioning Compositions using the revision system. When the platform team updates a Composition, existing composite resources remain on their original revision unless the updatePolicy on the XRD is set to Automatic. This prevents breaking changes from cascading across all environments simultaneously. The user-auth-service team can test the new Composition revision in their dev environment before the platform team rolls it to staging and production. Teams should also use readinessChecks in the Composition to define when a composed resource is considered ready, such as checking that the RDS instance status field equals available, which gates the overall composite resource readiness and prevents applications from attempting to connect to infrastructure that is still provisioning.
Code Example
# CompositeResourceDefinition - defines the custom API schema
apiVersion: apiextensions.crossplane.io/v1 # Crossplane API extensions version
kind: CompositeResourceDefinition # defines a new composite resource type
metadata: # resource metadata
name: xpostgresqlinstances.database.acme.io # must match plural.group format
spec: # XRD specification
group: database.acme.io # API group for the custom resource
names: # naming configuration
kind: XPostgreSQLInstance # composite resource kind
plural: xpostgresqlinstances # plural form for API
claimNames: # claim naming for namespace-scoped access
kind: PostgreSQLInstance # claim kind for app teams
plural: postgresqlinstances # plural form for claims
versions: # API version definitions
- name: v1alpha1 # version name
served: true # serve this version via API
referenceable: true # allow references to this version
schema: # OpenAPI v3 schema for validation
openAPIV3Schema: # schema definition block
type: object # top-level type is object
properties: # field definitions
spec: # spec section
type: object # spec is an object
properties: # spec fields
storageGB: # storage size field
type: integer # integer type
description: Storage size in gigabytes # field documentation
region: # AWS region field
type: string # string type
enum: [us-east-1, us-west-2, eu-west-1] # allowed regions
environment: # environment identifier
type: string # string type
enum: [dev, staging, prod] # allowed environments
required: [storageGB, region, environment] # mandatory fields
---
# Composition - maps XRD to actual AWS resources with patches
apiVersion: apiextensions.crossplane.io/v1 # Crossplane API extensions version
kind: Composition # Composition resource type
metadata: # resource metadata
name: xpostgresql-aws # composition name
labels: # labels for selection
provider: aws # provider label for matching
spec: # composition specification
compositeTypeRef: # reference to the XRD
apiVersion: database.acme.io/v1alpha1 # XRD API version
kind: XPostgreSQLInstance # XRD kind to compose
resources: # list of composed resources
- name: rds-instance # logical name for the RDS resource
base: # base resource template
apiVersion: rds.aws.upbound.io/v1beta1 # AWS RDS provider API
kind: Instance # RDS Instance kind
spec: # resource spec
forProvider: # provider-specific config
engine: postgres # PostgreSQL engine
engineVersion: "15" # Postgres version 15
instanceClass: db.t3.medium # default instance class
allocatedStorage: 20 # default 20 GB storage
publiclyAccessible: false # no public access
skipFinalSnapshot: false # require final snapshot
patches: # patches from composite to resource
- type: FromCompositeFieldPath # copy value from XR to resource
fromFieldPath: spec.storageGB # source field on the XR
toFieldPath: spec.forProvider.allocatedStorage # target field on RDS instance
- type: FromCompositeFieldPath # copy region from XR
fromFieldPath: spec.region # source region field
toFieldPath: spec.forProvider.region # target region on RDS
- type: FromCompositeFieldPath # dynamic instance naming
fromFieldPath: spec.environment # source environment field
toFieldPath: metadata.annotations[crossplane.io/external-name] # external name annotation
transforms: # apply transformations
- type: string # string transform type
string: # string transform config
type: Format # format transform
fmt: "payments-api-db-%s" # format pattern with environment
- type: FromCompositeFieldPath # map environment to instance size
fromFieldPath: spec.environment # source environment
toFieldPath: spec.forProvider.instanceClass # target instance class
transforms: # transformation pipeline
- type: map # map transform type
map: # mapping dictionary
dev: db.t3.micro # dev gets small instance
staging: db.t3.medium # staging gets medium instance
prod: db.r5.xlarge # prod gets large instance
connectionDetails: # propagate connection info
- type: FromConnectionSecretKey # copy from resource secret
fromConnectionSecretKey: endpoint # source key name
name: host # target key in composite secret
- type: FromConnectionSecretKey # copy password from secret
fromConnectionSecretKey: password # source password key
name: password # target key name
---
# Claim - what application developers submit
apiVersion: database.acme.io/v1alpha1 # custom API version from XRD
kind: PostgreSQLInstance # claim kind defined in XRD
metadata: # resource metadata
name: payments-api-db # claim name
namespace: payments # application namespace
spec: # claim specification
storageGB: 100 # request 100 GB storage
region: us-east-1 # deploy in us-east-1
environment: prod # production environment
writeConnectionSecretToRef: # where to store credentials
name: payments-api-db-conn # secret name for the app to use💡
Interview Tip
A junior engineer typically describes Compositions as simple templates, but interviewers want to see that you understand the full patch pipeline including transforms, policies, and connection detail propagation. Walk through a concrete example where a single claim creates multiple resources like an RDS instance, a security group, and a subnet group, and explain how each patch maps user intent to provider-specific configuration. Mention the difference between FromCompositeFieldPath and ToCompositeFieldPath directions and when each is used. Discuss how map transforms enforce organizational guardrails by converting abstract sizes like small and large into specific instance classes. If you can explain Composition revisions and how they protect running infrastructure from breaking changes during updates, you demonstrate production-level platform engineering knowledge.
◈ Architecture Diagram
┌─────────────────────────────────────────────────────────┐
│ Application Developer │
│ kubectl apply -f postgresql-claim.yaml │
└────────────────────────┬────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────┐
│ PostgreSQLInstance (Claim) │
│ namespace: payments │
│ spec: storageGB=100, region=us-east-1, env=prod │
└────────────────────────┬────────────────────────────────┘
│ XRD validates schema
↓
┌─────────────────────────────────────────────────────────┐
│ XPostgreSQLInstance (Composite Resource) │
│ cluster-scoped, owned by claim │
└────────────────────────┬────────────────────────────────┘
│ Composition selects resources
↓
┌─────────────────────────────────────────────────────────┐
│ Composition: xpostgresql-aws │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ RDS Instance │ │ Subnet Group │ │ Security Grp │ │
│ │ ● patches │ │ ● patches │ │ ● patches │ │
│ │ storageGB→ │ │ region→ │ │ env→cidr │ │
│ │ allocStorage│ │ subnetIds │ │ ingress │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
└─────────┼─────────────────┼─────────────────┼───────────┘
│ │ │
↓ ↓ ↓
┌─────────────────────────────────────────────────────────┐
│ AWS Cloud Account │
│ ● RDS: payments-api-db-prod (db.r5.xlarge, 100GB) │
│ ● Subnet Group: 3 AZs in us-east-1 │
│ ● Security Group: scoped ingress rules │
└─────────────────────────────────────────────────────────┘💬 Comments
⚡
Quick Answer
Crossplane ProviderConfigs define the credentials and configuration for authenticating to cloud providers, and you create multiple ProviderConfig resources to target different AWS accounts, GCP projects, or Azure subscriptions, then reference the appropriate ProviderConfig by name in each managed resource.
Detailed Answer
In enterprise environments, infrastructure rarely lives in a single cloud account or region. The payments-api may run in a PCI-compliant AWS account, the user-auth-service may live in a separate identity account, and the order-processing-service may span multiple regions for disaster recovery. Crossplane ProviderConfigs are the mechanism that maps each managed resource to the correct cloud account and credentials, enabling a single Crossplane control plane to manage infrastructure across dozens of accounts and regions from one Kubernetes cluster.
A ProviderConfig is a cluster-scoped custom resource that each Crossplane provider installs. For the AWS provider, the ProviderConfig specifies how to authenticate, whether through static credentials stored in a Kubernetes Secret, IAM Roles for Service Accounts (IRSA) when running on EKS, or AssumeRole chains for cross-account access. The most production-ready pattern is IRSA combined with AssumeRole. The Crossplane pod runs with a base IAM role that has permission only to assume other roles, and each ProviderConfig specifies a target role ARN in a different AWS account. When the checkout-service team requests a DynamoDB table in the commerce account, the managed resource references providerConfigRef with the name aws-commerce-account, and the provider assumes the appropriate role to create resources in that account.
For multi-region setups, the approach depends on whether the provider supports a region field directly on the managed resource or requires separate ProviderConfigs. The AWS Upbound provider allows setting spec.forProvider.region on each managed resource, so a single ProviderConfig can manage resources in any region. However, some organizations prefer creating region-specific ProviderConfigs like aws-prod-us-east-1 and aws-prod-eu-west-1 for additional isolation and to enforce regional guardrails through RBAC. The inventory-sync service might need DynamoDB global tables spanning three regions, and the Composition creates three managed resources each referencing a different region-specific ProviderConfig. GCP providers work similarly with project-scoped ProviderConfigs, where each config points to a different GCP project and uses Workload Identity Federation for keyless authentication from a GKE-hosted control plane.
Credential management is the most security-critical aspect of ProviderConfigs. Static credentials stored in Kubernetes Secrets are the simplest approach but create rotation burdens and blast-radius risks. If a Secret containing AWS access keys for the production account is compromised, the attacker gains full infrastructure access. IRSA eliminates long-lived credentials entirely by exchanging Kubernetes service account tokens for temporary AWS STS credentials. For cross-account scenarios, the ProviderConfig specifies an assumeRoleChain where the base role assumes an intermediate role in a networking account, which then assumes a final role in the target account. This chain pattern is common in organizations using AWS Control Tower or Landing Zone architectures where the checkout-service infrastructure spans a shared-services account for networking and a workload account for compute resources.
Operational best practices include naming ProviderConfigs with a consistent convention like provider-environment-account-region, applying labels for discoverability, and using Crossplane RBAC to restrict which teams can reference which ProviderConfigs. A namespace-scoped Claim should not be able to provision resources in the production payments account by simply changing a providerConfigRef name. Platform teams enforce this through Composition-level providerConfigRef settings that are not patchable from the claim, or through admission webhooks and OPA Gatekeeper policies that validate providerConfigRef values against team membership. Monitoring ProviderConfig health is also essential because a credential rotation that invalidates a ProviderConfig silently breaks all reconciliation for resources using that config, causing drift to go undetected until the user-auth-service database fails to apply a critical security patch.
Code Example
# ProviderConfig for production AWS account using IRSA and AssumeRole
apiVersion: aws.upbound.io/v1beta1 # AWS provider API version
kind: ProviderConfig # ProviderConfig resource type
metadata: # resource metadata
name: aws-prod-payments # name for the payments prod account
spec: # provider config specification
credentials: # authentication configuration
source: IRSA # use IAM Roles for Service Accounts
assumeRoleChain: # chain of roles to assume
- roleARN: arn:aws:iam::111111111111:role/crossplane-payments-prod # target role in payments account
externalID: crossplane-prod-external # external ID for security
---
# ProviderConfig for staging AWS account
apiVersion: aws.upbound.io/v1beta1 # AWS provider API version
kind: ProviderConfig # ProviderConfig resource type
metadata: # resource metadata
name: aws-staging-shared # name for staging shared account
spec: # provider config specification
credentials: # authentication configuration
source: IRSA # use IRSA for keyless auth
assumeRoleChain: # role assumption chain
- roleARN: arn:aws:iam::222222222222:role/crossplane-staging # staging account role
---
# ProviderConfig for GCP production project
apiVersion: gcp.upbound.io/v1beta1 # GCP provider API version
kind: ProviderConfig # ProviderConfig resource type
metadata: # resource metadata
name: gcp-prod-analytics # name for GCP analytics project
spec: # provider config specification
projectID: acme-analytics-prod-42 # target GCP project ID
credentials: # authentication configuration
source: InjectedIdentity # use Workload Identity Federation
---
# Composition using ProviderConfig references per environment
apiVersion: apiextensions.crossplane.io/v1 # Crossplane API extensions version
kind: Composition # Composition resource type
metadata: # resource metadata
name: xdynamodb-multi-account # composition for multi-account DynamoDB
spec: # composition specification
compositeTypeRef: # reference to the XRD
apiVersion: cache.acme.io/v1alpha1 # custom API group
kind: XDynamoDBTable # composite resource kind
resources: # composed resources list
- name: dynamodb-table # logical resource name
base: # base resource definition
apiVersion: dynamodb.aws.upbound.io/v1beta1 # DynamoDB provider API
kind: Table # DynamoDB Table kind
spec: # resource specification
forProvider: # provider configuration
region: us-east-1 # default region
hashKey: id # partition key name
attribute: # attribute definitions
- name: id # attribute name
type: S # string type attribute
billingMode: PAY_PER_REQUEST # on-demand billing
patches: # patches for dynamic config
- type: FromCompositeFieldPath # map environment to ProviderConfig
fromFieldPath: spec.environment # source environment field
toFieldPath: spec.providerConfigRef.name # target providerConfigRef
transforms: # transform environment to config name
- type: map # map transform
map: # mapping dictionary
dev: aws-staging-shared # dev uses staging account
staging: aws-staging-shared # staging uses staging account
prod: aws-prod-payments # prod uses production account
---
# Verify ProviderConfig health and managed resources
kubectl get providerconfigs.aws.upbound.io # list all AWS ProviderConfigs
kubectl get managed -l crossplane.io/provider-config=aws-prod-payments # list resources using prod config
kubectl describe providerconfig aws-prod-payments # check config status and conditions💡
Interview Tip
A junior engineer typically shows a single ProviderConfig with static credentials, which signals lack of production experience. Interviewers want to hear about IRSA or Workload Identity Federation for keyless authentication, and AssumeRole chains for cross-account access. Explain the security implications of static credentials versus federated identity, and describe how a compromised Secret could grant full infrastructure access. Walk through a real scenario where the payments-api needs resources in a PCI-compliant account while the analytics pipeline uses a separate data account, and show how Composition patches dynamically select the correct ProviderConfig based on the claim's environment field. Mention RBAC restrictions that prevent teams from referencing ProviderConfigs they should not have access to, and discuss how credential rotation failures silently break reconciliation.
◈ Architecture Diagram
┌─────────────────────────────────────────────────────────┐
│ Crossplane Control Plane (EKS Cluster) │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ AWS Provider │ │ GCP Provider │ │
│ │ Pod (IRSA) │ │ Pod (WI Fed) │ │
│ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │
│ ┌────────┴────────────────────┴────────┐ │
│ │ ProviderConfigs │ │
│ │ │ │
│ │ ┌──────────────────────────────┐ │ │
│ │ │ aws-prod-payments │ │ │
│ │ │ AssumeRole → 111111111111 │ │ │
│ │ └──────────────┬───────────────┘ │ │
│ │ ┌──────────────────────────────┐ │ │
│ │ │ aws-staging-shared │ │ │
│ │ │ AssumeRole → 222222222222 │ │ │
│ │ └──────────────┬───────────────┘ │ │
│ │ ┌──────────────────────────────┐ │ │
│ │ │ gcp-prod-analytics │ │ │
│ │ │ WI Federation → project-42 │ │ │
│ │ └──────────────┬───────────────┘ │ │
│ └─────────────────┼───────────────┘ │ │
└────────────────────┼────────────────────┘ │
│
┌────────────────┼────────────────────┐
│ ↓ │
│ ┌──────────────────────────────┐ │
│ │ AWS Account: 111111111111 │ │
│ │ ● RDS: payments-api-db-prod │ │
│ │ ● S3: payments-receipts │ │
│ └──────────────────────────────┘ │
│ ┌──────────────────────────────┐ │
│ │ AWS Account: 222222222222 │ │
│ │ ● DynamoDB: order-cache-stg │ │
│ │ ● SQS: checkout-queue-dev │ │
│ └──────────────────────────────┘ │
│ ┌──────────────────────────────┐ │
│ │ GCP Project: analytics-42 │ │
│ │ ● BigQuery: user-events │ │
│ │ ● Pub/Sub: inventory-sync │ │
│ └──────────────────────────────┘ │
└─────────────────────────────────────┘💬 Comments
⚡
Quick Answer
Crossplane uses a continuous reconciliation loop where each managed resource controller periodically compares the desired state in the Kubernetes spec with the actual state in the cloud provider, and automatically corrects any drift by updating the external resource to match the declared configuration.
Detailed Answer
Crossplane's reconciliation engine is built on the Kubernetes controller pattern, where every managed resource has a dedicated controller running a continuous observe-compare-act loop. When the platform team defines an RDS instance for the payments-api, the AWS provider controller watches for that resource, calls the cloud API to observe the current state, compares it against the desired spec, and takes corrective action if differences exist. This loop runs on a configurable interval, defaulting to one minute for most providers, and ensures that infrastructure configuration remains consistent with what is declared in the Kubernetes API server regardless of manual changes, automated policies, or external drift.
The reconciliation loop has four distinct phases that execute in sequence. The Observe phase calls the cloud provider API to read the current state of the external resource, such as querying the AWS DescribeDBInstances API for the payments-api RDS instance. The controller maps the cloud API response back into the Crossplane resource status fields, populating conditions like Ready, Synced, and LastAsyncOperation. The Compare phase checks whether the observed external state matches the desired spec fields. The provider performs a field-by-field diff, identifying properties that have diverged such as an instance class that was manually changed from db.r5.xlarge to db.r5.2xlarge through the AWS console. The Act phase issues update API calls to bring the external resource back into compliance. For the RDS instance, this might call ModifyDBInstance to revert the instance class. Finally, the Status Update phase writes the reconciliation result back to the managed resource status, setting the Synced condition to True if everything matches or False with a descriptive message if reconciliation failed.
Drift detection is automatic and continuous, but teams need to understand its nuances to operate Crossplane effectively. Not all fields on a managed resource are reconciled equally. Some fields are immutable after creation, meaning Crossplane detects drift but cannot correct it without destroying and recreating the resource. For example, changing the engine field on an RDS instance from postgres to mysql would require recreation. The checkout-service team might see a Synced=False condition with a message indicating the field cannot be updated in place. Late-initialized fields present another subtlety. Many cloud resources have properties that the provider sets at creation time and then writes back into the Crossplane spec through a process called late initialization. For instance, the VPC ID or the assigned availability zone for an order-processing-service ElastiCache cluster may not be specified in the claim but gets populated after creation. These late-initialized fields become part of the desired state going forward.
Tuning reconciliation behavior is essential for production stability and cost management. The pollInterval on the provider controls how frequently the observe loop runs. Setting it too low generates excessive API calls that can trigger rate limiting on the cloud provider, especially when managing hundreds of resources for services like inventory-sync and user-auth-service across multiple accounts. Setting it too high delays drift detection, allowing manual changes to persist longer. Most production teams settle on intervals between one and five minutes as a balance. The managementPolicy field on managed resources controls whether Crossplane should only observe without taking corrective action (ObserveOnly), create and observe but never update or delete (CreateAndObserve), or fully manage the lifecycle (Default). ObserveOnly mode is particularly useful for importing existing infrastructure that the checkout-service team created manually before adopting Crossplane, allowing visibility without risking unintended changes.
Monitoring drift events is critical for understanding operational patterns. Teams should configure alerts on the Synced condition transitioning to False, which indicates that either drift was detected and correction failed, or that the cloud API returned an error. Crossplane emits Kubernetes events on managed resources during reconciliation, and these events can be forwarded to observability platforms through tools like kube-state-metrics or Falco. A dashboard showing drift frequency per resource type helps platform teams identify systemic issues, such as an automation script in the payments-api account that repeatedly modifies security group rules, triggering constant reconciliation cycles. The events also provide timestamps that help correlate infrastructure changes with application incidents during post-mortems.
Code Example
# Managed resource with reconciliation annotations
apiVersion: rds.aws.upbound.io/v1beta1 # AWS RDS provider API
kind: Instance # RDS Instance resource type
metadata: # resource metadata
name: payments-api-db-prod # resource name
annotations: # annotations for behavior tuning
crossplane.io/external-name: payments-api-db-prod # maps to actual AWS resource name
spec: # desired state specification
forProvider: # provider-specific configuration
engine: postgres # PostgreSQL database engine
engineVersion: "15.4" # specific engine version
instanceClass: db.r5.xlarge # instance size for production
allocatedStorage: 200 # 200 GB storage allocation
region: us-east-1 # AWS region
publiclyAccessible: false # deny public access
storageEncrypted: true # enable encryption at rest
providerConfigRef: # which ProviderConfig to use
name: aws-prod-payments # reference to prod ProviderConfig
managementPolicies: # control reconciliation behavior
- "*" # full lifecycle management (default)
---
# ObserveOnly resource for importing existing infrastructure
apiVersion: ec2.aws.upbound.io/v1beta1 # AWS EC2 provider API
kind: VPC # VPC resource type
metadata: # resource metadata
name: legacy-checkout-vpc # name for the imported VPC
annotations: # annotations
crossplane.io/external-name: vpc-0abc123def456 # existing VPC ID in AWS
spec: # resource specification
forProvider: # provider configuration
region: us-east-1 # region where VPC exists
cidrBlock: 10.0.0.0/16 # VPC CIDR block
managementPolicies: # restrict management scope
- Observe # only observe, never modify or delete
providerConfigRef: # ProviderConfig reference
name: aws-prod-payments # production account config
---
# Check reconciliation status and drift detection
kubectl get managed -o custom-columns=NAME:.metadata.name,SYNCED:.status.conditions[?(@.type=="Synced")].status,READY:.status.conditions[?(@.type=="Ready")].status # list all managed resources with sync status
# Describe a resource to see detailed reconciliation events
kubectl describe instance.rds.aws.upbound.io payments-api-db-prod # show events and conditions for the RDS instance
# Watch for drift events in real time across all resources
kubectl get events --field-selector reason=ReconcileSuccess -w # stream successful reconciliation events
kubectl get events --field-selector reason=ReconcileError -w # stream failed reconciliation events
# Check provider pod logs for detailed reconciliation traces
kubectl logs -n crossplane-system -l pkg.crossplane.io/revision=provider-aws --tail=50 # tail AWS provider logs for reconciliation details
# Trigger immediate reconciliation by annotating the resource
kubectl annotate instance.rds.aws.upbound.io payments-api-db-prod crossplane.io/reconcile=now --overwrite # force immediate reconciliation cycle💡
Interview Tip
A junior engineer typically says Crossplane detects drift without explaining the observe-compare-act loop or the nuances of late initialization and immutable fields. Show the interviewer you understand that not all drift can be corrected in place by giving an example of an immutable field change that requires resource recreation. Discuss the managementPolicies field and when ObserveOnly mode is appropriate, such as importing a legacy VPC that the checkout-service relies on. Explain how pollInterval tuning affects API rate limits when managing hundreds of resources, and describe a monitoring strategy using Synced condition alerts and Kubernetes events to detect persistent drift. If you mention that late-initialized fields become part of desired state and can cause unexpected diffs during Composition updates, you demonstrate deep operational experience with Crossplane.
◈ Architecture Diagram
┌─────────────────────────────────────────────────────────┐
│ Crossplane Reconciliation Loop │
└────────────────────────┬────────────────────────────────┘
│
┌──────────────┴──────────────┐
↓ │
┌──────────────────┐ │
│ 1. OBSERVE │ │
│ Call AWS API │ │
│ DescribeDB │ │
│ Instance │ │
└────────┬─────────┘ │
↓ │
┌──────────────────┐ │
│ 2. COMPARE │ │
│ Desired State │ │
│ vs Live State │ │
│ │ │
│ instanceClass: │ │
│ spec: r5.xlarge │ │
│ live: r5.2xlarge│ ← manual drift │
│ ✗ MISMATCH │ │
└────────┬─────────┘ │
↓ │
┌──────────────────┐ │
│ 3. ACT │ │
│ ModifyDB │ │
│ Instance API │ │
│ Revert to │ │
│ r5.xlarge │ │
│ ✓ CORRECTED │ │
└────────┬─────────┘ │
↓ │
┌──────────────────┐ │
│ 4. STATUS │ │
│ Update K8s │ │
│ Synced: True │ │
│ Ready: True │ │
│ Emit Event │ │
└────────┬─────────┘ │
│ pollInterval (60s) │
└──────────────────────────────┘💬 Comments
⚡
Quick Answer
Crossplane Composition Functions are containerized programs that run during the Composition pipeline to generate, transform, or validate composed resources using languages like KCL or Go templates, replacing complex patch arrays with expressive programming logic for advanced infrastructure composition scenarios.
Detailed Answer
Crossplane Composition Functions represent a paradigm shift from the declarative patch-based Composition model to a programmable pipeline architecture. While patches handle simple field mapping well, real-world infrastructure for services like the payments-api or order-processing-service often requires conditional logic, loops, string manipulation, and data lookups that become unwieldy with dozens of nested patch transforms. Composition Functions solve this by letting platform teams write actual programs that receive the composite resource as input and return a list of desired composed resources as output, running as gRPC servers inside containers that the Crossplane runtime calls during each reconciliation cycle.
The Composition Function pipeline is defined in the Composition spec under the pipeline field, replacing the traditional resources array. Each step in the pipeline references a Function custom resource that points to a container image. The pipeline executes sequentially, with each function receiving the output of the previous step plus the original composite resource. The function-kcl step might generate the base resources, then a function-auto-ready step automatically sets readiness checks, and finally a function-cel-filter step validates that the output meets organizational policies. This pipeline architecture is composable itself, meaning platform teams can mix and match functions from the Crossplane community with custom functions built in-house.
KCL (Kusion Configuration Language) is one of the most popular function languages for Crossplane because it is purpose-built for configuration with strong typing, schema validation, and a familiar Python-like syntax. A KCL function for the checkout-service infrastructure can define a schema that validates input parameters, use conditional expressions to set different resource configurations per environment, iterate over a list of regions to create multi-region resources, and compute derived values like CIDR ranges from a base network address. The function-kcl runtime accepts KCL source code directly in the Composition through an inline source field or references external KCL packages from an OCI registry. When the inventory-sync team submits a claim requesting a multi-region cache, the KCL function generates ElastiCache clusters in each requested region, creates the replication group linking them, and sets up the appropriate security groups with computed CIDR rules.
Go template functions use the familiar Go text/template syntax that Helm users already know, making them an accessible entry point for teams transitioning from Helm-based infrastructure to Crossplane. The function-go-templating runtime renders Go templates with the composite resource available as a data context, supporting Sprig functions for string manipulation, math operations, and encoding. However, Go templates lack type safety and can produce subtle YAML formatting errors, so they are better suited for simpler compositions where KCL or a custom Go function would be overengineered. For the user-auth-service team, a Go template function might generate a standard set of IAM roles and policies based on the service name and environment, using range to iterate over a list of AWS actions and if conditions to add extra permissions for production environments.
Custom Go functions provide the most flexibility when KCL or Go templates are insufficient. Teams write a Go program implementing the Crossplane function SDK's RunFunction interface, build it into a container image, and reference it as a Function resource. The payments-api platform team might write a custom function that queries an internal service catalog API to look up compliance requirements, generates resources based on those requirements, and injects organization-specific annotations and labels. The function SDK provides helper methods for reading the composite resource, building desired resources, setting conditions, and returning results. Testing is done using the crossplane beta render command, which simulates the function pipeline locally and outputs the rendered resources without connecting to any cloud provider, enabling fast iteration during development.
Code Example
# Function resource - registers the KCL function runtime
apiVersion: pkg.crossplane.io/v1beta1 # Crossplane package API
kind: Function # Function resource type
metadata: # resource metadata
name: function-kcl # function name
spec: # function specification
package: xpkg.upbound.io/crossplane-contrib/function-kcl:v0.9.0 # OCI image reference
---
# Function resource - registers Go templating runtime
apiVersion: pkg.crossplane.io/v1beta1 # Crossplane package API
kind: Function # Function resource type
metadata: # resource metadata
name: function-go-templating # function name
spec: # function specification
package: xpkg.upbound.io/crossplane-contrib/function-go-templating:v0.7.0 # OCI image for Go templates
---
# Composition using KCL function pipeline
apiVersion: apiextensions.crossplane.io/v1 # Crossplane API extensions
kind: Composition # Composition resource type
metadata: # resource metadata
name: xdatabase-kcl # composition name
spec: # composition spec
compositeTypeRef: # reference to the XRD
apiVersion: database.acme.io/v1alpha1 # custom API version
kind: XDatabase # composite resource kind
mode: Pipeline # use pipeline mode instead of resources
pipeline: # pipeline steps definition
- step: generate-resources # first pipeline step name
functionRef: # reference to the function
name: function-kcl # use KCL function
input: # input to the function
apiVersion: krm.kcl.dev/v1alpha1 # KCL input API version
kind: KCLInput # KCL input kind
spec: # KCL spec section
source: | # inline KCL source code
# Import the Crossplane KCL SDK for resource generation
import crossplane.v1beta1 as xp
# Read composite resource fields from the XR
oxr = option("params").oxr
env = oxr.spec.environment # get environment from claim
region = oxr.spec.region # get region from claim
service_name = oxr.spec.serviceName # get service name from claim
# Map environment to instance sizing
_sizing = { # define sizing map
"dev": {"class": "db.t3.micro", "storage": 20} # dev sizing
"staging": {"class": "db.t3.medium", "storage": 50} # staging sizing
"prod": {"class": "db.r5.xlarge", "storage": 200} # production sizing
}
# Generate RDS instance with computed values
_rds = { # define RDS resource
apiVersion: "rds.aws.upbound.io/v1beta1" # AWS RDS API
kind: "Instance" # RDS Instance kind
metadata.name: "{}-db-{}".format(service_name, env) # computed name
metadata.annotations: { # resource annotations
"crossplane.io/external-name": "{}-db-{}".format(service_name, env) # external name
}
spec.forProvider: { # provider config block
engine: "postgres" # PostgreSQL engine
engineVersion: "15" # Postgres 15
instanceClass: _sizing[env].class # size from mapping
allocatedStorage: _sizing[env].storage # storage from mapping
region: region # region from claim
}
}
# Generate security group conditionally
_sg = { # define security group resource
apiVersion: "ec2.aws.upbound.io/v1beta1" # AWS EC2 API
kind: "SecurityGroup" # SecurityGroup kind
metadata.name: "{}-db-sg-{}".format(service_name, env) # computed SG name
spec.forProvider: { # provider config
region: region # same region as RDS
description: "DB access for {}".format(service_name) # dynamic description
}
}
# Return composed resources to the pipeline
items = [_rds, _sg] # output list of resources
- step: auto-ready # second pipeline step
functionRef: # reference to ready function
name: function-auto-ready # auto-set readiness checks
---
# Test the pipeline locally with crossplane beta render
crossplane beta render claim.yaml composition.yaml functions.yaml # render composed resources locally without cloud access💡
Interview Tip
A junior engineer typically only knows the patch-based Composition model and may not be aware that Composition Functions exist. Demonstrate awareness of when patches become insufficient, such as when you need loops to generate resources across multiple regions, conditional logic based on complex rules, or data lookups from external sources. Explain the pipeline execution model where each function receives the output of the previous step, and compare KCL versus Go templates in terms of type safety, debugging experience, and team familiarity. Mention the crossplane beta render command for local testing without cloud access, which shows you understand the development workflow. If you can describe writing a custom Go function using the function SDK for a scenario that KCL cannot handle, such as querying an external API during composition, you demonstrate advanced platform engineering skills.
◈ Architecture Diagram
┌─────────────────────────────────────────────────────────┐
│ Composition Pipeline Mode │
└────────────────────────┬────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────┐
│ Composite Resource (XR) │
│ spec: │
│ serviceName: payments-api │
│ environment: prod │
│ region: us-east-1 │
└────────────────────────┬────────────────────────────────┘
│
┌──────────────┴──────────────┐
↓ ↓
┌──────────────────┐ ┌──────────────────┐
│ Step 1: │ │ Step 2: │
│ function-kcl │────→ │ function-auto │
│ │ │ -ready │
│ ● Read XR spec │ │ │
│ ● Map env→size │ │ ● Set readiness │
│ ● Generate RDS │ │ checks on all │
│ ● Generate SG │ │ resources │
│ ● Compute names │ │ │
└────────┬─────────┘ └────────┬─────────┘
│ │
└──────────────┬──────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Composed Resources Output │
│ │
│ ┌──────────────────────────────────────────┐ │
│ │ RDS Instance: payments-api-db-prod │ │
│ │ instanceClass: db.r5.xlarge │ │
│ │ allocatedStorage: 200 │ │
│ │ ✓ readinessCheck configured │ │
│ └──────────────────────────────────────────┘ │
│ ┌──────────────────────────────────────────┐ │
│ │ SecurityGroup: payments-api-db-sg-prod │ │
│ │ description: DB access for payments-api│ │
│ │ ✓ readinessCheck configured │ │
│ └──────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘💬 Comments
⚡
Quick Answer
Environment-specific infrastructure in Crossplane is implemented through a combination of Composition patches that map environment labels to different resource configurations, separate ProviderConfigs for account isolation, and Composition selection labels that route claims to environment-appropriate Compositions with distinct sizing, redundancy, and security settings.
Detailed Answer
Implementing environment-specific infrastructure is one of the most common patterns in Crossplane and touches nearly every aspect of the platform architecture. The goal is to allow the payments-api team to submit the same claim shape in dev, staging, and production, with the underlying infrastructure automatically adapting to each environment's requirements for sizing, redundancy, security, networking, and cost optimization. This pattern eliminates the need for separate Terraform workspaces or CloudFormation stacks per environment, centralizing environment logic in Compositions that the platform team maintains.
The simplest approach uses patch transforms within a single Composition to vary resource properties based on an environment field in the claim. When the checkout-service team sets environment: dev in their database claim, a map transform converts that to db.t3.micro for the instance class, 20 for allocated storage, and false for multi-AZ deployment. The same claim with environment: prod maps to db.r5.xlarge, 200 GB storage, and true for multi-AZ. This approach keeps everything in one Composition file, making it easy to see all environment variations at a glance. String format transforms can generate environment-specific resource names like checkout-service-db-dev and checkout-service-db-prod, preventing naming collisions when dev and staging share an AWS account. Math transforms can apply multipliers, such as giving production three times the IOPS of staging by multiplying a base value by an environment-specific factor.
For organizations with more complex requirements, separate Compositions per environment provide stronger isolation. The XRD defines a compositionSelector field, and three Compositions are created with labels like environment: dev, environment: staging, and environment: prod. When the order-processing-service team submits a claim with compositionSelector matching environment: prod, Crossplane routes it to the production Composition that includes additional resources like CloudWatch alarms, SNS topics for alerting, cross-region read replicas, and automated backup configurations that the dev Composition omits entirely. This pattern gives platform teams fine-grained control over what exists in each environment without relying on conditional patches, and makes it easier to audit the production configuration separately from dev. The tradeoff is maintaining multiple Compositions that can drift from each other, so teams often use a shared library Composition that all environment Compositions reference through Composition Functions.
Account and network isolation between environments is handled through ProviderConfig references in the Composition. The dev and staging environments might share an AWS sandbox account with relaxed IAM policies and cost alerts, while production uses a dedicated account with strict SCPs, GuardDuty enabled, and CloudTrail logging to a separate audit account. The Composition patches the providerConfigRef based on the environment field, routing dev claims to aws-sandbox and prod claims to aws-prod-payments. Network isolation follows the same pattern, where dev resources go into a shared VPC with liberal security groups and prod resources go into a dedicated VPC with private subnets, NAT gateways, and VPC endpoints for AWS service access without traversing the public internet. The inventory-sync service in production might additionally get VPC peering to a data warehouse VPC that does not exist in dev.
Operational practices for environment promotion deserve careful consideration. Teams should never copy claims between environments by changing the environment field manually. Instead, a GitOps workflow stores claims in a Git repository with environment-specific directories or overlays. The user-auth-service team maintains a base claim definition and environment-specific patches using Kustomize overlays, where the dev overlay sets environment: dev with 10 GB storage and the prod overlay sets environment: prod with 500 GB storage and enables deletion protection. ArgoCD or Flux deploys each overlay to the appropriate cluster or namespace, and promotion from staging to production is a pull request that updates the prod overlay values. This workflow provides audit trails, peer review for production changes, and the ability to roll back infrastructure changes through git revert, ensuring that environment-specific configurations are tracked and reproducible.
Code Example
# XRD with compositionSelector support for environment routing
apiVersion: apiextensions.crossplane.io/v1 # Crossplane API extensions version
kind: CompositeResourceDefinition # XRD resource type
metadata: # resource metadata
name: xappinfra.platform.acme.io # XRD name in plural.group format
spec: # XRD specification
group: platform.acme.io # custom API group
names: # resource naming
kind: XAppInfra # composite resource kind
plural: xappinfras # plural form
claimNames: # namespace-scoped claim names
kind: AppInfra # claim kind for app teams
plural: appinfras # plural form
defaultCompositionRef: # fallback composition
name: xappinfra-dev # default to dev composition
versions: # API versions
- name: v1alpha1 # version name
served: true # serve via API
referenceable: true # allow references
schema: # validation schema
openAPIV3Schema: # OpenAPI v3 schema
type: object # top-level type
properties: # field definitions
spec: # spec section
type: object # spec is object
properties: # spec fields
serviceName: # application service name
type: string # string type
environment: # target environment
type: string # string type
enum: [dev, staging, prod] # allowed values
storageGB: # storage request
type: integer # integer type
required: [serviceName, environment] # required fields
---
# Dev Composition - minimal resources, shared account
apiVersion: apiextensions.crossplane.io/v1 # Crossplane API extensions
kind: Composition # Composition resource
metadata: # resource metadata
name: xappinfra-dev # dev composition name
labels: # labels for selection
environment: dev # environment label
spec: # composition spec
compositeTypeRef: # XRD reference
apiVersion: platform.acme.io/v1alpha1 # custom API version
kind: XAppInfra # composite kind
resources: # composed resources
- name: database # database resource
base: # base template
apiVersion: rds.aws.upbound.io/v1beta1 # AWS RDS API
kind: Instance # RDS Instance
spec: # resource spec
forProvider: # provider config
engine: postgres # PostgreSQL
engineVersion: "15" # version 15
instanceClass: db.t3.micro # small dev instance
allocatedStorage: 20 # minimal storage
multiAZ: false # single AZ for dev
skipFinalSnapshot: true # skip snapshot on delete
deletionProtection: false # allow easy teardown
region: us-east-1 # dev region
providerConfigRef: # use sandbox account
name: aws-sandbox # sandbox ProviderConfig
patches: # dynamic patches
- type: FromCompositeFieldPath # map service name to DB identifier
fromFieldPath: spec.serviceName # source service name
toFieldPath: metadata.annotations[crossplane.io/external-name] # external name
transforms: # name transformation
- type: string # string transform
string: # string config
type: Format # format type
fmt: "%s-db-dev" # append -db-dev suffix
---
# Prod Composition - full resources, dedicated account, HA
apiVersion: apiextensions.crossplane.io/v1 # Crossplane API extensions
kind: Composition # Composition resource
metadata: # resource metadata
name: xappinfra-prod # prod composition name
labels: # labels for selection
environment: prod # environment label
spec: # composition spec
compositeTypeRef: # XRD reference
apiVersion: platform.acme.io/v1alpha1 # custom API version
kind: XAppInfra # composite kind
resources: # composed resources
- name: database # database resource
base: # base template
apiVersion: rds.aws.upbound.io/v1beta1 # AWS RDS API
kind: Instance # RDS Instance
spec: # resource spec
forProvider: # provider config
engine: postgres # PostgreSQL
engineVersion: "15" # version 15
instanceClass: db.r5.xlarge # large prod instance
allocatedStorage: 200 # 200 GB storage
multiAZ: true # multi-AZ for HA
skipFinalSnapshot: false # require final snapshot
deletionProtection: true # prevent accidental deletion
storageEncrypted: true # encrypt at rest
region: us-east-1 # prod region
providerConfigRef: # use production account
name: aws-prod-payments # prod ProviderConfig
patches: # dynamic patches
- type: FromCompositeFieldPath # map service name to DB identifier
fromFieldPath: spec.serviceName # source service name
toFieldPath: metadata.annotations[crossplane.io/external-name] # external name
transforms: # name transformation
- type: string # string transform
string: # string config
type: Format # format type
fmt: "%s-db-prod" # append -db-prod suffix
- name: cloudwatch-alarm # monitoring resource (prod only)
base: # base template
apiVersion: cloudwatch.aws.upbound.io/v1beta1 # CloudWatch API
kind: MetricAlarm # metric alarm resource
spec: # resource spec
forProvider: # provider config
region: us-east-1 # same region as database
comparisonOperator: GreaterThanThreshold # alert when above threshold
evaluationPeriods: 3 # three consecutive periods
metricName: CPUUtilization # monitor CPU usage
namespace: AWS/RDS # RDS metrics namespace
period: 300 # 5-minute evaluation period
statistic: Average # average statistic
threshold: 80 # alert at 80 percent CPU
providerConfigRef: # use production account
name: aws-prod-payments # prod ProviderConfig
---
# Claim from the checkout-service team for production
apiVersion: platform.acme.io/v1alpha1 # custom API version
kind: AppInfra # claim kind
metadata: # resource metadata
name: checkout-service-infra # claim name
namespace: checkout # team namespace
spec: # claim spec
serviceName: checkout-service # application name
environment: prod # target environment
storageGB: 150 # requested storage
compositionSelector: # select composition by label
matchLabels: # label matching
environment: prod # route to prod composition💡
Interview Tip
A junior engineer typically proposes duplicating entire claim files for each environment, but interviewers want to see architectural thinking about how to minimize duplication while maximizing isolation. Explain the tradeoffs between single-Composition-with-patches versus separate-Compositions-per-environment. The single Composition approach is easier to maintain but harder to audit, while separate Compositions provide clear production boundaries but risk drift between environments. Discuss how compositionSelector with labels enables clean routing without modifying the Composition itself. Mention that prod Compositions should include monitoring resources like CloudWatch alarms and SNS topics that dev Compositions omit entirely. Walk through a GitOps promotion workflow using Kustomize overlays where environment promotion is a reviewed pull request, showing that you understand infrastructure changes should follow the same review process as application code changes.
◈ Architecture Diagram
┌─────────────────────────────────────────────────────────┐
│ Environment-Specific Routing │
└────────────────────────┬────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────┐
│ AppInfra Claim: checkout-service-infra │
│ compositionSelector: environment=prod │
└──────────┬──────────────────────────────┬───────────────┘
│ │
┌─────┴─────┐ ┌─────┴─────┐
↓ ↓ ↓ ↓
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
│ DEV │ │ STAGING │ │ PROD │ │ PROD │
│ Comp │ │ Comp │ │ Comp │ │ Comp │
│ │ │ │ │ ✓ Match │ │ │
└────┬────┘ └────┬────┘ └────┬────┘ └─────────┘
│ │ │
↓ ↓ ↓
┌─────────┐ ┌─────────┐ ┌──────────────────────┐
│ aws- │ │ aws- │ │ aws-prod-payments │
│ sandbox │ │ sandbox │ │ Account: 111111111 │
│ │ │ │ └──────────┬───────────┘
│ ● t3 │ │ ● t3 │ │
│ micro │ │ medium│ ↓
│ ● 20GB │ │ ● 50GB │ ┌──────────────────────┐
│ ● no HA │ │ ● no HA │ │ Resources Created: │
└─────────┘ └─────────┘ │ ● db.r5.xlarge │
│ ● 200GB, Multi-AZ │
│ ● Encryption ✓ │
│ ● Deletion Protect ✓ │
│ ● CloudWatch Alarm │
│ ● Final Snapshot ✓ │
└──────────────────────┘💬 Comments
⚡
Quick Answer
Crossplane handles resource dependencies through references and selectors that create implicit ordering, where a managed resource can reference another resource's output fields, and Crossplane automatically waits for the referenced resource to be ready before proceeding with the dependent resource's creation.
Detailed Answer
Managing dependencies between infrastructure resources is fundamental to provisioning complex architectures. The payments-api database cannot be created before the VPC and subnets exist, the security group needs the VPC ID before it can be created, and the RDS instance needs both the subnet group and security group to be ready before it can launch. Crossplane handles these dependencies through a reference system that both establishes ordering and propagates values between resources, ensuring that the order-processing-service infrastructure is provisioned in the correct sequence without explicit dependency declarations.
The primary dependency mechanism is the reference and selector system built into every managed resource's spec. When a managed resource has a field that accepts a reference to another resource, Crossplane automatically resolves that reference by watching the referenced resource and waiting for it to become ready. For example, the SecurityGroup for the checkout-service database has a vpcIdRef that points to the VPC resource by name. Crossplane watches the VPC, waits for its external ID to be populated, resolves the VPC ID, and injects it into the security group's vpcId field. Until the VPC is ready, the security group remains in a pending state with a Synced condition message indicating it is waiting for the reference to resolve. This implicit ordering means platform teams do not need to declare an explicit depends_on like in Terraform.
Selectors provide a more flexible alternative to direct references when the exact resource name is not known at authoring time. Instead of referencing a specific VPC by name, a selector can match by labels, such as selecting the VPC with labels app: payments-api and environment: prod. This pattern is particularly powerful in Compositions where the composed resources are generated dynamically. The Composition assigns matching labels to the VPC and security group resources, and the security group uses a vpcIdSelector with matchControllerRef set to true, which automatically selects the VPC resource owned by the same composite resource. This controller reference matching ensures that the checkout-service security group only references the checkout-service VPC, even when multiple composite resources create VPCs in the same cluster.
Within Compositions, dependencies between composed resources use a combination of patches and the readinessChecks field. The Composition defines a VPC, a subnet, a subnet group, a security group, and an RDS instance as composed resources. Patches flow values between these resources using the FromCompositeFieldPath and ToCompositeFieldPath patch types. The VPC's ID is patched ToCompositeFieldPath into a status field on the composite resource, and the security group reads it back FromCompositeFieldPath. This two-hop patching through the composite resource creates an implicit dependency chain. The readinessChecks on each composed resource define when that resource is considered ready, and dependent resources will show their references as unresolvable until the upstream resource passes its readiness check. For the inventory-sync service, the RDS instance readiness check verifies that the status field equals Available, and downstream resources like Route53 DNS records only get created after the database endpoint is populated.
Cross-resource dependencies in complex architectures sometimes require external resources that exist outside the current Composition. The user-auth-service database might need to reference a shared VPC managed by the networking team's Composition. Crossplane supports this through cross-resource references using the resource's name or through the usage resource type. The Usage resource declares that one resource depends on another, and Crossplane enforces that the depended-upon resource cannot be deleted while the usage relationship exists. This prevents scenarios where the networking team deletes the shared VPC while the user-auth-service RDS instance still needs it. Usage resources also support the reason field to document why the dependency exists, creating a self-documenting infrastructure graph that teams can query to understand blast radius before making changes.
A common production pitfall is circular dependencies where resource A references resource B and resource B references resource A. Crossplane detects some circular reference patterns and reports them as errors, but indirect cycles through the composite resource status fields can be harder to diagnose. Teams should design their Composition dependency graphs as directed acyclic graphs and document the flow in architecture diagrams. Another gotcha is that reference resolution failures are retried indefinitely, which means a typo in a reference name silently blocks provisioning without any timeout. Platform teams should monitor for resources stuck in the Synced=False state with reference resolution errors and set up alerts that trigger after a configurable threshold to catch these issues before the order-processing-service team opens a support ticket wondering why their database has been pending for hours.
Code Example
# VPC resource - no dependencies, created first
apiVersion: ec2.aws.upbound.io/v1beta1 # AWS EC2 provider API
kind: VPC # VPC resource type
metadata: # resource metadata
name: payments-api-vpc-prod # VPC name
labels: # labels for selector matching
app: payments-api # application label
environment: prod # environment label
spec: # VPC specification
forProvider: # provider configuration
region: us-east-1 # AWS region
cidrBlock: 10.0.0.0/16 # VPC CIDR range
enableDnsSupport: true # enable DNS resolution
enableDnsHostnames: true # enable DNS hostnames
providerConfigRef: # ProviderConfig reference
name: aws-prod-payments # production account
---
# Subnet resource - depends on VPC via reference
apiVersion: ec2.aws.upbound.io/v1beta1 # AWS EC2 provider API
kind: Subnet # Subnet resource type
metadata: # resource metadata
name: payments-api-subnet-a # subnet name
labels: # labels for selection
app: payments-api # application label
zone: us-east-1a # availability zone label
spec: # subnet specification
forProvider: # provider configuration
region: us-east-1 # AWS region
availabilityZone: us-east-1a # specific AZ
cidrBlock: 10.0.1.0/24 # subnet CIDR
vpcIdRef: # reference to VPC resource
name: payments-api-vpc-prod # VPC resource name (waits for VPC)
providerConfigRef: # ProviderConfig reference
name: aws-prod-payments # production account
---
# Security Group - depends on VPC via selector
apiVersion: ec2.aws.upbound.io/v1beta1 # AWS EC2 provider API
kind: SecurityGroup # SecurityGroup resource type
metadata: # resource metadata
name: payments-api-db-sg-prod # security group name
spec: # security group specification
forProvider: # provider configuration
region: us-east-1 # AWS region
description: Database access for payments-api # SG description
vpcIdSelector: # select VPC by labels instead of name
matchLabels: # label matching criteria
app: payments-api # match application label
environment: prod # match environment label
providerConfigRef: # ProviderConfig reference
name: aws-prod-payments # production account
---
# RDS Instance - depends on subnet group and security group
apiVersion: rds.aws.upbound.io/v1beta1 # AWS RDS provider API
kind: Instance # RDS Instance resource type
metadata: # resource metadata
name: payments-api-db-prod # RDS instance name
spec: # instance specification
forProvider: # provider configuration
region: us-east-1 # AWS region
engine: postgres # PostgreSQL engine
engineVersion: "15" # Postgres 15
instanceClass: db.r5.xlarge # production instance class
allocatedStorage: 200 # 200 GB storage
dbSubnetGroupNameRef: # reference to subnet group
name: payments-api-subnetgroup-prod # waits for subnet group to be ready
vpcSecurityGroupIdRefs: # references to security groups
- name: payments-api-db-sg-prod # waits for SG to be ready
providerConfigRef: # ProviderConfig reference
name: aws-prod-payments # production account
---
# Usage resource - prevents VPC deletion while RDS depends on it
apiVersion: apiextensions.crossplane.io/v1alpha1 # Crossplane Usage API
kind: Usage # Usage resource type
metadata: # resource metadata
name: payments-db-uses-vpc # usage relationship name
spec: # usage specification
of: # the resource being depended upon
apiVersion: ec2.aws.upbound.io/v1beta1 # VPC API version
kind: VPC # VPC resource kind
resourceRef: # specific resource reference
name: payments-api-vpc-prod # VPC resource name
by: # the resource that has the dependency
apiVersion: rds.aws.upbound.io/v1beta1 # RDS API version
kind: Instance # RDS Instance kind
resourceRef: # specific resource reference
name: payments-api-db-prod # RDS resource name
reason: RDS instance requires VPC for network connectivity # human-readable dependency reason
---
# Debug dependency resolution issues
kubectl get managed -o custom-columns=NAME:.metadata.name,SYNCED:.status.conditions[?(@.type=="Synced")].status,READY:.status.conditions[?(@.type=="Ready")].status,MESSAGE:.status.conditions[?(@.type=="Synced")].message # check sync status and resolution messages💡
Interview Tip
A junior engineer typically compares Crossplane dependencies to Terraform's depends_on and misses the fact that Crossplane uses an implicit reference-based system rather than explicit ordering. Explain how references and selectors create automatic ordering by making Crossplane wait for the referenced resource to become ready before creating the dependent resource. Discuss the matchControllerRef pattern in Compositions that scopes selector matching to resources owned by the same composite resource, preventing cross-contamination between claims. Mention the Usage resource as a deletion protection mechanism that prevents upstream resources from being removed while downstream resources still depend on them. If you can describe a debugging scenario where a resource is stuck pending due to an unresolvable reference and walk through the kubectl describe output showing the resolution error message, you demonstrate real operational troubleshooting experience.
◈ Architecture Diagram
┌─────────────────────────────────────────────────────────┐
│ Resource Dependency Graph: payments-api │
└─────────────────────────────────────────────────────────┘
┌──────────────┐
│ VPC │
│ 10.0.0.0/16 │
│ ✓ Ready │
└──────┬───────┘
│
├─────────────────────────────┐
│ │
↓ ↓
┌──────────────┐ ┌──────────────┐
│ Subnet A │ │ Security Grp │
│ 10.0.1.0/24 │ │ vpcIdRef → │
│ vpcIdRef → │ │ payments-vpc │
│ payments-vpc│ │ ✓ Ready │
│ ✓ Ready │ └──────┬───────┘
└──────┬───────┘ │
│ │
↓ │
┌──────────────┐ │
│ Subnet B │ │
│ 10.0.2.0/24 │ │
│ ✓ Ready │ │
└──────┬───────┘ │
│ │
↓ │
┌──────────────┐ │
│ Subnet Group │ │
│ subnetRefs → │ │
│ [A, B] │ │
│ ✓ Ready │ │
└──────┬───────┘ │
│ │
└──────────┬───────────────┘
│
↓
┌──────────────────┐
│ RDS Instance │
│ dbSubnetGroupRef │
│ vpcSGIdRefs → │
│ payments-api-db │
│ ● Waiting... │
└──────────────────┘
┌─────────────────────────────────────────┐
│ Usage: payments-db-uses-vpc │
│ VPC ←── cannot delete while ──→ RDS │
│ Deletion blocked until RDS removed │
└─────────────────────────────────────────┘💬 Comments
⚡
Quick Answer
Crossplane integrates with ArgoCD by storing Crossplane claims and compositions in a Git repository, where ArgoCD syncs them to the Kubernetes cluster, enabling GitOps workflows where infrastructure changes follow the same pull request review process as application deployments with full audit trails and automated drift correction.
Detailed Answer
The combination of Crossplane and ArgoCD creates a powerful GitOps infrastructure management platform where Git becomes the single source of truth for both application deployments and cloud infrastructure provisioning. ArgoCD handles the synchronization of desired state from Git to the Kubernetes cluster, and Crossplane handles the reconciliation of that desired state from Kubernetes to the cloud provider. This two-layer reconciliation means the payments-api team can request infrastructure through a pull request, get peer review from the platform team, merge to trigger automatic provisioning, and have continuous drift detection at both the Kubernetes layer through ArgoCD and the cloud layer through Crossplane.
The repository structure for a Crossplane-ArgoCD integration typically follows one of two patterns. The first pattern stores platform definitions and application claims in separate repositories. The platform repository contains CompositeResourceDefinitions, Compositions, ProviderConfigs, and provider packages, managed by the platform team through a dedicated ArgoCD Application that syncs to the crossplane-system namespace. The application repository contains Crossplane Claims organized by team and environment, such as teams/checkout-service/prod/database-claim.yaml and teams/order-processing-service/staging/cache-claim.yaml. Each team directory gets its own ArgoCD Application pointing to their folder, with RBAC restrictions ensuring the checkout-service team can only modify their own directory. This separation of concerns means platform changes go through the platform team's review process while application infrastructure requests go through the application team's process.
ArgoCD Application configuration for Crossplane requires specific health check customizations because ArgoCD does not natively understand Crossplane resource health. By default, ArgoCD considers a resource healthy if it exists in the cluster, but a Crossplane Claim might exist while the underlying cloud resources are still provisioning or in an error state. Custom health checks in the ArgoCD ConfigMap teach ArgoCD to inspect the Synced and Ready conditions on Crossplane resources. When the user-auth-service claim is synced by ArgoCD, the health check reads the Ready condition and maps True to Healthy, False to Degraded, and missing to Progressing. This integration ensures the ArgoCD dashboard accurately reflects whether the inventory-sync database is actually available in AWS, not just whether the Kubernetes resource exists. Without these health checks, teams get false confidence from green ArgoCD status while their infrastructure is actually broken.
Sync waves and hooks in ArgoCD orchestrate the ordering of Crossplane resource deployment. Platform resources like ProviderConfigs and XRDs must be synced before Compositions, and Compositions must be synced before Claims that reference them. ArgoCD sync waves handle this by assigning wave numbers through annotations, where wave zero deploys providers and ProviderConfigs, wave one deploys XRDs, wave two deploys Compositions, and wave three deploys Claims. ArgoCD processes waves sequentially and waits for resources in each wave to become healthy before proceeding to the next wave. For the checkout-service, this ensures the database Composition is registered before the database Claim is applied, preventing the rejected-by-API-server errors that occur when a Claim references a CRD that does not exist yet.
Operational considerations for this integration include handling the eventual consistency gap between ArgoCD sync and Crossplane reconciliation. When ArgoCD syncs a new database Claim, it creates the Kubernetes resource immediately, but the underlying RDS instance might take fifteen minutes to provision. ArgoCD should be configured with a sync timeout that accounts for cloud provisioning times, and the custom health check should return Progressing rather than Healthy during this window. Teams should also configure ArgoCD's resource exclusion rules to ignore Crossplane-managed external resources, because ArgoCD might detect drift on managed resources that are controlled by Crossplane's reconciliation loop rather than by Git. Setting the argocd.argoproj.io/managed-by annotation or using resource tracking annotations prevents ArgoCD from fighting with Crossplane over the same resource's state. The payments-api team benefits from ArgoCD notifications that integrate with Slack, sending messages when infrastructure claims are synced, when health checks transition to Degraded, or when sync operations fail due to validation errors.
Code Example
# ArgoCD Application for platform Crossplane resources
apiVersion: argoproj.io/v1alpha1 # ArgoCD API version
kind: Application # ArgoCD Application resource
metadata: # resource metadata
name: crossplane-platform # application name
namespace: argocd # ArgoCD namespace
annotations: # annotations
argocd.argoproj.io/sync-wave: "0" # deploy platform resources first
spec: # application specification
project: infrastructure # ArgoCD project for infra
source: # Git source configuration
repoURL: https://github.com/acme/infra-platform.git # platform repo URL
targetRevision: main # track main branch
path: crossplane/platform # path to platform definitions
destination: # target cluster and namespace
server: https://kubernetes.default.svc # in-cluster deployment
namespace: crossplane-system # Crossplane system namespace
syncPolicy: # sync behavior configuration
automated: # enable auto-sync
prune: true # remove resources deleted from Git
selfHeal: true # revert manual changes
syncOptions: # sync options
- CreateNamespace=true # create namespace if missing
- ServerSideApply=true # use server-side apply for CRDs
---
# ArgoCD Application for checkout-service infrastructure claims
apiVersion: argoproj.io/v1alpha1 # ArgoCD API version
kind: Application # ArgoCD Application resource
metadata: # resource metadata
name: checkout-service-infra # application name
namespace: argocd # ArgoCD namespace
annotations: # annotations
argocd.argoproj.io/sync-wave: "3" # deploy claims after compositions
spec: # application specification
project: checkout-team # team-scoped ArgoCD project
source: # Git source configuration
repoURL: https://github.com/acme/infra-claims.git # claims repository URL
targetRevision: main # track main branch
path: teams/checkout-service/prod # team and environment path
destination: # target cluster and namespace
server: https://kubernetes.default.svc # in-cluster deployment
namespace: checkout # checkout team namespace
syncPolicy: # sync behavior
automated: # enable auto-sync
prune: true # prune deleted resources
selfHeal: true # self-heal drift
retry: # retry on failure
limit: 5 # retry up to 5 times
backoff: # backoff configuration
duration: 30s # initial retry delay
factor: 2 # exponential backoff factor
maxDuration: 5m # maximum retry delay
---
# ArgoCD ConfigMap for Crossplane health checks
apiVersion: v1 # Kubernetes core API
kind: ConfigMap # ConfigMap resource type
metadata: # resource metadata
name: argocd-cm # ArgoCD main ConfigMap
namespace: argocd # ArgoCD namespace
data: # ConfigMap data
resource.customizations.health.database.acme.io_PostgreSQLInstance: | # health check for claims
hs = {} # initialize health status table
if obj.status ~= nil then # check if status exists
if obj.status.conditions ~= nil then # check if conditions exist
for i, condition in ipairs(obj.status.conditions) do # iterate over conditions
if condition.type == "Ready" then # find the Ready condition
if condition.status == "True" then # check if Ready is True
hs.status = "Healthy" # mark as Healthy
hs.message = "Resource is ready" # set healthy message
elseif condition.status == "False" then # check if Ready is False
hs.status = "Degraded" # mark as Degraded
hs.message = condition.message # use condition message
end # end if-else
end # end type check
end # end for loop
end # end conditions check
end # end status check
if hs.status == nil then # if no status determined
hs.status = "Progressing" # default to Progressing
hs.message = "Waiting for resource to be ready" # set waiting message
end # end default check
return hs # return health status
---
# Git repository structure for infrastructure claims
# infra-claims/
# teams/
# checkout-service/
# dev/
# database-claim.yaml # dev database claim
# cache-claim.yaml # dev cache claim
# prod/
# database-claim.yaml # prod database claim
# cache-claim.yaml # prod cache claim
# order-processing-service/
# staging/
# database-claim.yaml # staging database claim
# payments-api/
# prod/
# database-claim.yaml # prod payments database💡
Interview Tip
A junior engineer typically describes ArgoCD and Crossplane as independent tools without explaining the two-layer reconciliation model or the custom health check requirement. Show the interviewer you understand that ArgoCD reconciles Git to Kubernetes while Crossplane reconciles Kubernetes to cloud, and that without custom health checks ArgoCD shows green status even when cloud resources are broken. Walk through the sync wave ordering that ensures ProviderConfigs and XRDs are deployed before Compositions, and Compositions before Claims. Discuss the repository structure separating platform definitions from application claims with team-scoped RBAC. Mention the ServerSideApply sync option for CRD handling, the retry configuration for handling transient cloud API errors, and the selfHeal setting that reverts manual kubectl changes to infrastructure claims. If you can explain how to prevent ArgoCD and Crossplane from fighting over managed resource state using annotation-based ownership, you demonstrate advanced integration knowledge.
◈ Architecture Diagram
┌─────────────────────────────────────────────────────────┐
│ Git Repository │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ infra-platform/ │ │ infra-claims/ │ │
│ │ ● XRDs │ │ teams/ │ │
│ │ ● Compositions │ │ checkout-svc/ │ │
│ │ ● ProviderConfigs│ │ prod/ │ │
│ │ ● Providers │ │ db-claim.yaml │ │
│ └────────┬─────────┘ └────────┬─────────┘ │
└───────────┼───────────────────────┼─────────────────────┘
│ │
↓ ↓
┌─────────────────────────────────────────────────────────┐
│ ArgoCD │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ App: crossplane │ │ App: checkout │ │
│ │ -platform │ │ -service-infra │ │
│ │ sync-wave: 0 │ │ sync-wave: 3 │ │
│ │ ✓ Synced │ │ ✓ Synced │ │
│ └────────┬─────────┘ └────────┬─────────┘ │
│ │ Layer 1: Git→K8s │ │
└───────────┼───────────────────────┼─────────────────────┘
│ │
↓ ↓
┌─────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
│ │
│ ┌──────────────┐ ┌───────────────────────┐ │
│ │ XRDs │ │ PostgreSQLInstance │ │
│ │ Compositions │ │ Claim: checkout-db │ │
│ │ Providers │ │ Ready: True │ │
│ └──────────────┘ └───────────┬───────────┘ │
│ │ │
│ Layer 2: K8s→Cloud (Crossplane) │
└────────────────────────────────┼────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────┐
│ AWS Cloud │
│ ● RDS: checkout-service-db-prod │
│ ● Security Group: checkout-db-sg │
│ ● CloudWatch Alarm: checkout-db-cpu │
│ ✓ All resources reconciled │
└─────────────────────────────────────────────────────────┘💬 Comments
⚡
Quick Answer
Troubleshooting Crossplane failures involves a systematic approach of checking the managed resource's Synced and Ready conditions, examining Kubernetes events on the resource and its composite parent, reviewing provider pod logs for API errors, and validating the ProviderConfig credentials and cloud-side permissions.
Detailed Answer
Crossplane resource provisioning failures are inevitable in production, and having a systematic troubleshooting methodology is essential for platform engineers. When the payments-api team reports that their database claim has been pending for thirty minutes, the investigation follows a layered approach starting from the user-facing claim and drilling down through the composite resource, composed managed resources, provider logs, and cloud API responses. Each layer reveals different types of failures, from schema validation errors at the claim level to IAM permission denials at the cloud API level.
The first diagnostic step is examining the claim and composite resource conditions. Running kubectl describe on the PostgreSQLInstance claim shows conditions including Ready, Synced, and any custom conditions set by the Composition. A Synced=False condition indicates that Crossplane cannot reconcile the desired state with the external resource, and the condition message provides the first clue. Common messages include cannot resolve reference meaning a dependency is not ready, cannot assume role indicating credential issues, or InvalidParameterCombination revealing an incompatible set of cloud resource configurations. The composite resource sits above the claim and aggregates the readiness of all composed resources, so describing the XPostgreSQLInstance resource shows which specific composed resource is blocking overall readiness. If four out of five composed resources show Ready=True and one shows Ready=False, the problem is isolated to that single resource.
The second layer involves inspecting the managed resources individually. The kubectl get managed command lists all Crossplane-managed resources with their Synced and Ready status, providing a cluster-wide view of provisioning health. For the failing order-processing-service database, kubectl describe on the specific RDS Instance managed resource reveals detailed events. Crossplane emits events like cannot create external resource with an error message from the cloud API, such as DBSubnetGroupNotFoundFault indicating that the subnet group referenced in the RDS configuration does not exist or is not ready yet. The events section also shows timestamps for each reconciliation attempt, helping correlate failures with other cluster or cloud events. Field-level validation errors appear as events with messages like spec.forProvider.instanceClass is invalid, pointing directly to the misconfigured field.
Provider pod logs contain the most granular debugging information. The AWS provider pod runs in the crossplane-system namespace and logs every API call it makes to AWS, including the request parameters, response status codes, and error messages. For the checkout-service provisioning failure, filtering logs by the resource name reveals the exact AWS API call that failed and the error response. Common discoveries include throttling errors when the provider is managing hundreds of resources and hitting AWS API rate limits, access denied errors when the IAM role in the ProviderConfig lacks the required permissions, and resource limit exceeded errors when the AWS account has hit a service quota. Provider logs also show the reconciliation timing, helping identify whether the issue is intermittent due to rate limiting or persistent due to misconfiguration. Setting the provider log level to debug through the ControllerConfig resource increases verbosity for troubleshooting complex issues.
Credential and permission validation is the third critical troubleshooting domain. The ProviderConfig might reference a Kubernetes Secret that has been deleted or rotated, an IRSA service account annotation that does not match the IAM role trust policy, or an AssumeRole chain that fails at an intermediate step. The kubectl describe providerconfig command shows whether the ProviderConfig itself is healthy. For IRSA-based authentication, verifying the service account annotation, the IAM role trust policy, and the OIDC provider configuration requires cross-checking between Kubernetes, IAM, and EKS resources. A common failure mode for the user-auth-service is that the ProviderConfig works fine for existing resources but fails for new resource types because the IAM policy attached to the assumed role lacks permissions for the new AWS service. Testing permissions with aws sts assume-role followed by the specific API call from outside the cluster helps isolate whether the issue is in the Crossplane authentication chain or in the IAM policy itself.
Preventive troubleshooting practices include setting up Prometheus alerts on the crossplane_managed_resource_ready metric that fires when resources remain not-ready beyond a threshold, configuring PagerDuty integration for Synced=False conditions on production resources, and running crossplane beta validate in CI pipelines to catch schema validation errors before they reach the cluster. The crossplane beta trace command is particularly valuable because it shows the complete resource tree from claim through composite to all managed resources with their status in a single output, eliminating the need to manually describe each resource in the chain when the inventory-sync team reports a failure.
Code Example
# Step 1: Check the claim status and conditions
kubectl describe postgresqlinstance payments-api-db -n payments # describe the claim to see Ready and Synced conditions
# Step 2: Get the composite resource linked to the claim
kubectl get xpostgresqlinstance -o wide # list all composite resources with their status
# Step 3: Trace the full resource tree from claim to managed resources
crossplane beta trace postgresqlinstance payments-api-db -n payments # show complete resource tree with status
# Step 4: List all managed resources and their sync status
kubectl get managed -o custom-columns=KIND:.kind,NAME:.metadata.name,SYNCED:.status.conditions[?(@.type=="Synced")].status,READY:.status.conditions[?(@.type=="Ready")].status,AGE:.metadata.creationTimestamp # comprehensive managed resource status view
# Step 5: Describe the failing managed resource for detailed events
kubectl describe instance.rds.aws.upbound.io payments-api-db-prod # show events, conditions, and spec for the RDS instance
# Step 6: Check events for all Crossplane resources in the namespace
kubectl get events --field-selector involvedObject.name=payments-api-db-prod --sort-by='.lastTimestamp' # sorted events for the failing resource
# Step 7: Check provider pod health and logs
kubectl get pods -n crossplane-system -l pkg.crossplane.io/revision # list all provider pods
kubectl logs -n crossplane-system -l pkg.crossplane.io/revision=provider-aws -c provider --tail=100 # tail AWS provider logs
kubectl logs -n crossplane-system -l pkg.crossplane.io/revision=provider-aws -c provider --since=10m | grep -i error # filter for errors in last 10 minutes
# Step 8: Verify ProviderConfig health
kubectl describe providerconfig.aws.upbound.io aws-prod-payments # check ProviderConfig conditions and credential status
# Step 9: Check for IRSA service account configuration
kubectl get serviceaccount -n crossplane-system -o yaml | grep -A2 annotations # verify IRSA annotation on provider service account
# Step 10: Validate Composition and claim schema before applying
crossplane beta validate extensions.yaml claim.yaml # validate claim against XRD schema locally
# Step 11: Check cloud-side resource state directly
aws rds describe-db-instances --db-instance-identifier payments-api-db-prod --region us-east-1 # verify AWS-side resource state
# Step 12: Force reconciliation on a stuck resource
kubectl annotate instance.rds.aws.upbound.io payments-api-db-prod crossplane.io/reconcile=now --overwrite # trigger immediate reconciliation
# Step 13: Increase provider log verbosity for deep debugging
kubectl patch deploymentruntimeconfig default --type=merge -p '{"spec":{"deploymentTemplate":{"spec":{"template":{"spec":{"containers":[{"name":"package-runtime","args":["--debug"]}]}}}}}}' # enable debug logging on provider💡
Interview Tip
A junior engineer typically jumps straight to checking provider pod logs without following a systematic top-down approach. Interviewers want to see that you start from the user-facing claim, check its conditions, then drill into the composite resource to identify which composed resource is failing, and only then examine provider logs for the specific resource. Demonstrate knowledge of the crossplane beta trace command that shows the complete resource tree in one output. Walk through common failure categories: reference resolution failures where a dependency is not ready, credential failures where the ProviderConfig cannot authenticate, permission failures where the IAM role lacks required actions, and cloud API errors like service quota limits or invalid parameter combinations. Mention preventive measures like Prometheus alerts on crossplane_managed_resource_ready metrics and CI validation with crossplane beta validate. If you can describe a real scenario where you traced a failure from a pending claim through provider logs to an IAM policy missing a specific action, you demonstrate genuine operational experience.
◈ Architecture Diagram
┌─────────────────────────────────────────────────────────┐
│ Troubleshooting Flow: Top-Down Approach │
└─────────────────────────────────────────────────────────┘
┌──────────────────┐
│ 1. CLAIM │
│ kubectl describe│
│ postgresqlinstance│
│ payments-api-db │
│ │
│ Ready: False │
│ Synced: True │
└────────┬─────────┘
│ Check composite
↓
┌──────────────────┐
│ 2. COMPOSITE │
│ crossplane beta │
│ trace │
│ │
│ XR: 4/5 Ready │
│ ✓ VPC │
│ ✓ Subnet Group │
│ ✓ Security Group│
│ ✗ RDS Instance │← failing resource
│ ✓ DNS Record │
└────────┬─────────┘
│ Describe failing resource
↓
┌──────────────────┐
│ 3. MANAGED │
│ kubectl describe│
│ instance.rds │
│ │
│ Event: │
│ cannot create: │
│ AccessDenied │
└────────┬─────────┘
│ Check provider
↓
┌──────────────────┐
│ 4. PROVIDER │
│ kubectl logs │
│ provider-aws │
│ │
│ Error: iam role │
│ missing action: │
│ rds:CreateDB │
│ Instance │
└────────┬─────────┘
│ Verify credentials
↓
┌──────────────────┐
│ 5. CREDENTIALS │
│ ProviderConfig │
│ → IRSA → Role │
│ → IAM Policy │
│ │
│ Fix: Add rds:* │
│ to IAM policy │
│ ✓ Resolved │
└──────────────────┘💬 Comments
⚡
Quick Answer
Upjet is a code generation framework that creates Crossplane providers from existing Terraform providers. You configure resource mappings in a provider-config repository, run the Upjet code generator to produce CRDs and controllers, then build and publish the provider as an OCI image that Crossplane installs as a standard provider package.
Detailed Answer
Building a custom Crossplane Provider with Upjet allows organizations to extend Crossplane's control plane capabilities by wrapping existing Terraform providers with Kubernetes-native APIs. This approach is significantly faster than writing a provider from scratch because Upjet generates the CRD schemas, controller reconciliation logic, and conversion functions from the Terraform provider's schema. At a company running services like payments-api and order-processing-service, a platform team might build an Upjet-based provider to manage internal services such as a proprietary message queue or a custom database provisioning system that already has a Terraform provider.
The process begins by scaffolding a new provider repository using the upjet-provider-template. This template contains the generator pipeline configuration, Makefile targets for code generation, and the directory structure for resource configurations. You configure the Terraform provider source and version in the generator config, then define which Terraform resources and data sources should be exposed as Crossplane managed resources. Each resource configuration specifies the API group, kind name, references to other resources (for automatic cross-resource dependency resolution), and sensitive field paths that Crossplane should store in Kubernetes Secrets rather than in the managed resource spec.
The code generation phase is where Upjet does the heavy lifting. Running make generate triggers a pipeline that reads the Terraform provider schema, applies your resource configurations, and produces Go types for CRDs, zz_controller.go files with reconciliation logic, and conversion functions between the Crossplane API types and the underlying Terraform resource state. Upjet handles complexities like nested block flattening, reference resolution for cross-resource dependencies, and late initialization of server-side defaults. For the payments-api team, this means they get a PaymentsQueue custom resource that can reference a VPC managed resource by name instead of hardcoding IDs.
After generation, you build the provider binary and package it as an OCI image. The Makefile includes targets for building multi-architecture images and pushing them to a container registry. The provider package includes a crossplane.yaml manifest that declares the provider's CRDs and controller image. You install the provider into your Crossplane control plane using a Provider resource that references the OCI image URL. Crossplane's package manager pulls the image, installs the CRDs, and starts the controller pod.
Production considerations for Upjet-based providers include version pinning the underlying Terraform provider to prevent unexpected schema changes during regeneration, implementing comprehensive end-to-end tests that provision real resources against a sandbox environment, configuring provider credentials using Crossplane's ProviderConfig resources with Kubernetes Secrets or external secret stores, and monitoring the controller's reconciliation metrics. When inventory-sync or checkout-service teams consume the custom provider's resources, they should see the same declarative experience as with official Crossplane providers, with status conditions reporting readiness and connection details published to Secrets.
Code Example
# Clone the Upjet provider template repository
git clone https://github.com/crossplane/upjet-provider-template provider-internal-services
# Navigate into the newly cloned provider directory
cd provider-internal-services
# Edit the Makefile to set the provider name and organization
# PROJECT_NAME := provider-internal-services
# PROJECT_REPO := github.com/acme-corp/provider-internal-services
# config/provider.go - Configure the Terraform provider source
# package config
# import (
# ujconfig "github.com/crossplane/upjet/pkg/config"
# )
# func GetProvider() *ujconfig.Provider {
# // Initialize the provider with the Terraform source
# pc := ujconfig.NewProvider(
# []byte(providerSchema),
# resourcePrefix,
# modulePath,
# // Specify the Terraform provider to wrap
# ujconfig.WithRootGroup("internal.acme.io"),
# ujconfig.WithIncludeList(ExternalNameConfigured()),
# ujconfig.WithFeaturesPackage("internal/features"),
# )
# return pc
# }
# config/paymentsqueue/config.go - Configure a specific resource
# package paymentsqueue
# import "github.com/crossplane/upjet/pkg/config"
# func Configure(p *config.Provider) {
# // Map the Terraform resource to a Crossplane managed resource
# p.AddResourceConfigurator("internal_payments_queue", func(r *config.Resource) {
# r.ShortGroup = "messaging"
# r.Kind = "PaymentsQueue"
# // Reference the VPC resource for cross-resource resolution
# r.References["vpc_id"] = config.Reference{
# Type: "github.com/acme-corp/provider-internal-services/apis/network/v1alpha1.VPC",
# }
# // Mark connection credentials as sensitive
# r.Sensitive.AdditionalConnectionDetailsFn = func(...) {}
# })
# }
# Run the code generator to produce CRDs and controllers
make generate
# Build the provider binary and OCI image
make build
# Push the provider image to the container registry
make publish REGISTRY=registry.acme-corp.io
# Install the custom provider in Crossplane
kubectl apply -f - <<EOF
# Define the Provider resource for Crossplane to install
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
# Name the provider for internal services management
name: provider-internal-services
spec:
# Reference the OCI image in the corporate registry
package: registry.acme-corp.io/provider-internal-services:v0.1.0
# Set the revision activation policy to automatic
revisionActivationPolicy: Automatic
# Keep one inactive revision for rollback capability
revisionHistoryLimit: 1
EOF
# Verify the provider is installed and healthy
kubectl get providers provider-internal-services
# Create a ProviderConfig with credentials for the internal API
kubectl apply -f - <<EOF
# Configure authentication for the internal services provider
apiVersion: internal.acme.io/v1alpha1
kind: ProviderConfig
metadata:
# Default config used when resources omit providerConfigRef
name: default
spec:
credentials:
# Pull credentials from a Kubernetes Secret
source: Secret
secretRef:
# Secret containing the internal API authentication token
namespace: crossplane-system
name: internal-api-credentials
key: token
EOF💡
Interview Tip
A junior engineer typically says Crossplane providers are pre-built packages you install from the marketplace, without understanding how to create custom ones. To demonstrate depth, walk through the Upjet code generation pipeline: how it reads the Terraform provider schema, generates CRD types and controller reconciliation logic, and handles cross-resource references. Explain why Upjet is preferred over writing a provider from scratch (months of effort versus days), discuss how sensitive fields are automatically routed to Kubernetes Secrets, and mention production concerns like version-pinning the underlying Terraform provider to prevent schema drift during regeneration cycles.
◈ Architecture Diagram
┌───────────────────────────────────────────────────────────────┐ │ Upjet Provider Code Generation Pipeline │ ├───────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────┐ │ │ │ Terraform Provider│ │ │ │ Schema (JSON) │ │ │ └────────┬─────────┘ │ │ │ │ │ ↓ │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ Upjet Generator │←──│ Resource Configs │ │ │ │ │ │ (config/*.go) │ │ │ │ ● Schema parser │ │ ● API groups │ │ │ │ ● Type generator │ │ ● Kind names │ │ │ │ ● Controller gen │ │ ● References │ │ │ │ ● CRD generator │ │ ● Sensitive fields │ │ │ └────────┬─────────┘ └──────────────────┘ │ │ │ │ │ │ make generate │ │ ↓ │ │ ┌──────────────────────────────────────────┐ │ │ │ Generated Output │ │ │ │ ┌────────────────┐ ┌──────────────────┐ │ │ │ │ │ apis/v1alpha1/ │ │ internal/ │ │ │ │ │ │ ● types.go │ │ controller/ │ │ │ │ │ │ ● zz_generated │ │ ● zz_controller │ │ │ │ │ │ _deepcopy.go │ │ .go │ │ │ │ │ └────────────────┘ └──────────────────┘ │ │ │ └──────────────────────┬───────────────────┘ │ │ │ │ │ │ make build → make publish │ │ ↓ │ │ ┌──────────────────────────────────────────┐ │ │ │ OCI Provider Package │ │ │ │ ┌────────────────┐ ┌──────────────────┐ │ │ │ │ │ crossplane.yaml│ │ Provider Binary │ │ │ │ │ │ (CRD manifest) │ │ (controller) │ │ │ │ │ └────────────────┘ └──────────────────┘ │ │ │ └──────────────────────┬───────────────────┘ │ │ │ │ │ ↓ │ │ ┌──────────────────────────────────────────┐ │ │ │ Crossplane Control Plane │ │ │ │ ● Provider resource → installs CRDs │ │ │ │ ● ProviderConfig → sets credentials │ │ │ │ ● PaymentsQueue CR → provisions resource │ │ │ └──────────────────────────────────────────┘ │ └───────────────────────────────────────────────────────────────┘
💬 Comments
⚡
Quick Answer
Crossplane Claims (XRCs) provide namespace-scoped interfaces to platform-defined Composite Resources (XRs). Platform engineers create CompositeResourceDefinitions (XRDs) and Compositions that define what infrastructure gets provisioned, while application teams submit Claims in their own namespaces to request resources without needing cluster-level permissions or infrastructure knowledge.
Detailed Answer
Designing a self-service platform with Crossplane Claims creates a clean separation between platform engineering teams who define infrastructure blueprints and application teams who consume them. The architecture follows a producer-consumer model where the platform team authors XRDs and Compositions at the cluster scope, and development teams like payments-api, user-auth-service, and order-processing-service submit Claims in their respective namespaces to provision infrastructure on demand. This model mirrors the organizational boundary between infrastructure ownership and application ownership.
The foundation is the CompositeResourceDefinition (XRD), which defines both the cluster-scoped Composite Resource (XR) and the namespace-scoped Claim (XRC) API. The XRD schema specifies what parameters application teams can configure, such as database size, region, engine version, and backup retention, while hiding implementation details like subnet placement, security group rules, and encryption configuration. For the checkout-service team, the Claim API might expose a simple interface with fields like size (small, medium, large), environment, and team, while the underlying Composition translates these into dozens of managed resources across multiple providers.
Compositions define the implementation behind Claims. A single XRD can have multiple Compositions selected by composition selectors or labels. For example, a PostgreSQLInstance XRD might have compositions for AWS Aurora, GCP Cloud SQL, and Azure Database for PostgreSQL. When the order-processing-service team submits a Claim with a cloud: aws label, Crossplane selects the AWS composition and provisions an RDS Aurora cluster with the appropriate VPC configuration, parameter groups, and IAM authentication. The composition patches map Claim spec fields to managed resource fields, with transforms for value conversion and string formatting.
Namespace-scoped Claims provide natural RBAC boundaries. Kubernetes RBAC controls which teams can create Claims in which namespaces. The payments-api team might have permission to create DatabaseInstance and CacheCluster Claims in the payments namespace, while the user-auth-service team can only create DatabaseInstance Claims in the auth namespace. Connection details such as endpoints, usernames, and passwords are published as Secrets in the Claim's namespace, making them accessible only to pods in that namespace. This prevents cross-team credential leakage without any additional secret management infrastructure.
The self-service workflow integrates with existing developer tools. Teams can define Claims in their GitOps repositories alongside application manifests, so ArgoCD or Flux deploys both the application and its infrastructure dependencies. The platform team maintains a service catalog documenting available Claim types with examples. When the inventory-sync team needs a new Redis cache, they add a CacheCluster Claim to their GitOps repo, open a pull request, and after review and merge, ArgoCD applies the Claim, Crossplane provisions the ElastiCache cluster, and connection details appear as a Secret that the inventory-sync deployment mounts.
Production considerations for self-service platforms include implementing resource quotas using Crossplane's Usage resources or OPA Gatekeeper policies to prevent teams from over-provisioning, setting up Crossplane's composition functions for complex logic that patches alone cannot express (like conditional resource creation based on environment), monitoring Claim readiness with custom Prometheus metrics scraped from the Crossplane controllers, and establishing a clear upgrade path for XRD schema evolution that maintains backward compatibility with existing Claims across the checkout-service, payments-api, and order-processing-service namespaces.
Code Example
# Define the XRD that creates both the XR and the Claim API
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
# Name follows the convention: plural.apigroup
name: postgresqlinstances.database.platform.acme.io
spec:
# API group for the custom database resources
group: database.platform.acme.io
# Resource naming for the composite and claim types
names:
# Kind used for the cluster-scoped Composite Resource
kind: XPostgreSQLInstance
# Plural form for API registration
plural: xpostgresqlinstances
# Enable namespace-scoped Claims for self-service
claimNames:
# Kind used by application teams in their namespaces
kind: PostgreSQLInstance
# Plural form for the Claim API
plural: postgresqlinstances
# Connection secret keys published to the Claim namespace
connectionSecretKeys:
# Database endpoint for application connection strings
- endpoint
# Port number for the PostgreSQL service
- port
# Master username for database authentication
- username
# Master password stored as a Kubernetes Secret
- password
# Schema versions for the Claim API
versions:
# Initial stable version of the database Claim API
- name: v1alpha1
# Mark this version as served by the API server
served: true
# Set as the storage version for etcd persistence
referenceable: true
# OpenAPI schema defining Claim parameters
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
# T-shirt size parameter for application teams
size:
type: string
enum: [small, medium, large]
description: "Database instance size tier"
# Target cloud environment for multi-cloud support
region:
type: string
default: us-east-1
description: "AWS region for database deployment"
required:
- size
---
# Composition mapping Claims to AWS Aurora managed resources
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
# Name identifies this as the AWS implementation
name: postgresqlinstance-aws-aurora
labels:
# Label for composition selection by Claims
provider: aws
# Service tier label for filtering
tier: production
spec:
# Link this composition to the XRD defined above
compositeTypeRef:
apiVersion: database.platform.acme.io/v1alpha1
kind: XPostgreSQLInstance
# Define the managed resources provisioned by this composition
resources:
# RDS Aurora cluster as the primary database resource
- name: aurora-cluster
base:
apiVersion: rds.aws.crossplane.io/v1alpha1
kind: Cluster
spec:
forProvider:
# Aurora PostgreSQL engine for ACID compliance
engine: aurora-postgresql
# Engine version validated by the DBA team
engineVersion: "15.4"
# Skip final snapshot in non-production environments
skipFinalSnapshot: false
# Enable storage encryption with default KMS key
storageEncrypted: true
# Publish connection details to the Claim namespace
writeConnectionSecretToRef:
namespace: crossplane-system
# Patch Claim spec fields into the managed resource
patches:
# Map the region field from the Claim to the resource
- fromFieldPath: spec.region
toFieldPath: spec.forProvider.region
# Transform t-shirt size to Aurora instance class
- fromFieldPath: spec.size
toFieldPath: spec.forProvider.dbClusterInstanceClass
transforms:
- type: map
map:
small: db.r6g.large
medium: db.r6g.xlarge
large: db.r6g.2xlarge
# Map connection detail keys from the provider response
connectionDetails:
- name: endpoint
fromConnectionSecretKey: endpoint
- name: port
fromConnectionSecretKey: port
- name: username
fromConnectionSecretKey: username
- name: password
fromConnectionSecretKey: password
---
# Application team submits this Claim in their namespace
apiVersion: database.platform.acme.io/v1alpha1
kind: PostgreSQLInstance
metadata:
# Descriptive name for the payments database
name: payments-db
# Deployed in the payments team namespace
namespace: payments
spec:
# Request a medium-sized database instance
size: medium
# Deploy in the primary US East region
region: us-east-1
# Connection secret created in the same namespace as the Claim
writeConnectionSecretToRef:
name: payments-db-credentials
---
# RBAC allowing the payments team to create database Claims
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
# Role scoped to the payments namespace
name: database-claim-creator
namespace: payments
rules:
# Allow CRUD operations on PostgreSQLInstance Claims
- apiGroups: ["database.platform.acme.io"]
resources: ["postgresqlinstances"]
verbs: ["get", "list", "create", "update", "patch", "delete"]💡
Interview Tip
A junior engineer typically describes Crossplane as a tool that creates cloud resources from YAML, without distinguishing between managed resources, composite resources, and claims. To stand out, explain the three-layer abstraction: managed resources map one-to-one to cloud APIs, composite resources bundle multiple managed resources with opinionated defaults, and claims provide namespace-scoped self-service interfaces. Walk through how a payments-api developer submits a simple Claim with a size field, the composition translates it into an Aurora cluster with security groups and subnet groups, and connection details appear as a Secret in the developer's namespace. Emphasize how RBAC on Claims provides natural multi-tenancy without custom admission controllers.
◈ Architecture Diagram
┌───────────────────────────────────────────────────────────────┐ │ Crossplane Self-Service Platform Architecture │ ├───────────────────────────────────────────────────────────────┤ │ │ │ Platform Team (cluster-scoped) │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ XRD │ │ Composition │ │ │ │ PostgreSQLInstance│───→│ AWS Aurora impl │ │ │ │ │ │ ● RDS Cluster │ │ │ │ Defines: │ │ ● Subnet Group │ │ │ │ ● size (S/M/L) │ │ ● Security Group │ │ │ │ ● region │ │ ● Parameter Group │ │ │ └──────────────────┘ └──────────────────┘ │ │ │ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ │ │ App Teams (namespace-scoped Claims) │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ namespace: payments │ │ │ │ ┌────────────────┐ ┌────────────────────┐ │ │ │ │ │ Claim: │───→│ Secret: │ │ │ │ │ │ payments-db │ │ payments-db-creds │ │ │ │ │ │ size: medium │ │ ● endpoint │ │ │ │ │ └────────────────┘ │ ● username │ │ │ │ │ │ ● password │ │ │ │ │ └────────────────────┘ │ │ │ └─────────────────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ namespace: orders │ │ │ │ ┌────────────────┐ ┌────────────────────┐ │ │ │ │ │ Claim: │───→│ Secret: │ │ │ │ │ │ orders-db │ │ orders-db-creds │ │ │ │ │ │ size: large │ │ ● endpoint │ │ │ │ │ └────────────────┘ │ ● username │ │ │ │ │ │ ● password │ │ │ │ │ └────────────────────┘ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ Crossplane Control Plane │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Claim → XR → Composition → Managed Resources │ │ │ │ │ │ │ │ payments-db ──→ XPostgreSQLInstance ──→ RDS Cluster │ │ │ │ (namespace) (cluster-scoped) (AWS API) │ │ │ └─────────────────────────────────────────────────────┘ │ └───────────────────────────────────────────────────────────────┘
💬 Comments
⚡
Quick Answer
Deploy a dedicated Crossplane control plane cluster that manages infrastructure across multiple cloud providers, with separate ProviderConfigs for each cloud account. Use GitOps to sync Crossplane resources, implement Composition selection to route Claims to the correct cloud, and use Crossplane's built-in health checks to monitor cross-cloud resource status from a single pane of glass.
Detailed Answer
Implementing Crossplane in a multi-cluster, multi-cloud architecture requires designing a control plane topology that balances centralized governance with operational isolation. The most common production pattern is a dedicated control plane cluster that runs only Crossplane and its providers, separate from workload clusters where applications like payments-api, user-auth-service, and checkout-service run. This separation ensures that control plane failures do not affect running applications and that infrastructure reconciliation loops do not compete with application workloads for cluster resources.
The control plane cluster hosts multiple Crossplane providers simultaneously, typically provider-aws, provider-gcp, and provider-azure, each configured with ProviderConfigs that hold cloud credentials. For a multi-account setup, you create multiple ProviderConfigs per cloud: one for the production AWS account, one for staging, one for the GCP project hosting analytics workloads, and so on. Each managed resource or composition references a specific ProviderConfig via providerConfigRef, directing Crossplane to use the correct credentials when reconciling. The order-processing-service might reference providerConfigRef: aws-production while inventory-sync references providerConfigRef: gcp-analytics, all managed from the same control plane.
Composition selection enables multi-cloud abstraction. A single XRD like XDatabaseInstance can have multiple Compositions: one for AWS Aurora, one for GCP Cloud SQL, and one for Azure Database. The Claim includes a compositionSelector with matchLabels to pick the right implementation. This allows the platform team to define a consistent API across clouds while the underlying provisioning logic differs per provider. When the payments-api team moves from AWS to GCP, they change one label in their Claim rather than rewriting their infrastructure manifests. Composition functions can add runtime logic like checking cloud-specific quotas before provisioning.
Multi-cluster resource distribution requires connecting the Crossplane control plane to workload clusters. Crossplane itself provisions the workload clusters using managed resources like EKS Cluster, GKE Cluster, or AKS Cluster. After cluster creation, Crossplane can deploy applications and configurations into those clusters using provider-kubernetes or provider-helm, which connect to the workload cluster API servers using kubeconfig credentials stored as Secrets. This creates a hierarchy: the control plane cluster provisions workload clusters across clouds, then deploys applications into them. The checkout-service runs on EKS in AWS while user-auth-service runs on GKE in GCP, all orchestrated from the central Crossplane control plane.
Production resilience for multi-cloud Crossplane requires careful credential management, failure domain isolation, and observability. Store cloud credentials in external secret managers like HashiCorp Vault using External Secrets Operator, rotate them automatically, and audit access. Implement circuit breakers for cloud API rate limiting by configuring provider poll intervals and concurrency limits. Monitor Crossplane's reconciliation metrics per provider to detect when a cloud API is degraded. Use Velero to back up the control plane cluster's etcd, because losing the Crossplane state means losing the mapping between desired and actual infrastructure across all clouds. For disaster recovery, maintain a standby control plane cluster that can be promoted if the primary fails, with shared state stored in an external database or replicated etcd cluster.
Code Example
# ProviderConfig for the AWS production account
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
# Identifies this config as the AWS production account
name: aws-production
spec:
credentials:
# Use IRSA (IAM Roles for Service Accounts) for keyless auth
source: IRSA
# Assume a role in the production account from the control plane
assumeRoleChain:
# Cross-account role for infrastructure provisioning
- roleARN: arn:aws:iam::111111111111:role/CrossplaneProvisionerRole
---
# ProviderConfig for the GCP analytics project
apiVersion: gcp.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
# Identifies this config as the GCP analytics project
name: gcp-analytics
spec:
# GCP project where analytics resources are provisioned
projectID: acme-analytics-prod
credentials:
# Pull GCP service account key from a Kubernetes Secret
source: Secret
secretRef:
# Secret containing the GCP service account JSON key
namespace: crossplane-system
name: gcp-analytics-credentials
key: credentials.json
---
# Composition for AWS Aurora (selected when cloud=aws)
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
# AWS-specific implementation of the database XRD
name: xdatabaseinstance-aws-aurora
labels:
# Label used by compositionSelector in Claims
cloud: aws
# Production-grade composition with encryption and backups
tier: production
spec:
compositeTypeRef:
apiVersion: database.platform.acme.io/v1alpha1
kind: XDatabaseInstance
resources:
# AWS RDS Aurora cluster resource
- name: rds-cluster
base:
apiVersion: rds.aws.crossplane.io/v1alpha1
kind: Cluster
spec:
forProvider:
# Aurora PostgreSQL for ACID transactions
engine: aurora-postgresql
# Encryption enabled for compliance
storageEncrypted: true
# Use the AWS production ProviderConfig
providerConfigRef:
name: aws-production
patches:
# Map the Claim's region to the RDS cluster region
- fromFieldPath: spec.region
toFieldPath: spec.forProvider.region
---
# Composition for GCP Cloud SQL (selected when cloud=gcp)
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
# GCP-specific implementation of the database XRD
name: xdatabaseinstance-gcp-cloudsql
labels:
# Label used by compositionSelector in Claims
cloud: gcp
# Production-grade composition
tier: production
spec:
compositeTypeRef:
apiVersion: database.platform.acme.io/v1alpha1
kind: XDatabaseInstance
resources:
# GCP Cloud SQL instance resource
- name: cloudsql-instance
base:
apiVersion: sql.gcp.crossplane.io/v1beta1
kind: DatabaseInstance
spec:
forProvider:
# PostgreSQL engine for compatibility with AWS Aurora
databaseVersion: POSTGRES_15
settings:
# High availability configuration for production
availabilityType: REGIONAL
# Enable automated backups
backupConfiguration:
enabled: true
# Use the GCP analytics ProviderConfig
providerConfigRef:
name: gcp-analytics
patches:
# Map the Claim's region to the GCP location
- fromFieldPath: spec.region
toFieldPath: spec.forProvider.region
---
# Claim from order-processing-service requesting AWS database
apiVersion: database.platform.acme.io/v1alpha1
kind: DatabaseInstance
metadata:
# Database for the order processing service
name: orders-db
namespace: orders
spec:
# Request a medium-sized database
size: medium
# Deploy in AWS US East region
region: us-east-1
# Select the AWS composition via label matching
compositionSelector:
matchLabels:
cloud: aws
# Connection secret for the orders namespace
writeConnectionSecretToRef:
name: orders-db-credentials💡
Interview Tip
A junior engineer typically describes multi-cloud as installing multiple providers on Crossplane, without addressing control plane topology, credential isolation, or composition selection patterns. To demonstrate expertise, explain the dedicated control plane cluster pattern and why it is separated from workload clusters. Walk through how ProviderConfigs isolate credentials per cloud account, how compositionSelector routes Claims to cloud-specific implementations, and how provider-kubernetes deploys into remote workload clusters. Discuss disaster recovery for the control plane: etcd backups, standby clusters, and what happens if the control plane loses connectivity to a cloud provider while managed resources exist.
◈ Architecture Diagram
┌───────────────────────────────────────────────────────────────┐ │ Multi-Cluster Multi-Cloud Crossplane Architecture │ ├───────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Crossplane Control Plane Cluster │ │ │ │ │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │ │ │ │provider │ │provider │ │provider │ │ │ │ │ │-aws │ │-gcp │ │-kubernetes │ │ │ │ │ └────┬─────┘ └────┬─────┘ └────────┬─────────┘ │ │ │ │ │ │ │ │ │ │ │ ┌────┴─────┐ ┌────┴─────┐ │ │ │ │ │ │PCfg: │ │PCfg: │ │ │ │ │ │ │aws-prod │ │gcp-anlyt │ │ │ │ │ │ │aws-stage │ │gcp-prod │ │ │ │ │ │ └──────────┘ └──────────┘ │ │ │ │ └──────────┬──────────┬─────────────────┼────────────┘ │ │ │ │ │ │ │ ┌───────┘ │ └──────────┐ │ │ ↓ ↓ ↓ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │ │ AWS Cloud │ │ GCP Cloud │ │ Workload │ │ │ │ │ │ │ │ Clusters │ │ │ │ ┌──────────┐ │ │ ┌──────────┐ │ │ │ │ │ │ │EKS Cluster│ │ │ │GKE Cluster│ │ │ ┌──────────────┐ │ │ │ │ │payments- │ │ │ │user-auth-│ │ │ │ EKS: checkout│ │ │ │ │ │api │ │ │ │service │ │ │ │ -service │ │ │ │ │ └──────────┘ │ │ └──────────┘ │ │ └──────────────┘ │ │ │ │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────────┐ │ │ │ │ │RDS Aurora │ │ │ │Cloud SQL │ │ │ │ GKE: order- │ │ │ │ │ │orders-db │ │ │ │auth-db │ │ │ │ processing │ │ │ │ │ └──────────┘ │ │ └──────────┘ │ │ └──────────────┘ │ │ │ └──────────────┘ └──────────────┘ └──────────────────┘ │ │ │ │ Claim Routing: │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Claim(cloud:aws) ──→ Composition(aws-aurora) ──→ RDS│ │ │ │ Claim(cloud:gcp) ──→ Composition(gcp-cloudsql)──→SQL│ │ │ └─────────────────────────────────────────────────────┘ │ └───────────────────────────────────────────────────────────────┘
💬 Comments
⚡
Quick Answer
Manage Crossplane upgrades using a staged rollout strategy: test in a non-production control plane first, upgrade the Crossplane core via Helm with careful CRD migration, then upgrade providers individually using revision activation policies. Use provider version pinning, revision history limits, and automated health checks to enable safe rollbacks if a provider upgrade introduces regressions.
Detailed Answer
Handling Crossplane upgrades and provider version management in production requires a systematic approach because the Crossplane ecosystem has three independent versioning axes: the Crossplane core runtime, individual providers, and the Compositions that define infrastructure blueprints. A careless upgrade can break reconciliation for thousands of managed resources across services like payments-api, checkout-service, and inventory-sync, so organizations must treat Crossplane upgrades with the same rigor as Kubernetes control plane upgrades.
Crossplane core upgrades are managed through Helm. The Crossplane Helm chart deploys the core controllers, RBAC resources, and webhook configurations. Before upgrading, review the release notes for breaking changes in the API extensions, package manager, or composition engine. CRD schema changes are particularly sensitive because they affect how existing XRDs and Compositions are validated. The upgrade process starts with applying the new CRDs before upgrading the controllers, which the Helm chart handles automatically. However, if you have custom webhook configurations or admission controllers that validate Crossplane resources, these must be compatible with the new CRD schemas before the upgrade proceeds.
Provider version management leverages Crossplane's package revision system. Each provider version is a separate PackageRevision object in the cluster. The Provider resource's revisionActivationPolicy controls how new versions are activated. With the Automatic policy, installing a new provider version immediately activates it and deactivates the old revision. With the Manual policy, the new revision is installed but remains inactive until explicitly activated, allowing the platform team to test compatibility before switching. For production environments managing order-processing-service and user-auth-service infrastructure, the Manual activation policy provides a safety net.
The staged upgrade process follows a pipeline pattern. First, upgrade in a development control plane cluster that mirrors production's provider configurations and compositions but manages sandbox cloud accounts. Run comprehensive tests that provision and destroy sample resources for each composition to verify the new provider version handles the full lifecycle correctly. Second, upgrade in a staging control plane cluster that manages pre-production infrastructure. Monitor for reconciliation errors over a burn-in period of at least 24 hours. Third, upgrade in the production control plane using the Manual activation policy. Activate the new provider revision, monitor reconciliation metrics, and keep the previous revision available for instant rollback.
Provider revision rollback is straightforward but must be tested. Each provider maintains a revision history controlled by revisionHistoryLimit. If a new provider-aws version introduces a regression that breaks RDS cluster reconciliation for the payments-api database, you can revert by changing the Provider resource to reference the previous version. Crossplane deactivates the current revision and reactivates the previous one. However, if the new provider version modified the CRD schema, rolling back might require manual CRD patching because Kubernetes does not automatically downgrade CRDs.
Automated version management integrates with GitOps. Store Provider resources in a Git repository and use Renovate Bot or Dependabot to create pull requests when new provider versions are released. The CI pipeline runs integration tests against the proposed provider version before the PR is merged. Use Kyverno or OPA Gatekeeper policies to enforce that all Provider resources in the production control plane must pin exact versions rather than floating tags, preventing unexpected provider updates. Monitor the crossplane_provider_revision_health metric to alert when a provider revision enters an unhealthy state after an upgrade.
Code Example
# Provider resource with version pinning and manual activation
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
# AWS provider for managing production infrastructure
name: provider-aws
spec:
# Pin to an exact version for reproducible deployments
package: xpkg.upbound.io/upbound/provider-aws-ec2:v1.2.1
# Manual activation requires explicit approval after install
revisionActivationPolicy: Manual
# Keep 3 revisions for rollback capability
revisionHistoryLimit: 3
# Controller configuration for resource limits
runtimeConfigRef:
name: provider-aws-runtime
---
# Runtime config to tune provider controller resources
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
# Runtime config for the AWS provider controller
name: provider-aws-runtime
spec:
deploymentTemplate:
spec:
# Set replica count for high availability
replicas: 2
template:
spec:
containers:
- name: package-runtime
resources:
limits:
# Memory limit to prevent OOM during large reconciliation cycles
memory: 1Gi
# CPU limit for predictable scheduling
cpu: 500m
requests:
# Memory request for baseline operation
memory: 512Mi
# CPU request for guaranteed scheduling
cpu: 250m
---
# Helm values for Crossplane core upgrade
# helm upgrade crossplane crossplane-stable/crossplane \
# --namespace crossplane-system \
# --version 1.15.0 \
# --set args='{--enable-composition-functions}' \
# --set metrics.enabled=true \
# --wait --timeout 5m
# Check provider revision status after upgrade
kubectl get providerrevisions -o wide
# Activate a specific provider revision manually
kubectl patch providerrevision provider-aws-abc123 \
--type merge -p '{"spec":{"desiredState":"Active"}}'
# Roll back to previous revision if issues detected
kubectl patch providerrevision provider-aws-xyz789 \
--type merge -p '{"spec":{"desiredState":"Active"}}'
# Monitor reconciliation health after provider upgrade
kubectl get managed -o wide | grep -v 'True'
# Check for resources stuck in a non-ready state
kubectl get managed -o json | jq '.items[] | select(.status.conditions[]? | select(.type=="Ready" and .status!="True")) | .metadata.name'
# Verify no CRD schema conflicts after upgrade
kubectl get crds -l pkg.crossplane.io/package=provider-aws -o name | wc -l
# Kyverno policy to enforce exact version pinning on Providers
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
# Policy name describing its enforcement purpose
name: require-provider-version-pin
spec:
# Block non-compliant Provider resources from being created
validationFailureAction: Enforce
rules:
# Rule to validate provider package version format
- name: check-version-pin
match:
resources:
# Apply to all Provider resources
kinds:
- Provider
validate:
message: "Provider package must use an exact version tag (e.g., v1.2.1)"
pattern:
spec:
# Require the package reference to contain a version tag
package: "*:v*"💡
Interview Tip
A junior engineer typically says they upgrade Crossplane by running helm upgrade and updating the provider version, without discussing the risks or rollback strategy. To demonstrate production experience, explain the three versioning axes (core, providers, compositions) and why they must be upgraded independently. Describe the Manual revision activation policy that lets you install a new provider version without immediately activating it, the staged rollout through dev, staging, and production control planes, and how provider revision history enables instant rollback. Mention CRD schema compatibility as the most dangerous aspect of upgrades because Kubernetes does not automatically downgrade CRDs, and describe how Kyverno policies enforce exact version pinning in production.
◈ Architecture Diagram
┌───────────────────────────────────────────────────────────────┐ │ Crossplane Upgrade and Rollback Pipeline │ ├───────────────────────────────────────────────────────────────┤ │ │ │ Version Axes: │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │ │Crossplane │ │Provider │ │Compositions │ │ │ │Core v1.15.0 │ │-aws v1.2.1 │ │XRD v1alpha1 │ │ │ └──────────────┘ └──────────────┘ └──────────────────┘ │ │ │ │ Staged Rollout: │ │ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │ │ Dev │───→│ Staging │───→│ Production │ │ │ │ Control │ │ Control │ │ Control Plane │ │ │ │ Plane │ │ Plane │ │ │ │ │ │ │ │ │ │ Manual Activation│ │ │ │ Auto │ │ Auto │ │ Policy │ │ │ │ Activate │ │ Activate │ │ │ │ │ │ │ │ 24hr burn │ │ ✓ Activate new │ │ │ │ Run tests │ │ -in │ │ ✗ Rollback if │ │ │ └──────────┘ └──────────┘ │ issues │ │ │ └──────────────────┘ │ │ │ │ Provider Revision Management: │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Provider: provider-aws │ │ │ │ │ │ │ │ Revision 1 (v1.1.0) ── Inactive ── Keep for rollback│ │ │ │ Revision 2 (v1.2.0) ── Inactive ── Previous version │ │ │ │ Revision 3 (v1.2.1) ── Active ──── Current version │ │ │ │ │ │ │ │ revisionHistoryLimit: 3 │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ Rollback Flow: │ │ ┌────────────┐ ┌────────────┐ ┌────────────────┐ │ │ │ Detect │───→│ Deactivate │───→│ Reactivate │ │ │ │ regression │ │ v1.2.1 │ │ v1.2.0 │ │ │ │ (metrics) │ │ revision │ │ revision │ │ │ └────────────┘ └────────────┘ └────────────────┘ │ └───────────────────────────────────────────────────────────────┘
💬 Comments
⚡
Quick Answer
Implement Crossplane security using layered RBAC: cluster-scoped roles for platform engineers who manage XRDs, Compositions, and Providers; namespace-scoped roles for application teams who create Claims; and policy engines like OPA Gatekeeper or Kyverno to enforce guardrails on resource specifications. Combine with credential isolation through ProviderConfigs and audit logging through Kubernetes API server audit policies.
Detailed Answer
Implementing RBAC and security policies for Crossplane in enterprise environments requires addressing four security domains: API access control (who can create what resources), credential isolation (which cloud accounts each team can provision into), policy enforcement (what resource configurations are allowed), and audit logging (who did what and when). In a large organization where payments-api, user-auth-service, checkout-service, and inventory-sync teams all share a Crossplane control plane, these security layers prevent privilege escalation, credential leakage, and non-compliant infrastructure provisioning.
Kubernetes RBAC forms the foundation of Crossplane access control. The key insight is that Crossplane's three-layer abstraction (managed resources, composite resources, claims) maps naturally to RBAC tiers. Platform engineers need cluster-scoped permissions to create and modify XRDs, Compositions, Providers, and ProviderConfigs. These are the most privileged operations because they define what infrastructure can be provisioned and with which credentials. Application teams need only namespace-scoped permissions to create Claims in their namespaces. They never interact with managed resources or composite resources directly. This separation means the checkout-service team can request a PostgreSQL database by creating a Claim but cannot modify the Composition to change the instance size limits or the ProviderConfig to access a different AWS account.
Credential isolation through ProviderConfigs is critical for multi-tenant security. Each ProviderConfig binds to specific cloud credentials, and Compositions reference ProviderConfigs by name. By controlling which ProviderConfigs exist and which Compositions reference them, platform engineers control which cloud accounts each Composition can provision into. For enterprise environments, use IRSA (IAM Roles for Service Accounts) on AWS or Workload Identity on GCP instead of static credentials stored in Secrets. These mechanisms provide short-lived, automatically-rotated credentials that are auditable through cloud provider logs. Restrict who can create or modify ProviderConfig resources to a small group of platform security engineers.
Policy enforcement through OPA Gatekeeper or Kyverno adds guardrails that RBAC alone cannot provide. While RBAC controls who can create a Claim, policies control what values the Claim can contain. For example, a Gatekeeper ConstraintTemplate can enforce that all database Claims must have storageEncrypted set to true, that the size field cannot exceed 'large' for non-production namespaces, that Claims in the payments namespace must use the aws-production ProviderConfig (not the staging one), and that all managed resources must include cost-center and team tags. These policies run as admission webhooks and reject non-compliant resources before they reach Crossplane's reconciliation loop.
Audit logging captures the complete lifecycle of infrastructure changes through Crossplane. Configure the Kubernetes API server audit policy to log all create, update, and delete operations on Crossplane resource types including Claims, XRs, Compositions, and managed resources. Enrich audit logs with the requesting user's identity, namespace, and the specific fields that changed. Forward audit logs to a SIEM like Splunk or Elastic for correlation with cloud provider audit trails (CloudTrail, Cloud Audit Logs). This creates an end-to-end audit chain: the order-processing-service developer who submitted a database Claim is linked to the specific AWS API calls that provisioned the RDS cluster, through the Kubernetes audit log and CloudTrail entries sharing the Crossplane controller's IAM role session name.
Network policies and pod security standards for the Crossplane control plane itself are often overlooked. The provider controller pods run with cloud credentials and should be isolated in the crossplane-system namespace with restrictive NetworkPolicies that allow only egress to cloud API endpoints and the Kubernetes API server. Implement Pod Security Standards at the Restricted level for provider pods, and use runtime security tools like Falco to detect unexpected process execution or network connections from provider containers.
Code Example
# ClusterRole for platform engineers managing Crossplane internals
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
# Role for platform engineers with full Crossplane management access
name: crossplane-platform-admin
rules:
# Full access to XRDs for defining platform APIs
- apiGroups: ["apiextensions.crossplane.io"]
resources: ["compositeresourcedefinitions", "compositions", "compositionrevisions"]
verbs: ["*"]
# Full access to provider and package management
- apiGroups: ["pkg.crossplane.io"]
resources: ["providers", "providerrevisions", "configurations", "functions"]
verbs: ["*"]
# Access to view all managed resources across providers
- apiGroups: ["rds.aws.crossplane.io", "ec2.aws.crossplane.io", "sql.gcp.crossplane.io"]
resources: ["*"]
verbs: ["get", "list", "watch"]
---
# Namespace-scoped Role for application teams creating Claims
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
# Role for the payments team to manage their infrastructure Claims
name: payments-claim-manager
namespace: payments
rules:
# Allow creating and managing database Claims
- apiGroups: ["database.platform.acme.io"]
resources: ["postgresqlinstances"]
verbs: ["get", "list", "create", "update", "patch", "delete"]
# Allow creating and managing cache Claims
- apiGroups: ["cache.platform.acme.io"]
resources: ["redisclusters"]
verbs: ["get", "list", "create", "update", "patch", "delete"]
# Allow reading connection secrets produced by Claims
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]
resourceNames: ["payments-db-credentials", "payments-cache-credentials"]
---
# RoleBinding granting the payments team their Claim permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
# Bind the payments team group to their Claim role
name: payments-team-claims
namespace: payments
subjects:
# OIDC group from the corporate identity provider
- kind: Group
name: team-payments-api
apiGroup: rbac.authorization.k8s.io
roleRef:
# Reference the namespace-scoped claim manager role
kind: Role
name: payments-claim-manager
apiGroup: rbac.authorization.k8s.io
---
# OPA Gatekeeper ConstraintTemplate enforcing encryption on databases
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
# Template for enforcing database encryption policy
name: requiredbencryption
spec:
crd:
spec:
names:
# Kind used when creating constraint instances
kind: RequireDBEncryption
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
# Rego policy enforcing storage encryption on database Claims
package requiredbencryption
# Deny Claims that do not request encrypted storage
violation[{"msg": msg}] {
# Match PostgreSQLInstance Claims
input.review.object.kind == "PostgreSQLInstance"
# Check if encryption is explicitly disabled
not input.review.object.spec.encrypted
# Construct the violation message with resource details
msg := sprintf("Database Claim %v must have encryption enabled", [input.review.object.metadata.name])
}
---
# Constraint applying the encryption policy to all namespaces
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequireDBEncryption
metadata:
# Constraint enforcing encryption across all database Claims
name: all-databases-must-be-encrypted
spec:
match:
kinds:
# Apply to PostgreSQLInstance Claims in all namespaces
- apiGroups: ["database.platform.acme.io"]
kinds: ["PostgreSQLInstance"]
# Enforcement action blocks non-compliant resources
enforcementAction: deny
---
# NetworkPolicy isolating Crossplane provider pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
# Restrict egress from provider pods to cloud APIs only
name: provider-egress-restriction
namespace: crossplane-system
spec:
# Apply to all provider controller pods
podSelector:
matchLabels:
pkg.crossplane.io/revision: provider-aws
policyTypes:
- Egress
egress:
# Allow egress to AWS API endpoints over HTTPS
- ports:
- port: 443
protocol: TCP
# Allow egress to Kubernetes API server
- to:
- ipBlock:
cidr: 10.0.0.1/32
ports:
- port: 443
protocol: TCP💡
Interview Tip
A junior engineer typically says Crossplane uses standard Kubernetes RBAC and stops there, without explaining the tiered permission model or policy enforcement. To demonstrate enterprise security expertise, describe how Crossplane's three-layer abstraction maps to RBAC tiers: cluster-scoped roles for platform engineers managing XRDs and Compositions, namespace-scoped roles for application teams creating Claims, and no direct access to managed resources for anyone except platform admins. Explain credential isolation through ProviderConfigs with IRSA or Workload Identity, policy enforcement with Gatekeeper ConstraintTemplates that validate Claim fields before admission, and the audit chain from Kubernetes audit logs through to CloudTrail entries. Mention NetworkPolicies for provider pod isolation as a defense-in-depth measure.
◈ Architecture Diagram
┌───────────────────────────────────────────────────────────────┐ │ Crossplane Enterprise Security Architecture │ ├───────────────────────────────────────────────────────────────┤ │ │ │ RBAC Tiers: │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Tier 1: Platform Admins (ClusterRole) │ │ │ │ ● XRDs, Compositions, Providers, ProviderConfigs │ │ │ │ ● Full control over platform API definitions │ │ │ └─────────────────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Tier 2: App Teams (namespace Role) │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │ │ │ │payments │ │orders │ │checkout │ │ │ │ │ │namespace │ │namespace │ │namespace │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │Claims │ │Claims │ │Claims only │ │ │ │ │ │only │ │only │ │ │ │ │ │ │ └──────────┘ └──────────┘ └──────────────────┘ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ Policy Enforcement: │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ OPA Gatekeeper / Kyverno │ │ │ │ │ │ │ │ Claim ──→ Admission Webhook ──→ Policy Check │ │ │ │ │ │ │ │ │ ├── ✓ Encryption enabled │ │ │ │ ├── ✓ Size within quota │ │ │ │ ├── ✓ Tags present │ │ │ │ └── ✗ Rejected if non-compliant │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ Credential Isolation: │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ ProviderConfig: aws-prod ──→ IRSA Role (prod acct) │ │ │ │ ProviderConfig: aws-stage──→ IRSA Role (stage acct) │ │ │ │ ProviderConfig: gcp-prod ──→ Workload Identity │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ Audit Chain: │ │ ┌────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │ │ K8s Audit │→│ SIEM │←│ CloudTrail / │ │ │ │ Log │ │ (Correlation) │ │ Cloud Audit Log │ │ │ │ (who/what) │ │ │ │ (API calls) │ │ │ └────────────┘ └──────────────┘ └──────────────────┘ │ └───────────────────────────────────────────────────────────────┘
💬 Comments
⚡
Quick Answer
Optimize Crossplane for large-scale deployments by tuning provider controller concurrency and poll intervals, splitting providers into family packages to reduce memory footprint, implementing resource-level annotations for poll interval overrides, using DeploymentRuntimeConfig for resource limits, and sharding managed resources across multiple ProviderConfigs to distribute cloud API rate limit consumption.
Detailed Answer
Optimizing Crossplane performance for managing thousands of resources requires understanding the reconciliation mechanics of both the Crossplane core and individual provider controllers. Each managed resource in Crossplane is watched by a controller that periodically polls the cloud API to detect drift and reconcile toward the desired state. When you scale to thousands of resources across services like payments-api, order-processing-service, checkout-service, user-auth-service, and inventory-sync, the default controller settings become bottlenecks: cloud API rate limits are exhausted, controller pods consume excessive memory caching resource state, and reconciliation latency increases as the controller work queue grows.
The first optimization is tuning provider controller concurrency. Each provider controller has a maxReconcileRate parameter that controls how many resources can be reconciled simultaneously. The default is typically 10, which is conservative for production. Increasing this to 50 or 100 allows the controller to process more resources in parallel, but you must balance this against cloud API rate limits. AWS, for example, has per-service rate limits that vary by API: EC2 DescribeInstances allows 100 requests per second, but RDS DescribeDBClusters allows only 25. If the payments-api team has 200 RDS-related managed resources and the controller tries to reconcile them all at maxReconcileRate: 100, it will hit rate limits and trigger exponential backoff, actually slowing down reconciliation.
The second optimization involves using Crossplane's provider family packages instead of monolithic providers. The monolithic provider-aws installs CRDs for all AWS services (500+ CRDs), even if you only use EC2, RDS, and S3. Each CRD consumes API server memory and increases watch establishment time. Provider family packages like provider-aws-ec2, provider-aws-rds, and provider-aws-s3 install only the CRDs for the services you use. For a deployment managing infrastructure for checkout-service (EC2, ALB), order-processing-service (SQS, RDS), and inventory-sync (S3, DynamoDB), you install only four provider family packages instead of the monolithic provider, reducing CRD count from 500+ to under 50.
The third optimization is poll interval management. Crossplane's default poll interval is 1 minute, meaning every managed resource makes a cloud API call every 60 seconds. For stable, long-lived resources like VPCs, subnets, and IAM roles that rarely change, this is wasteful. Use the crossplane.io/paused annotation to completely stop reconciliation on resources that should not change, or configure longer poll intervals at the provider level. Some providers support resource-level poll interval overrides via annotations, allowing you to set a 10-second poll for actively changing resources (like a database being restored) and a 10-minute poll for stable resources.
The fourth optimization is etcd and API server tuning. Thousands of managed resources mean thousands of objects in etcd, each with status subresources that update on every reconciliation cycle. This generates significant write I/O in etcd and watch event traffic through the API server. Tune etcd's --quota-backend-bytes to accommodate the larger dataset, increase the API server's --max-requests-inflight for higher concurrent watch connections, and consider using an etcd cluster with SSD-backed storage. Monitor etcd's database size and compaction frequency to prevent performance degradation.
The fifth optimization is sharding and resource distribution. For very large deployments, split managed resources across multiple ProviderConfigs that point to different cloud API endpoints or accounts. This distributes API rate limit consumption across multiple credential sets. Additionally, consider running multiple instances of the same provider with different ProviderConfig assignments, using label selectors to partition which resources each instance reconciles. This horizontal scaling approach allows the platform team to manage 10,000+ resources across the order-processing-service, payments-api, and user-auth-service without any single controller instance becoming a bottleneck.
Code Example
# DeploymentRuntimeConfig for tuning provider controller performance
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
# Runtime config optimized for large-scale resource management
name: provider-aws-highperf
spec:
deploymentTemplate:
spec:
# Run 2 replicas for high availability
replicas: 2
template:
spec:
containers:
- name: package-runtime
# Set controller concurrency via command arguments
args:
# Increase max reconcile rate for high resource counts
- --max-reconcile-rate=50
# Set poll interval to 5 minutes for stable resources
- --poll-interval=5m
# Enable leader election for multi-replica safety
- --leader-election=true
resources:
limits:
# 2Gi memory for caching thousands of resource states
memory: 2Gi
# 1 CPU core for reconciliation processing
cpu: "1"
requests:
# Baseline memory for controller operation
memory: 1Gi
# Baseline CPU for steady-state reconciliation
cpu: 500m
---
# Provider using family packages instead of monolithic provider
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
# EC2-specific provider for compute resources only
name: provider-aws-ec2
spec:
# Install only EC2 CRDs instead of all 500+ AWS CRDs
package: xpkg.upbound.io/upbound/provider-aws-ec2:v1.2.1
# Use the high-performance runtime configuration
runtimeConfigRef:
name: provider-aws-highperf
---
# Provider for RDS resources used by payments-api and order-processing
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
# RDS-specific provider for database resources only
name: provider-aws-rds
spec:
# Install only RDS CRDs for database management
package: xpkg.upbound.io/upbound/provider-aws-rds:v1.2.1
# Use the high-performance runtime configuration
runtimeConfigRef:
name: provider-aws-highperf
---
# Pause reconciliation on stable resources that should not change
kubectl annotate managed vpc-payments-production \
crossplane.io/paused="true" --overwrite
# Resume reconciliation when maintenance is needed
kubectl annotate managed vpc-payments-production \
crossplane.io/paused- --overwrite
# Monitor reconciliation queue depth via Prometheus
# promql: workqueue_depth{name="managed/rds.aws.crossplane.io"}
# Monitor API rate limit errors from provider controllers
# promql: rate(controller_runtime_reconcile_errors_total{controller=~"managed/.*"}[5m])
# Check resource count per provider for capacity planning
kubectl get managed -o json | jq '[.items[] | .apiVersion] | group_by(.) | map({api: .[0], count: length}) | sort_by(-.count)'
# Monitor etcd database size for capacity planning
# etcdctl endpoint status --write-out=table
# Verify provider controller memory usage is within limits
kubectl top pods -n crossplane-system -l pkg.crossplane.io/revision💡
Interview Tip
A junior engineer typically says Crossplane can manage cloud resources without discussing the performance implications of scale. To demonstrate depth, explain the reconciliation mechanics: each managed resource triggers a cloud API call on every poll interval, so 1000 resources at a 1-minute poll means 1000 API calls per minute per provider. Walk through the optimization layers: family packages to reduce CRD count, maxReconcileRate tuning balanced against cloud API rate limits, poll interval increases for stable resources, and the crossplane.io/paused annotation for frozen resources. Discuss etcd pressure from status subresource updates and how to monitor reconciliation health using Prometheus metrics like workqueue_depth and reconcile_errors_total. Mention provider sharding as the horizontal scaling strategy for managing 10,000+ resources.
◈ Architecture Diagram
┌───────────────────────────────────────────────────────────────┐ │ Crossplane Performance Optimization Layers │ ├───────────────────────────────────────────────────────────────┤ │ │ │ Layer 1: Provider Family Packages │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Before: provider-aws (500+ CRDs, 2Gi memory) │ │ │ │ After: provider-aws-ec2 (20 CRDs, 256Mi) │ │ │ │ provider-aws-rds (15 CRDs, 256Mi) │ │ │ │ provider-aws-s3 (10 CRDs, 128Mi) │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ Layer 2: Reconciliation Tuning │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ maxReconcileRate: 50 │ │ │ │ │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │ │ │ │ Resource │ │Controller│ │ Cloud API │ │ │ │ │ │ Queue │───→│ Workers │───→│ (rate-limited)│ │ │ │ │ │ (1000) │ │ (50) │ │ │ │ │ │ │ └──────────┘ └──────────┘ └──────────────┘ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ Layer 3: Poll Interval Optimization │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Active resources: poll every 1m (databases, LBs) │ │ │ │ Stable resources: poll every 10m (VPCs, subnets) │ │ │ │ Frozen resources: paused (legacy infra) │ │ │ │ │ │ │ │ 1000 resources × 1/min = 1000 API calls/min ✗ │ │ │ │ 200 active × 1/min + 800 stable × 1/10min │ │ │ │ = 280 API calls/min ✓ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ Layer 4: Sharding │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ ┌──────────────┐ ┌──────────────┐ │ │ │ │ │Provider Inst 1│ │Provider Inst 2│ │ │ │ │ │PCfg: aws-prod│ │PCfg: aws-stg │ │ │ │ │ │500 resources │ │500 resources │ │ │ │ │ └──────────────┘ └──────────────┘ │ │ │ │ Separate rate limit pools per credential set │ │ │ └─────────────────────────────────────────────────────┘ │ └───────────────────────────────────────────────────────────────┘
💬 Comments
⚡
Quick Answer
Implement cost tracking by tagging all managed resources with team, project, and environment labels through Composition patches, then aggregating costs via cloud provider billing APIs. Enforce resource quotas using Crossplane's Usage resources, OPA Gatekeeper policies that count existing Claims per namespace, and Composition-level transforms that map t-shirt sizes to predefined instance types, preventing teams from requesting arbitrarily expensive resources.
Detailed Answer
Implementing usage-based cost tracking and resource quotas with Crossplane requires a combination of cloud-native billing integration, Kubernetes-native policy enforcement, and Crossplane's own resource management features. In an enterprise where teams like payments-api, order-processing-service, checkout-service, user-auth-service, and inventory-sync provision infrastructure through Claims, the platform team needs visibility into which team is spending how much and the ability to set guardrails that prevent runaway costs.
Cost tracking starts with consistent tagging. Every managed resource provisioned through Crossplane must carry tags that identify the owning team, project, environment, and cost center. Compositions are the enforcement point: the platform team adds patches that automatically propagate labels from the Claim to tags on every managed resource in the Composition. When the payments-api team creates a database Claim in the payments namespace, the Composition patches the namespace, Claim name, and team label into AWS resource tags. These tags flow through to AWS Cost Explorer, GCP Billing, or Azure Cost Management, enabling per-team cost allocation without relying on developers to remember to add tags.
For real-time cost visibility, integrate cloud billing APIs with a cost management platform like Kubecost, Vantage, or Infracost. These tools can correlate Crossplane-managed resources (identified by their tags) with actual cloud spending. The platform team builds dashboards showing cost trends per team, per environment, and per resource type. Alerts fire when a team's monthly spending exceeds a threshold. For pre-provisioning cost estimation, Infracost can analyze Crossplane Compositions and estimate the monthly cost of each Claim type. The checkout-service team sees that a 'large' database Claim costs approximately $2,400 per month before they submit it, enabling informed decisions about resource sizing.
Resource quotas in Crossplane operate at multiple levels. The first level is Composition-level constraints: t-shirt size mappings (small, medium, large) in Compositions prevent teams from requesting arbitrary instance types. The Composition's transform map converts 'large' to db.r6g.2xlarge, and there is no way for the checkout-service team to request a db.r6g.16xlarge because the Composition does not include that mapping. The second level is namespace-level quotas enforced through OPA Gatekeeper or Kyverno policies. A policy can count the number of existing PostgreSQLInstance Claims in a namespace and reject new Claims if the count exceeds the quota. This prevents the inventory-sync team from creating 50 database instances when their allocation is 5.
Crossplane's Usage resource provides dependency-based protection rather than quotas, but it can be combined with quotas for comprehensive resource governance. A Usage resource declares that a managed resource is in use by another resource, preventing accidental deletion. For cost governance, Usage resources ensure that expensive shared infrastructure like a Transit Gateway or a dedicated VPN connection cannot be deleted while dependent resources from the payments-api or order-processing-service teams still reference it.
Advanced cost governance integrates with the organization's FinOps workflow. The platform team publishes a service catalog with cost estimates for each Claim type and size combination. Budget owners approve Claims that exceed a threshold amount through a custom approval webhook. Monthly cost reports are generated by querying both the cloud billing APIs and the Crossplane control plane to correlate spending with specific Claims and the teams that created them. Chargeback or showback models assign costs to business units based on the tags propagated through Compositions, creating accountability for infrastructure spending across the checkout-service, payments-api, user-auth-service, and inventory-sync teams.
Code Example
# Composition with automatic cost-tracking tags on all resources
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
# Composition with built-in cost tracking tags
name: postgresqlinstance-aws-with-cost-tags
spec:
compositeTypeRef:
apiVersion: database.platform.acme.io/v1alpha1
kind: XPostgreSQLInstance
resources:
# RDS cluster with cost-tracking tags patched from Claim metadata
- name: aurora-cluster
base:
apiVersion: rds.aws.crossplane.io/v1alpha1
kind: Cluster
spec:
forProvider:
# Aurora PostgreSQL engine
engine: aurora-postgresql
# Enable encryption for compliance
storageEncrypted: true
# Default tags applied to all RDS resources
tags:
# Static tag identifying Crossplane as the provisioner
managed-by: crossplane
# Static tag for the platform team's cost center
platform: acme-internal-platform
patches:
# Patch the Claim's namespace as the team identifier tag
- fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
toFieldPath: spec.forProvider.tags.team-namespace
# Patch the Claim name as the resource identifier tag
- fromFieldPath: metadata.labels[crossplane.io/claim-name]
toFieldPath: spec.forProvider.tags.claim-name
# Patch the environment label for cost segmentation
- fromFieldPath: spec.environment
toFieldPath: spec.forProvider.tags.environment
# Transform t-shirt size to instance class (cost control)
- fromFieldPath: spec.size
toFieldPath: spec.forProvider.dbClusterInstanceClass
transforms:
- type: map
map:
# small: ~$200/month (db.r6g.large)
small: db.r6g.large
# medium: ~$800/month (db.r6g.xlarge)
medium: db.r6g.xlarge
# large: ~$2400/month (db.r6g.2xlarge)
large: db.r6g.2xlarge
---
# Kyverno policy enforcing maximum Claim count per namespace
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
# Policy limiting the number of database Claims per namespace
name: limit-database-claims-per-namespace
spec:
# Block Claims that exceed the namespace quota
validationFailureAction: Enforce
rules:
# Rule to count existing Claims and enforce the limit
- name: max-five-databases-per-namespace
match:
resources:
# Apply to PostgreSQLInstance Claims
kinds:
- PostgreSQLInstance
# Context loads existing Claims for counting
context:
- name: claimCount
apiCall:
# Query existing Claims in the request namespace
urlPath: /apis/database.platform.acme.io/v1alpha1/namespaces/{{request.namespace}}/postgresqlinstances
# Extract the count of existing Claims
jmesPath: "items | length(@)"
validate:
# Deny if existing count is already at the limit
message: "Namespace {{request.namespace}} already has {{claimCount}} database Claims (max 5)"
deny:
conditions:
any:
# Block when existing claim count reaches 5
- key: "{{claimCount}}"
operator: GreaterThanOrEquals
value: 5
---
# Kyverno policy enforcing cost-center labels on all Claims
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
# Policy requiring cost-center label on all infrastructure Claims
name: require-cost-center-label
spec:
# Block Claims missing the cost-center label
validationFailureAction: Enforce
rules:
# Rule to validate cost-center label presence
- name: check-cost-center
match:
resources:
# Apply to all Claim types across the platform
kinds:
- PostgreSQLInstance
- RedisCluster
- S3Bucket
validate:
message: "All infrastructure Claims must have a cost-center label"
pattern:
metadata:
labels:
# Require the cost-center label with any non-empty value
cost-center: "?*"
---
# Usage resource preventing deletion of shared infrastructure
apiVersion: apiextensions.crossplane.io/v1alpha1
kind: Usage
metadata:
# Protect the shared VPC from deletion while databases depend on it
name: payments-vpc-in-use-by-orders-db
spec:
of:
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: VPC
resourceRef:
# The shared VPC used by multiple services
name: payments-vpc-production
by:
apiVersion: rds.aws.crossplane.io/v1alpha1
kind: Cluster
resourceRef:
# The orders database that depends on the VPC
name: orders-db-cluster💡
Interview Tip
A junior engineer typically says they use cloud provider tags for cost tracking without explaining how those tags are consistently applied or how quotas are enforced. To stand out, explain the Composition-as-enforcement-point pattern: platform engineers embed patches that automatically propagate Claim metadata (namespace, name, team) into cloud resource tags, ensuring 100% tag coverage without relying on developers. Describe the multi-level quota strategy: Composition transforms constrain instance types to t-shirt sizes with known costs, Kyverno policies count existing Claims per namespace to enforce quantity limits, and Usage resources prevent deletion of shared infrastructure. Mention pre-provisioning cost estimation with Infracost and the chargeback workflow that correlates cloud billing data with Claim ownership through tags.
◈ Architecture Diagram
┌───────────────────────────────────────────────────────────────┐ │ Crossplane Cost Tracking and Quota Architecture │ ├───────────────────────────────────────────────────────────────┤ │ │ │ Tag Propagation Flow: │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Claim │───→│ Composition │───→│ AWS Resource │ │ │ │ │ │ Patches │ │ Tags │ │ │ │ namespace: │ │ │ │ │ │ │ │ payments │ │ fromField: │ │ team: payments│ │ │ │ name: │ │ namespace │ │ claim: pay-db │ │ │ │ payments-db │ │ → tag.team │ │ env: prod │ │ │ │ cost-center: │ │ │ │ cost: CC-1234 │ │ │ │ CC-1234 │ │ fromField: │ │ managed-by: │ │ │ └──────────────┘ │ cost-center │ │ crossplane │ │ │ │ → tag.cost │ └──────┬───────┘ │ │ └──────────────┘ │ │ │ ↓ │ │ ┌──────────────┐ │ │ │ AWS Cost │ │ │ │ Explorer │ │ │ │ (group by tag)│ │ │ └──────────────┘ │ │ │ │ Quota Enforcement: │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Level 1: Composition (instance type constraints) │ │ │ │ ┌───────┐ ┌────────┐ ┌──────────┐ │ │ │ │ │small │ │medium │ │large │ No XXL option │ │ │ │ │$200/mo│ │$800/mo │ │$2400/mo │ by design │ │ │ │ └───────┘ └────────┘ └──────────┘ │ │ │ └─────────────────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Level 2: Kyverno Policy (quantity limits) │ │ │ │ │ │ │ │ payments namespace: 3/5 Claims used ✓ allow │ │ │ │ orders namespace: 5/5 Claims used ✗ deny new │ │ │ └─────────────────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Level 3: Usage (dependency protection) │ │ │ │ │ │ │ │ VPC ←── Usage ──→ RDS Cluster (cannot delete VPC) │ │ │ └─────────────────────────────────────────────────────┘ │ └───────────────────────────────────────────────────────────────┘
💬 Comments
⚡
Quick Answer
Build an IDP by combining Crossplane as the infrastructure control plane with a developer portal (Backstage), GitOps engine (ArgoCD), and policy engine (Kyverno). Crossplane provides the API layer through XRDs and Claims, Backstage provides the UI and service catalog, ArgoCD syncs Claims from Git, and Kyverno enforces organizational policies, creating a golden path where developers self-serve infrastructure without tickets.
Detailed Answer
Building a complete internal developer platform (IDP) using Crossplane involves assembling a layered architecture where Crossplane serves as the infrastructure API and reconciliation engine, integrated with complementary tools that provide developer experience, GitOps automation, observability, and governance. The goal is to create a golden path where developers on teams like payments-api, user-auth-service, order-processing-service, checkout-service, and inventory-sync can provision complete application environments through a self-service interface without filing infrastructure tickets or learning cloud-specific APIs.
The API layer is where Crossplane provides the most value. Platform engineers design XRDs that represent the organization's infrastructure primitives: DatabaseInstance, CacheCluster, MessageQueue, ObjectStorage, and ApplicationEnvironment. The ApplicationEnvironment is the highest-level abstraction: a single Claim that provisions an entire application stack including a Kubernetes namespace, database, cache, message queue, DNS entry, and TLS certificate. The Composition for ApplicationEnvironment uses nested composite resources, where each sub-resource (database, cache) is itself an XR backed by its own Composition. This creates a Composition hierarchy that is maintainable and testable at each level. When a developer on the checkout-service team requests an ApplicationEnvironment, Crossplane provisions 15-20 managed resources across multiple providers, all from a single Claim with five fields.
The developer portal layer, typically Backstage, provides the UI and service catalog. Backstage templates map to Crossplane Claim types: a 'New Microservice' template generates a Git repository with application scaffolding, a Claim YAML file for the infrastructure, and an ArgoCD Application manifest for deployment. Developers fill out a form in Backstage specifying their service name, team, environment, and resource requirements. Backstage creates a pull request with the generated files, and after review and merge, ArgoCD deploys the Claim into the Crossplane control plane. The developer never writes YAML manually and does not need to understand Crossplane, Kubernetes, or cloud provider APIs.
The GitOps layer connects Backstage's output to Crossplane's input. ArgoCD watches the Git repository containing Claim manifests and syncs them to the Crossplane control plane cluster. Each team's Claims are stored in team-specific directories or repositories, with ArgoCD ApplicationSets generating Applications per team. This GitOps approach provides audit trails (every infrastructure change is a Git commit), rollback capability (revert a commit to undo a Claim), and drift detection (ArgoCD alerts when the cluster state diverges from Git). The order-processing-service team manages their Claims alongside their application manifests, creating a single source of truth for both application code and infrastructure.
The observability layer provides feedback to developers about their provisioned infrastructure. Crossplane's status conditions on Claims indicate whether resources are ready, synced, or in error. A custom Kubernetes operator or Prometheus exporter translates Claim status into metrics that power Grafana dashboards. Developers see a dashboard showing their database endpoint, connection status, storage utilization, and cost estimate, all linked from their Backstage service page. When the payments-api team's database Claim enters a degraded state, an alert fires in their PagerDuty rotation with a link to the Backstage service page showing the affected infrastructure.
The governance layer ensures that the golden path remains compliant and cost-effective. Kyverno policies enforce tagging standards, resource size limits, and required security configurations on Claims. Crossplane's composition functions add runtime logic like checking available budget before provisioning expensive resources. The platform team publishes a monthly cost report per team, generated by correlating cloud billing data with Claim ownership tags. The entire IDP operates as a product, with the platform team treating developers as customers: collecting feedback, iterating on Claim APIs based on usage patterns, and publishing a changelog when XRD schemas or Compositions are updated. This product mindset differentiates a successful IDP from a collection of tools that happens to include Crossplane.
Code Example
# High-level ApplicationEnvironment XRD for the IDP
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
# XRD for provisioning complete application environments
name: applicationenvironments.platform.acme.io
spec:
# Platform API group for all IDP resources
group: platform.acme.io
names:
# Cluster-scoped composite resource kind
kind: XApplicationEnvironment
# Plural form for API registration
plural: xapplicationenvironments
# Enable namespace-scoped Claims for developer self-service
claimNames:
# Developer-facing Claim kind
kind: ApplicationEnvironment
# Plural form for the Claim API
plural: applicationenvironments
# Connection secrets published to the developer's namespace
connectionSecretKeys:
- database-endpoint
- database-password
- cache-endpoint
- queue-url
versions:
- name: v1alpha1
served: true
referenceable: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
# Service name used for resource naming
serviceName:
type: string
description: "Name of the application service"
# Team identifier for cost allocation
team:
type: string
description: "Team owning this environment"
# Environment tier affecting resource sizing
tier:
type: string
enum: [development, staging, production]
description: "Environment tier"
# Enable or disable optional components
components:
type: object
properties:
database:
type: boolean
default: true
cache:
type: boolean
default: false
messageQueue:
type: boolean
default: false
required: [serviceName, team, tier]
---
# Backstage template that generates Claims and ArgoCD Application
# catalog-info.yaml for the Backstage service catalog
apiVersion: backstage.io/v1alpha1
kind: Template
metadata:
# Backstage template for provisioning new microservices
name: new-microservice
title: New Microservice
description: Provision a new microservice with infrastructure
spec:
# Template owner is the platform engineering team
owner: platform-engineering
type: service
parameters:
# Step 1: Basic service information
- title: Service Details
properties:
serviceName:
title: Service Name
type: string
description: "Name for the new microservice"
team:
title: Team
type: string
enum: [payments, orders, checkout, auth, inventory]
tier:
title: Environment
type: string
enum: [development, staging, production]
# Step 2: Infrastructure components
- title: Infrastructure
properties:
needsDatabase:
title: PostgreSQL Database
type: boolean
default: true
needsCache:
title: Redis Cache
type: boolean
default: false
# Steps executed by Backstage to scaffold the service
steps:
# Generate the Crossplane Claim YAML from the template
- id: generate-claim
name: Generate Infrastructure Claim
action: fetch:template
input:
url: ./skeleton
values:
serviceName: ${{ parameters.serviceName }}
team: ${{ parameters.team }}
tier: ${{ parameters.tier }}
# Create a pull request with the generated files
- id: create-pr
name: Create Pull Request
action: publish:github:pull-request
input:
repoUrl: github.com/acme-corp/infrastructure-claims
title: "Add ${{ parameters.serviceName }} environment"
branchName: "add-${{ parameters.serviceName }}"
---
# ArgoCD ApplicationSet generating apps per team namespace
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
# ApplicationSet managing infrastructure Claims per team
name: team-infrastructure-claims
namespace: argocd
spec:
generators:
# Generate one Application per team directory in the Git repo
- git:
repoURL: https://github.com/acme-corp/infrastructure-claims
revision: main
directories:
# Each team has a directory with their Claim manifests
- path: "teams/*"
template:
metadata:
# Application name derived from the team directory name
name: "infra-{{path.basename}}"
spec:
project: infrastructure
source:
# Git repository containing team Claim manifests
repoURL: https://github.com/acme-corp/infrastructure-claims
# Target revision for production Claims
targetRevision: main
# Path to the team's Claim directory
path: "{{path}}"
destination:
# Deploy Claims to the Crossplane control plane cluster
server: https://crossplane-control-plane.internal
# Each team's Claims deployed to their namespace
namespace: "{{path.basename}}"
syncPolicy:
automated:
# Auto-sync Claims when Git changes are detected
selfHeal: true
# Auto-prune Claims removed from Git
prune: true
---
# Developer submits this Claim through Backstage UI
apiVersion: platform.acme.io/v1alpha1
kind: ApplicationEnvironment
metadata:
# Environment for the checkout service
name: checkout-service-prod
namespace: checkout
labels:
# Cost center for FinOps chargeback
cost-center: CC-5678
spec:
# Service name used for resource naming conventions
serviceName: checkout-service
# Team owning this environment
team: checkout
# Production tier with full HA and backups
tier: production
# Components to provision for this service
components:
database: true
cache: true
messageQueue: true
# Connection secrets appear in the checkout namespace
writeConnectionSecretToRef:
name: checkout-service-infra-credentials💡
Interview Tip
A junior engineer typically describes an IDP as a collection of tools without explaining how they integrate or what developer experience they create. To demonstrate platform engineering expertise, walk through the complete flow: a developer fills out a Backstage form specifying their service name and components, Backstage generates a Crossplane Claim and creates a pull request, after review ArgoCD syncs the Claim to the control plane, Crossplane provisions 15+ managed resources across providers, and connection details appear as Secrets in the developer's namespace. Emphasize the product mindset: XRD APIs are versioned and documented, Compositions are the implementation that developers never see, and the platform team iterates based on developer feedback. Mention the ApplicationEnvironment pattern as the highest-level abstraction that provisions an entire application stack from a single Claim.
◈ Architecture Diagram
┌───────────────────────────────────────────────────────────────┐ │ Internal Developer Platform Architecture │ ├───────────────────────────────────────────────────────────────┤ │ │ │ Developer Experience Layer │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Backstage Portal │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │ │ │ │ Service │ │ Template │ │ Service Catalog │ │ │ │ │ │ Form │ │ Engine │ │ (all services) │ │ │ │ │ │ │──→│ Generate │──→│ Dashboards │ │ │ │ │ │ name,team │ │ Claim │ │ Status, Costs │ │ │ │ │ │ tier,comp │ │ YAML │ │ │ │ │ │ │ └──────────┘ └────┬─────┘ └──────────────────┘ │ │ │ └──────────────────────┼──────────────────────────────┘ │ │ │ Pull Request │ │ ↓ │ │ GitOps Layer │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Git Repository (infrastructure-claims) │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │ │ │ │ teams/ │ │ teams/ │ │ teams/ │ │ │ │ │ │ payments/│ │ orders/ │ │ checkout/ │ │ │ │ │ │ claim.yml│ │ claim.yml│ │ claim.yaml │ │ │ │ │ └──────────┘ └──────────┘ └──────────────────┘ │ │ │ └──────────────────────┬──────────────────────────────┘ │ │ │ ArgoCD Sync │ │ ↓ │ │ Infrastructure Control Plane │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Crossplane Control Plane │ │ │ │ │ │ │ │ Claim ──→ XR ──→ Composition ──→ Managed Resources │ │ │ │ │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │ │ │ │ Database │ │ Cache │ │ Queue │ │ │ │ │ │ (RDS) │ │ (Elasti- │ │ (SQS) │ │ │ │ │ │ │ │ Cache) │ │ │ │ │ │ │ └──────────┘ └──────────┘ └──────────────────┘ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ Governance Layer │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ ┌──────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ │ │ Kyverno │ │ Cost Tracking│ │ Audit Logs │ │ │ │ │ │ Policies │ │ (Tags → │ │ (Git + K8s + │ │ │ │ │ │ (guardrls)│ │ Billing) │ │ CloudTrail) │ │ │ │ │ └──────────┘ └──────────────┘ └──────────────┘ │ │ │ └─────────────────────────────────────────────────────┘ │ └───────────────────────────────────────────────────────────────┘
💬 Comments