You need to implement a custom Kubernetes controller (operator) that automatically provisions cloud databases (RDS instances) when a custom resource is created. Describe the controller architecture, reconciliation loop design, and how you handle eventual consistency and error scenarios.
Quick Answer
Build a controller using controller-runtime (kubebuilder/operator-sdk) with a reconciliation loop that watches custom resources, creates RDS instances via AWS SDK, updates status subresource with connection details, and uses finalizers for cleanup. Handle eventual consistency with requeue-after and status conditions.
Detailed Answer
Controller Architecture
The operator pattern extends Kubernetes by defining Custom Resource Definitions (CRDs) that represent desired state (e.g., 'I want a PostgreSQL 15 database with 100GB storage') and a controller that reconciles actual state to match. Use kubebuilder to scaffold the project, which generates the CRD, controller skeleton, RBAC manifests, and webhook configurations.
Reconciliation Loop Design
The reconciler follows a level-triggered (not edge-triggered) pattern. Every reconciliation must be idempotent - it's called when the resource changes, when dependencies change, or on periodic resyncs. The loop should: 1. Fetch the CR (handle NotFound - resource was deleted) 2. Check finalizer (add if missing, handle deletion if being deleted) 3. Read current state from AWS (DescribeDBInstances) 4. Compare desired vs actual state 5. Take action (CreateDBInstance, ModifyDBInstance, or no-op) 6. Update status subresource with current state, conditions, and connection details 7. Return result (requeue after duration if provisioning is in-progress)
Eventual Consistency Handling
RDS operations are asynchronous - CreateDBInstance returns immediately but the instance takes 5-10 minutes to become available. Use ctrl.Result{RequeueAfter: 30 * time.Second} to poll until the instance is ready. Track the provisioning lifecycle using status conditions (Provisioning, Available, Failed, Deleting).
Error Handling
- Transient AWS errors (throttling, service unavailable): Requeue with exponential backoff - Permanent errors (invalid parameters): Set status condition to Failed with error message, don't requeue - Concurrent modifications: Use resource version for optimistic concurrency on status updates - Finalizer for cleanup: When CR is deleted, the finalizer ensures the RDS instance is deleted before the CR is removed from etcd