Kubernetes Operators: Zero to Hero

Extend Kubernetes with your own resources and automation — encode the operational knowledge of running an app (install, upgrade, backup, failover) into a controller that manages it for you. You'll understand CRDs and the reconcile loop, scaffold an operator, implement reconciliation, and learn when to build one (and when not to). Concepts-and-hands-on.

The journey:

1. What an operator is and why
2. Custom Resource Definitions (CRDs)
3. The controller and the reconcile loop
4. Level-triggered reconciliation
5. Scaffold an operator (Kubebuilder)
6. Implement Reconcile
7. Status, events, and finalizers
8. When to build one (and when not)
9. Production practices

---

Prerequisites

Do the Kubernetes tutorial first — operators build directly on its concepts. Confirm your cluster: kubectl get nodes.

---

Part 1 — What an operator is and why

Kubernetes already runs controllers that drive actual state toward desired state: you declare replicas: 3, the Deployment controller makes it so. An operator applies that same pattern to your application's operational tasks.

Formally: operator = a Custom Resource Definition (CRD) + a custom controller. The CRD lets users declare a high-level object (kind: PostgresCluster); the controller watches those objects and does the real work — provisioning StatefulSets, configuring replication, taking backups, handling failover. It captures the operational knowledge a human SRE would apply, and runs it continuously. That's why databases, message queues, and platforms ship operators: they automate day-2 operations, not just day-1 install.

---

Part 2 — Custom Resource Definitions (CRDs)

A CRD teaches the Kubernetes API a new resource type. Once installed, kubectl treats it like a built-in:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: { name: postgresclusters.acme.io }
spec:
  group: acme.io
  names: { kind: PostgresCluster, plural: postgresclusters, shortNames: [pgc] }
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas: { type: integer, minimum: 1 }
                version:  { type: string }

Now users create instances:

apiVersion: acme.io/v1alpha1
kind: PostgresCluster
metadata: { name: orders-db }
spec: { replicas: 3, version: "16" }

The CRD gives you API validation (the OpenAPI schema), kubectl get postgresclusters, RBAC, and storage in etcd — all for free. But a CRD alone does nothing; it's just a new noun until a controller acts on it.

---

Part 3 — The controller and the reconcile loop

The controller is the verb. It watches your custom resources and, on any change, runs a reconcile function whose job is: make the world match the spec.

        watch PostgresCluster ──▶ event ──▶ Reconcile(orders-db)
                                                │
                    read desired (spec) ────────┤
                    read actual (StatefulSet…) ─┤
                    compute + apply the diff ───┘
                    (create/update/delete children)

The controller doesn't script "do step 1, then 2." Each reconcile observes desired vs actual state and drives toward desired — creating a StatefulSet if missing, scaling it if replicas changed, updating config, etc. It runs on every relevant change and periodically, converging over time.

---

Part 4 — Level-triggered reconciliation

The most important design principle: operators are level-triggered, not edge-triggered. Reconcile doesn't react to "the event that happened" — it reads the current desired and actual state and makes them match, every time, from scratch.

Why it matters: events can be missed, duplicated, or arrive out of order; the controller might restart mid-operation. A level-triggered reconcile is idempotent — running it once or ten times converges to the same correct state — so it's robust to all of that. The rule of thumb for a good Reconcile: "look at the world as it is now, and take one step toward desired state; you'll be called again." Never assume you saw every change or that state is what you last left it.

---

Part 5 — Scaffold an operator (Kubebuilder)

You rarely write controllers from raw client-go. Kubebuilder (built on controller-runtime, the same library the Operator SDK uses) scaffolds the CRD, controller, and manager:

kubebuilder init --domain acme.io --repo github.com/acme/pg-operator
kubebuilder create api --group acme --version v1alpha1 --kind PostgresCluster
# fills in api/v1alpha1/postgrescluster_types.go and controllers/postgrescluster_controller.go
make manifests        # generates the CRD YAML from your Go structs
make install run      # install CRDs + run the controller locally against your cluster

You edit the types (the Go struct = the CRD schema, with Spec and Status) and the Reconcile function. make manifests keeps the CRD YAML in sync with your Go types, and RBAC is generated from //+kubebuilder markers. This is the standard toolchain; the Operator SDK adds OLM packaging on top.

---

Part 6 — Implement Reconcile

The heart of the operator. A simplified reconcile that ensures a StatefulSet exists and matches the desired replicas:

func (r *PostgresClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // 1. Fetch the custom resource (desired state)
    var pgc acmev1alpha1.PostgresCluster
    if err := r.Get(ctx, req.NamespacedName, &pgc); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)   // deleted? nothing to do
    }

    // 2. Fetch/derive actual state (the StatefulSet we own)
    var sts appsv1.StatefulSet
    err := r.Get(ctx, types.NamespacedName{Name: pgc.Name, Namespace: pgc.Namespace}, &sts)

    // 3. Reconcile the diff — create or update
    if apierrors.IsNotFound(err) {
        sts = buildStatefulSet(&pgc)              // desired from spec
        ctrl.SetControllerReference(&pgc, &sts, r.Scheme)  // own it (GC + re-reconcile)
        return ctrl.Result{}, r.Create(ctx, &sts)
    }
    if *sts.Spec.Replicas != pgc.Spec.Replicas {
        sts.Spec.Replicas = &pgc.Spec.Replicas
        return ctrl.Result{}, r.Update(ctx, &sts)
    }
    return ctrl.Result{}, nil                     // already converged
}

The pattern: Get desired → observe actual → apply the difference → return. SetControllerReference makes the StatefulSet owned by the custom resource, so Kubernetes garbage-collects it when the CR is deleted and re-triggers reconcile when the child changes. Return an error (or Result{RequeueAfter}) to be called again.

---

Part 7 — Status, events, and finalizers

Production operators do more than create children:

• Status — write observed state back to the CR's .status (ready replicas, phase, conditions) so users and other controllers can see what's happening. Update status separately from spec.
• Events — record kubectl describe-visible events for important actions/failures.
• Finalizers — a finalizer on the CR blocks deletion until your controller does cleanup (deprovision a cloud DB, take a final backup). You add the finalizer, and on delete you run cleanup then remove it, letting the delete complete.

// Finalizer pattern: on deletion, run external cleanup before letting it go
if !pgc.DeletionTimestamp.IsZero() {
    // ... take final backup, delete external resources ...
    controllerutil.RemoveFinalizer(&pgc, myFinalizer)
    return ctrl.Result{}, r.Update(ctx, &pgc)
}

Finalizers are how operators handle external state (cloud resources, backups) that Kubernetes GC can't clean up on its own.

---

Part 8 — When to build one (and when not)

Operators are powerful but not free — a controller is software you must build, test, secure, and run. Build one when:

• You manage a stateful or complex app with real day-2 operations (databases, brokers) — install, upgrade, backup, failover encoded as automation.
• You're building a platform and want teams to consume a simple CR instead of dozens of low-level objects.

Don't build one when a Helm chart or Kustomize already expresses your app — those handle templating and install without a running controller. The test: is there ongoing operational logic that must react to state over time? If it's just "render these manifests," you want Helm (see the Helm tutorial), not an operator. Also check whether an operator already exists (OperatorHub) before writing your own.

---

Part 9 — Production practices

• Reconcile idempotently and level-triggered — read current state, take one step, expect to be called again.
• Own child resources (SetControllerReference) for GC and re-reconcile on child changes.
• Report .status + events so the CR is observable; use conditions.
• Use finalizers for external cleanup; make cleanup idempotent too.
• Scope RBAC tightly — the controller's ServiceAccount should have only the permissions it needs.
• Handle errors by requeueing (return err / RequeueAfter), not by panicking or scripting long sequences.
• Version your CRD (v1alpha1 → v1) with conversion; never break users' stored objects.
• Test with envtest; watch reconcile latency and error rate like any controller.

---

Common mistakes to avoid

• Edge-triggered thinking. Reacting to specific events (and assuming you saw them all) breaks on restarts/missed events — reconcile from current state.
• Non-idempotent Reconcile. It runs repeatedly; creating duplicates or side effects each time is a bug.
• Not owning child resources. Without owner references, children leak and changes don't re-trigger reconcile.
• Doing external cleanup without a finalizer. Deleting the CR then leaves orphaned cloud resources.
• Over-broad RBAC. A cluster-admin controller is a big attack surface — least privilege.
• Building an operator where Helm suffices. No ongoing logic ⇒ no operator needed.
• Breaking CRD compatibility. Changing a stored schema without conversion corrupts existing CRs.

---

Practice: try it yourself

1. Define a CRD. Write a CRD for a simple kind with a spec.replicas and spec.version, install it, and kubectl apply an instance.
2. Scaffold. kubebuilder init + create api, then make manifests and inspect the generated CRD and controller.
3. Reconcile. Implement a Reconcile that creates a Deployment from the CR's spec and sets an owner reference; change replicas and watch it reconcile.
4. Status. Write ready-replica count back to .status and confirm it shows in kubectl get -o yaml.
5. Finalizer. Add a finalizer, delete the CR, and confirm your cleanup runs before deletion completes.

Self-check:

• What two things make up an operator, and what does each provide?
• Why must reconciliation be level-triggered and idempotent?
• What does an owner reference give you?
• When should you use a Helm chart instead of an operator?

---

Where to go next

• The foundation — the Kubernetes tutorial and Go tutorial (operators are Go).
• Operators you already use — KEDA, Kyverno, and Chaos Mesh are all operators; revisit them with this lens.
• Practice with Kubernetes interview questions in the question bank.

You now have the model: CRD → controller → reconcile loop → level-triggered idempotence → status/finalizers → build only when there's real day-2 logic. That's extending Kubernetes the right way.