What is the OpenShift cluster upgrade process. What is the order of operations and what gets upgraded first?

Think of an OpenShift upgrade like renovating a multi-story building floor by floor while tenants keep living in it. You don't shut the whole building down — you start with the management office (control plane), then renovate one floor at a time (worker nodes), making sure tenants can always use at least some floors.

The upgrade starts with the Cluster Version Operator (CVO), which is the orchestrator. When you initiate an upgrade (via console or oc adm upgrade), the CVO first validates the upgrade path — not all version jumps are supported. For example, 4.14 can go to 4.15 but not directly to 4.17. The CVO checks the upgrade graph from Red Hat's Cincinnati service to confirm the path is valid and no known blocking issues exist.

Once validated, the CVO begins updating Cluster Operators one by one in dependency order. These are the built-in platform services: the authentication operator, ingress operator, monitoring operator, machine-config operator, and about 30 others. Each operator updates its own managed components (containers, configs, RBAC). The CVO waits for each operator to report 'Available=True, Degraded=False' before proceeding to the next. If any operator gets stuck, the entire upgrade pauses.

Next, the control plane nodes upgrade. The Machine Config Operator (MCO) renders a new MachineConfig for the master MachineConfigPool, which includes the new kubelet version, CRI-O version, and OS packages. Master nodes are updated one at a time (the MCO respects maxUnavailable, defaulting to 1 for masters). Each master is cordoned, drained, rebooted with the new config, and uncordoned. Since etcd requires quorum (2 of 3 nodes), only one master can be down at a time.

Finally, worker nodes upgrade through the worker MachineConfigPool. The MCO renders the new MachineConfig, then updates workers according to the MCP's maxUnavailable setting (default is 1). Each worker is cordoned (no new Pods scheduled), drained (existing Pods evicted respecting PDBs), rebooted with new OS and configs, and uncordoned. This is where PDB issues most commonly stall upgrades — if a PDB prevents all Pods from being evicted, the drain hangs indefinitely.

The most common upgrade failure: a worker node stuck in 'draining' because a PodDisruptionBudget prevents eviction. The MCO logs and oc get nodes will show the node as SchedulingDisabled. You need to identify the blocking PDB with oc get pdb -A and either adjust it or manually handle the stuck Pods.