What changed in Airflow 3 that an operator of a large Airflow 2 fleet actually cares about, and how do you plan the upgrade?
Quick Answer
The operator-relevant changes: DAG versioning (runs execute against the DAG version they started with — ending mid-run mutation bugs), a new React UI and stable REST API, the Task SDK / task execution interface decoupling task code from scheduler internals (and enabling remote execution), scheduler-managed backfills, and removed legacy paths (SubDAGs gone, direct metadata-DB access from tasks deprecated). Plan it like a platform migration: inventory deprecated usage with upgrade tooling, fix on 2.x latest first, then cut over DAG-by-DAG cohorts.
Detailed Answer
Why each matters operationally: DAG versioning fixes a whole incident class — under Airflow 2, editing a DAG mid-backfill made running DagRuns pick up mixed old/new task definitions; Airflow 3 pins a run to its version snapshot. The Task SDK boundary means task code stops importing scheduler internals (those imports were why minor upgrades broke DAGs), and execution can happen in isolated/remote environments — the security model improves too, since tasks no longer need direct metadata-DB credentials (a long-standing 'any task can read every connection' problem). Scheduler-managed backfills replace the CLI-spawned kind that operated outside normal scheduling (and its concurrency limits). Removals are the migration work: SubDAGs (convert to TaskGroups), various context/import shims, implicit DB access from workers. Upgrade plan: (1) get to latest 2.x and burn down every deprecation warning — the ruff-based upgrade rules/AIR checks automate much of the inventory; (2) audit tasks for metadata-DB and internal-API imports — these are the hard 20%; (3) stand up an Airflow 3 environment in parallel, migrate DAG cohorts by criticality with dual-running for a cycle (idempotent tasks make this safe), and keep the 2.x environment as rollback until the last cohort has run a full schedule cleanly.
Code Example
# inventory the migration surface ruff check dags/ --select AIR # Airflow-specific upgrade rules grep -rn 'airflow.settings\|airflow.models.base\|Session()' dags/ # DB-touching tasks grep -rn 'SubDagOperator' dags/ # must become TaskGroups # cohort cutover: low-risk DAGs first, dual-run 1 cycle, compare task outcomes
Interview Tip
DAG versioning ('a run executes the code it started with') and the task/metadata-DB decoupling are the two changes to explain — they map to real Airflow 2 incidents interviewers have lived through.