Compare Airflow executors — Local, Celery, Kubernetes (and CeleryKubernetes) — and explain how you'd choose for a given workload profile.
Quick Answer
LocalExecutor runs tasks as scheduler-host subprocesses (simple, single-node ceiling); CeleryExecutor runs a standing worker fleet via a broker (low task latency, great for many small tasks, but static capacity and dependency coupling); KubernetesExecutor launches a fresh pod per task (perfect isolation, per-task resources, scale-to-zero — at pod-startup latency cost). Choose by task volume/duration mix: high-frequency small tasks favor Celery, heterogeneous/heavy/bursty favor Kubernetes, and CeleryKubernetesExecutor routes per-task to get both.
Detailed Answer
The decision inputs: (1) task latency sensitivity — Celery workers are warm, so a task starts in milliseconds; K8s pays pod scheduling + image pull (seconds to minutes), painful for DAGs of many 5-second tasks; (2) isolation and dependencies — K8s gives each task its own image (data-science tasks with conflicting libraries stop fighting), per-task CPU/memory/GPU requests, and no noisy-neighbor between tasks; Celery workers share one environment, making dependency upgrades fleet-wide events; (3) capacity shape — Celery fleets are provisioned for peak (idle cost overnight), K8s scales with the cluster autoscaler to near-zero; (4) operational surface — Celery adds a broker (Redis/RabbitMQ) with its own failure modes (visibility_timeout zombie interactions are a classic), K8s adds pod-churn observability needs and image management. CeleryKubernetesExecutor (or multiple executors in Airflow 2.10+/3.x) routes tasks by queue: the ETL heartbeat tasks stay on warm Celery workers, the memory-hungry ML tasks get dedicated pods. Interview-grade nuance: executor choice doesn't change DAG-parsing or scheduler capacity — a slow scheduler bottlenecks any executor, and 'the executor is slow' complaints are often DAG-parsing or pool starvation in disguise.
Code Example
# airflow.cfg
[core] executor = CeleryKubernetesExecutor
[celery_kubernetes_executor] kubernetes_queue = kubernetes
# DAG: route per task
light = BashOperator(task_id='rollup', ...) # default: celery
heavy = PythonOperator(task_id='train', queue='kubernetes',
executor_config={'pod_override': k8s.V1Pod(spec=...gpu...)})Interview Tip
The routing answer (CeleryKubernetes / multiple executors: warm workers for small tasks, pods for heavy ones) shows you've operated past the either/or framing — and note that scheduler bottlenecks masquerade as executor problems.