Apache Airflow: Zero to Hero

The workflow orchestrator that runs the data pipelines behind analytics, ETL, and ML — schedule, sequence, retry, and monitor tasks defined as code. You'll run Airflow locally, write DAGs, wire task dependencies, use operators and sensors, manage connections, and make pipelines idempotent and production-ready. Hands-on throughout.

The journey:

1. What Airflow is and the DAG model
2. Run Airflow locally
3. Your first DAG
4. The TaskFlow API and dependencies
5. Operators, hooks, and connections
6. Scheduling, backfills, and idempotency
7. Sensors and cross-task data
8. Executors and scaling
9. Production practices

---

Prerequisites

Airflow orchestrates other systems — it triggers and monitors work, it isn't a data-processing engine itself. Confirm Docker Compose: docker compose version.

---

Part 1 — What Airflow is and the DAG model

Airflow is a platform to author, schedule, and monitor workflows as code. A workflow is a DAG (Directed Acyclic Graph) — tasks with dependencies, no cycles:

extract ──▶ transform ──▶ load
                └────────▶ notify

Each task is a unit of work; edges define order. Airflow's scheduler runs tasks when their dependencies are met and their schedule fires, retries failures, and shows everything in a web UI (task status, logs, run history). The defining idea: pipelines are Python code, so they're versioned, reviewed, tested, and dynamic — not clicked together in a GUI.

---

Part 2 — Run Airflow locally

The official Docker Compose brings up the scheduler, webserver, and metadata DB:

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml'
mkdir -p dags logs plugins
echo -e "AIRFLOW_UID=$(id -u)" > .env
docker compose up airflow-init
docker compose up -d

Open http://localhost:8080 (login airflow/airflow). Drop .py files in dags/ and Airflow discovers them. The core services: the scheduler (decides what runs when), the webserver (UI), a metadata database (state), and workers/executor (run the tasks).

---

Part 3 — Your first DAG

A DAG file defines the schedule and the tasks. Save dags/etl.py:

from airflow import DAG
from airflow.operators.bash import BashOperator
import pendulum

with DAG(
    dag_id="daily_etl",
    schedule="0 2 * * *",                 # 02:00 daily (cron)
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,                        # don't backfill history on first deploy
    default_args={"retries": 2},
) as dag:

    extract   = BashOperator(task_id="extract",   bash_command="echo extracting")
    transform = BashOperator(task_id="transform", bash_command="echo transforming")
    load      = BashOperator(task_id="load",      bash_command="echo loading")

    extract >> transform >> load          # dependency: extract, then transform, then load

The >> operator sets order. schedule accepts cron or presets (@daily); start_date + catchup control history; default_args (retries, retry_delay) apply to every task. Airflow now runs this DAG at 02:00 daily, retrying failed tasks twice.

---

Part 4 — The TaskFlow API and dependencies

Modern Airflow uses the TaskFlow API — decorators that make Python functions into tasks and pass data between them naturally:

from airflow.decorators import dag, task
import pendulum

@dag(schedule="@daily", start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), catchup=False)
def etl():
    @task
    def extract() -> dict:
        return {"rows": 1000}

    @task
    def transform(data: dict) -> int:
        return data["rows"] * 2

    @task
    def load(n: int):
        print(f"loading {n} rows")

    load(transform(extract()))            # calling tasks wires the DAG

etl()

Calling one task's result into another defines the dependency and passes the value — cleaner than manual >> and XCom. Dependencies can also be explicit (a >> [b, c] for fan-out, [b, c] >> d for fan-in). TaskFlow is the recommended style for new DAGs.

---

Part 5 — Operators, hooks, and connections

Airflow's power is its operators — pre-built task types for systems you integrate with, so you don't reinvent them:

Operators use hooks (reusable clients for a system) that read credentials from Connections — secrets you store once in Airflow (UI or a secrets backend), never in the DAG:

from airflow.providers.postgres.operators.postgres import PostgresOperator
run_sql = PostgresOperator(task_id="agg", postgres_conn_id="warehouse", sql="INSERT INTO summary ...")

The postgres_conn_id points at a stored Connection — so DAG code has no credentials, and the same DAG runs against dev/prod by swapping the connection.

---

Part 6 — Scheduling, backfills, and idempotency

Airflow runs a DAG once per data interval (each schedule window). Two concepts trip everyone up:

• catchup — if True, deploying a DAG with an old start_date runs every missed interval (a backfill). Set catchup=False unless you truly want history reprocessed.
• Idempotency — because tasks retry and intervals can re-run, a task must produce the same result if run twice. Write to a partition keyed by the run's date, use INSERT ... ON CONFLICT/MERGE, and never append blindly.

@task
def load(ds=None):                        # ds = the run's logical date
    # write to a date-partitioned target so a re-run overwrites, not duplicates
    write_partition(date=ds)

The templating variable {{ ds }} (logical date) makes tasks reference their interval, not "now" — essential for correct backfills and idempotent reruns.

---

Part 7 — Sensors and cross-task data

Sensors wait for a condition before downstream tasks run — a file to land, a partition to appear, an external job to finish:

from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
wait = S3KeySensor(task_id="wait_for_file", bucket_key="s3://data/{{ ds }}/input.csv", mode="reschedule")

Use mode="reschedule" (not the default poke) for long waits so the sensor frees its worker slot between checks instead of blocking it. For passing small values between tasks, XCom (what TaskFlow uses under the hood) stores them in the metadata DB — keep XCom payloads tiny (ids, paths), never large datasets (write those to storage and pass the path).

---

Part 8 — Executors and scaling

The executor decides where tasks run:

• LocalExecutor — tasks as subprocesses on one machine. Fine for small setups.
• CeleryExecutor — a pool of worker machines pulling tasks from a queue. Classic horizontal scaling.
• KubernetesExecutor — each task runs as its own pod, created on demand and destroyed after. Great isolation, elastic, no idle workers.

The scheduler stays the brain; the executor is the muscle. For Kubernetes shops, KubernetesExecutor (or the KubernetesPodOperator per task) gives per-task isolation, independent dependencies/images, and autoscaling — each task gets exactly the resources it declares.

---

Part 9 — Production practices

• Idempotent tasks keyed on the logical date ({{ ds }}) — reruns and backfills must not duplicate.
• catchup=False by default; enable backfill only intentionally.
• Credentials in Connections/secrets backend, never in DAG code.
• Keep DAG files light — they're parsed constantly; do heavy imports/work inside tasks, not at module top level.
• Sensors in reschedule mode for long waits; small XCom payloads only.
• Isolate tasks with KubernetesPodOperator/KubernetesExecutor for dependency and resource isolation.
• Set retries + alerting (on-failure callbacks / SLA misses) so failures page someone.
• Version DAGs in Git, test them, and deploy via CI.

---

Common mistakes to avoid

• Non-idempotent tasks. Retries/backfills then duplicate or corrupt data — key writes on the run date.
• Leaving catchup=True unintentionally. Deploying floods the scheduler with historical runs.
• Heavy work at the top of a DAG file. It runs on every parse and slows the scheduler — put it inside tasks.
• Credentials hardcoded in DAGs. Use Connections/secrets backends.
• Large data in XCom. It bloats the metadata DB — pass paths, store data externally.
• poke-mode sensors for long waits. They hog worker slots; use reschedule.
• Using Airflow to process data. It orchestrates; do the compute in Spark/SQL/containers it triggers.

---

Practice: try it yourself

1. Run it. Bring up Airflow with Compose and confirm the example DAGs appear in the UI.
2. First DAG. Write a 3-task extract >> transform >> load DAG and trigger it; read each task's logs.
3. TaskFlow. Rewrite it with @dag/@task, passing a value from extract → transform → load.
4. Idempotency. Make load write to a {{ ds }}-partitioned target and re-run the same date — confirm no duplicates.
5. Sensor + connection. Add an S3KeySensor in reschedule mode and a database operator using a stored Connection.

Self-check:

• What is a DAG, and how do task dependencies get defined (two ways)?
• Why must tasks be idempotent, and how does {{ ds }} help?
• What do operators, hooks, and connections each provide?
• Local vs Celery vs Kubernetes executor — what's the trade-off?

---

Where to go next

• Run tasks as pods — the Kubernetes tutorial; the KubernetesExecutor/PodOperator is the modern default.
• Orchestrate ML — the Kubeflow tutorial for the training pipelines Airflow can feed.
• Practice with data-engineering interview questions in the question bank.

You now have the loop: DAG as code → schedule → dependencies → operators/connections → idempotent runs → sensors → scale with executors. That's Airflow orchestrating production pipelines.