What makes an Airflow task idempotent, and why do the scheduler's retry and catchup semantics make idempotency non-negotiable?
Quick Answer
An idempotent task produces the same end state no matter how many times it runs for a given logical date — typically by overwriting partitions keyed on the interval (INSERT OVERWRITE / delete-then-write) instead of appending. Airflow WILL run your task multiple times: retries after partial failure, zombie-task reschedules, backfills, catchup, and manual re-runs — non-idempotent tasks turn each of those normal mechanisms into a data-corruption event.
Detailed Answer
The contract: task(logical_date) should be a pure function to the warehouse — same input interval, same resulting state. Violations and their blast: an INSERT-appending task that gets retried after writing half its rows duplicates data; a task computing 'last 7 days from now()' instead of from the interval produces different results on backfill than it did live (breaking reproducibility and making incident recovery rewrite history); a task mutating shared state (rotating a credential, sending emails) re-executes on zombie reschedule and double-fires. Patterns that achieve it: partition-overwrite writes keyed on data_interval_start (the modern parameter, not the deprecated execution_date), staging-table-then-atomic-swap for warehouses without overwrite, deterministic keys/dedup for event emission (idempotency keys downstream), and separating detection from action for side-effecting tasks so the action step can check 'already done?'. Design smell to name: any task reading wall-clock now() — time comes from the interval parameters, always. This is also what makes zombie-task tuning (visibility_timeout vs task duration) survivable: if double-execution is safe, scheduler edge cases are annoyances, not incidents.
Code Example
@task
def load(data_interval_start=None, data_interval_end=None):
df = extract(start=data_interval_start, end=data_interval_end) # never now()
# idempotent write: same interval -> same partition, overwritten
df.write.mode('overwrite').partitionBy('ds') \
.save(f's3://lake/events/ds={data_interval_start:%Y-%m-%d}')Interview Tip
'Same logical date, same end state — because Airflow will re-run it, that's a feature' is the framing. Bonus points for banning now() in favor of interval params and mentioning the zombie-reschedule interaction.