How would you troubleshoot a production issue involving Apache Airflow?
Quick Answer
Start with user impact, recent changes, health/status APIs, logs, metrics, dependencies, and configuration drift. Then isolate whether the issue is caused by Apache Airflow, an upstream producer, a downstream consumer, infrastructure, credentials, or capacity.
Detailed Answer
A good troubleshooting flow for Apache Airflow starts by defining the symptom in measurable terms: error rate, latency, failed jobs, missing data, failed deployments, or customer impact. Next, compare current state against the last known good state. Review recent releases, config changes, credential rotations, certificate updates, node pressure, network ACL changes, and dependency outages.
For Apache Airflow, focus the technical diagnosis on DAG design, operators and sensors, executor types (Celery/Kubernetes/Local), scheduler and worker scaling, task retries and backfills, XComs, and connection/variable management. Gather logs and metrics from the component itself and from anything it talks to. If the tool has agents, workers, controllers, delegates, brokers, or sidecars, inspect each layer separately. Keep remediation reversible: pause risky automation, roll back config, scale safely, restore from backup, or route around a failed dependency.
Code Example
# Generic production triage checklist for Apache Airflow # status: check component health and version # logs: filter by incident window and correlation id # metrics: compare p50/p95/p99, error rate, queue depth, saturation # config: diff desired vs live configuration # dependency: test DNS, network, auth, certs, storage, and API limits
Interview Tip
Narrate your diagnosis order and explain why each signal proves or eliminates a hypothesis.