DAG design, task dependencies, sensors, and production Airflow deployment.
Every production data team needs orchestration, and Airflow is the industry standard. Whether you run MWAA, Astronomer, or self-hosted on Kubernetes, the same DAG / executor / sensor / backfill / idempotency decisions decide whether your pipelines wake the on-call. This path teaches the decisions, not just the syntax.
Why pipelines break, your first working DAG, and the TaskFlow-era patterns the rest of the path builds on.
A realistic incident — an upstream rename, a missed sensor, a stuck retry — that motivates every orchestration decision: idempotency, retries, alerting, lineage. The forensic timeline that frames the whole path.
Local Docker setup, the JobManager / Scheduler / Worker model, your first DAG file end-to-end, operator vs task, and the Airflow UI tour every team uses to triage incidents.
TaskFlow API (the @task decorator), XComs and data passing, TaskGroups for readable graphs, dynamic task mapping, and the patterns that separate junior DAGs from production-ready ones.
Time and idempotency, external-system integration, debugging, resilience + CI/CD, performance, and a production capstone — everything between a working DAG and a DAG you trust on-call.
execution_date vs logical_date, the catchup gotcha, backfill design that doesn't double-count, idempotent UPSERTs + watermarks, and the time-handling decisions that decide whether reruns are safe.
Sensors (file / S3 / external task), provider packages, hook design, connection management, secrets backends (Vault, AWS Secrets Manager), and the patterns for talking to Snowflake / BigQuery / Postgres / Kafka from a DAG.
Reading Airflow logs efficiently, on-failure callbacks, structured logging, OpenTelemetry traces across tasks, replaying failed DAG runs, and the runbook for triaging a stuck or zombie task.
DAG unit + integration tests (pytest fixtures, dag_test), CI-gated linting + schema checks, pre-merge DAG-parse validation, staged deployment with GitHub Actions, and the tests that catch breakage before main.
Scheduler heap profiling, parallelism + concurrency tuning, the cost of too-many-DAGs, task-level resource limits, queue + pool design, and the perf checklist for a 1000+ DAG deployment.
Ship a production-grade orchestration build: multi-source ingestion DAG with sensors + TaskFlow, idempotent retries, GE-style data quality gates, CI tests, alerting, and a runbook you'd hand to on-call.
The platform-engineering layer — Kubernetes deployment, custom operator design, multi-tenant ops, and the advanced orchestration patterns that mature teams use to scale Airflow across an org.
KubernetesExecutor architecture, the official Helm chart vs Astronomer / MWAA tradeoffs, pod resource limits + spec design, KubernetesPodOperator, autoscaling, secrets via K8s, and the cluster runbook.
BaseOperator + BaseHook design, packaging a provider, the deferrable-operator (async/triggers) pattern that replaces sensors, plugin distribution across teams, and operator testing strategies.
StatsD + Prometheus metrics, scheduler SLOs, DAG-level alerting, RBAC + connection isolation for multi-tenant Airflow, audit logs, and the platform-team / domain-team operating model.
Cross-DAG dependencies (TriggerDagRunOperator, datasets, Airflow 3 assets), dataset-driven scheduling, branching + dynamic DAGs at scale, data-aware orchestration patterns, and the migration paths to Dagster / Prefect when they make sense.
Without production-grade Airflow, you risk:
Apache Airflow is an open-source workflow orchestration platform for scheduling, monitoring, and managing data pipelines. Written in Python, Airflow uses DAGs (Directed Acyclic Graphs) to define task dependencies and execution order. Used by Airbnb (where it was created), Uber, and thousands of companies to orchestrate their data infrastructure.
Every production data team needs orchestration, and Airflow is the industry standard. At Airbnb, Airflow manages tens of thousands of DAGs that coordinate data ingestion, transformation, and ML training. Production Airflow requires understanding executor types, connection management, and failure handling patterns that keep pipelines running reliably.
Airflow is the most widely adopted orchestrator with the largest ecosystem. Prefect offers a more modern Python API and better local development. Airflow dominates enterprise adoption; Prefect is growing in modern teams.
Airflow focuses on scheduling and task orchestration. Dagster emphasizes software-defined assets and data-aware orchestration. Airflow has broader adoption; Dagster offers better data lineage and testing.
Airflow orchestrates entire data platforms. dbt Cloud manages dbt-specific scheduling. Most teams use Airflow to orchestrate dbt alongside other tools, or use dbt Cloud for dbt and Airflow for everything else.
Airflow is the most-requested orchestration skill in DE job listings. Senior + Staff roles at data-mature orgs (Airbnb, Uber, Stripe, Pinterest, Reddit) hire specifically for engineers who can defend executor choice, backfill strategy, K8s deployment patterns, and idempotency design — the exact decisions this path makes you defensible on.
Airflow schedules and monitors data pipelines. Data engineers use it to orchestrate ETL jobs, dbt runs, Spark processing, and any workflow that requires task dependencies and scheduling.
Airflow remains the dominant orchestration tool. Airflow 2.x brought major improvements, and the ecosystem continues to grow. Alternatives like Prefect and Dagster complement rather than replace it.
Basic DAGs take 1-2 weeks. Production Airflow with custom operators, dynamic DAGs, and deployment patterns takes 6-8 weeks of hands-on practice.
Airflow is the most requested orchestration skill in data engineering job descriptions. Even if you use managed services like MWAA or Astronomer, Airflow concepts are essential.
Airflow has the largest ecosystem and enterprise adoption. Prefect offers a more Pythonic API. Dagster provides better data-aware orchestration. Most job listings still require Airflow.
A DAG (Directed Acyclic Graph) defines the workflow — tasks and their dependencies. Each DAG is a Python file that specifies what runs, in what order, and on what schedule.
CeleryExecutor runs tasks on a pre-provisioned pool of long-lived workers using a Celery broker (Redis or RabbitMQ) — good for steady-state DAG volume with low per-task isolation overhead. KubernetesExecutor spins up a fresh pod per task using the K8s scheduler — better resource isolation, per-task resource limits, and elastic scaling, but with cold-start latency and a heavier cluster dependency. Most modern teams pick KubernetesExecutor (or the hybrid CeleryKubernetesExecutor) once DAG volume is variable enough that idle Celery workers become a cost or scaling problem.