What is Apache Airflow?

What is Apache Airflow?

Apache Airflow was created at Airbnb in 2014 and open-sourced in 2015. It became an Apache Software Foundation top-level project in 2019. Today it's the de-facto standard for orchestrating batch data pipelines at companies from startups to Fortune 500s.

Unlike cron or simple schedulers, Airflow treats workflows as code. Your pipeline logic lives in a Python file, versioned in git, reviewed in PRs, and deployed like any other software. This makes data pipelines auditable, reproducible, and testable.

The Airflow ecosystem splits into two flavors. Apache Airflow OSS is self-hosted on Docker, Kubernetes, or bare metal — full control, full operational responsibility. Managed Airflow services (Astronomer, AWS MWAA, Google Cloud Composer) run the same DAGs with less ops overhead and a higher price tag.

Why does Airflow matter?

• Automatic retries with exponential backoff replace silent cron failures
• Task-level dependency graphs are enforced at runtime — no more out-of-order ETL
• Web UI ships with full run history, logs, and Gantt views for every task
• Slack/email alerts on failure or SLA miss keep on-call engineers in the loop
• All pipelines live as code in git — pull requests, code review, version history
• Backfilling re-runs historical date ranges with a single CLI command

How does Airflow work?

Airflow has four core components that work together to run your DAGs. The scheduler parses Python DAG files and decides which tasks are ready to run. The executor dispatches those tasks (locally, on Celery workers, or as Kubernetes pods). Workers run the actual task code. A metadata database (Postgres or MySQL) stores DAG state, run history, and task logs.

A minimal Airflow DAG using the TaskFlow API:

# dags/daily_etl.py
from airflow.decorators import dag, task
from datetime import datetime

@dag(schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False)
def daily_etl():

    @task()
    def extract() -> dict:
        return {'rows': fetch_api_data()}

    @task()
    def load(data: dict) -> None:
        write_to_warehouse(data)

    load(extract())

daily_etl()

The TaskFlow API uses Python decorators (@dag, @task) to define tasks and pass data between them automatically via XCom. The older PythonOperator style still works and is common in legacy codebases.

Airflow vs Prefect vs cron

Use cron for one-line schedules with no dependencies. Use Airflow for enterprise batch pipelines with complex dependencies, mature operators, and predictable scheduling. Choose Prefect when you need dynamic workflows, easier local dev, and a cloud-first developer experience.

What can you build with Airflow?

Airflow is the orchestration glue for almost every batch data workload. The most common patterns:

• ETL pipelines — extract from APIs and databases, transform with Python or dbt, load to warehouses on a schedule
• dbt orchestration — run dbt models in dependency order, with retries and Slack alerts when a model fails
• ML training pipelines — schedule feature engineering, model training, evaluation, and model registry updates as a DAG
• API data ingestion — poll REST APIs, handle pagination and rate limits, land data in S3 or Snowflake on a cron schedule
• Data quality checks — run Great Expectations or dbt tests after each pipeline run; alert and halt on failures
• Backfill and reprocessing — re-run historical date ranges with a single CLI command using Airflow's built-in backfill support

Common mistakes (and what to do instead)

• Putting business logic in DAG files — DAG files should define structure and dependencies only. Move actual logic into operators or Python modules imported by tasks.
• Using XCom for large data — XCom is stored in the metadata database. Designed for small values (IDs, paths, counts). Passing large DataFrames through XCom will kill your scheduler.
• Setting catchup=True without thinking — if your DAG start_date is in the past and catchup=True, Airflow will queue every historical interval at once. Default to catchup=False unless you explicitly want backfill behavior.
• Heavy imports at DAG-file level — top-level imports slow scheduler parsing. Put provider imports inside the task function or use lazy imports.
• Trying to do streaming with BranchOperator — Airflow is batch-first. Don't build event-driven logic with BranchOperator — use Kafka or Flink for that.
• Skipping the metadata DB upgrade — every Airflow upgrade ships migrations. Skipping them is the most common reason production schedulers crash after a version bump.

Who is Airflow for?

Airflow is built for data engineers and platform engineers who own batch data reliability. If you write Python, schedule jobs against warehouses or APIs, and need observability across hundreds of tasks, Airflow is almost certainly the right tool.

Teams that benefit most:

• Analytics teams running dbt models in dependency order with SLA monitoring
• Data platform teams orchestrating Spark, Snowflake, and ML training pipelines across the org
• Growth and finance teams scheduling API ingestion from Salesforce, HubSpot, and ad platforms
• ML engineers orchestrating feature engineering, training, evaluation, and model registry updates