Why We Migrated from Airflow to Kubernetes-Native Orchestration

For three years our orchestration ran on a single t3.xlarge Airflow scheduler with 400 DAGs and 12,000 daily task runs. It worked — until it didn't. By Q3 2025 we were paging the on-call every other night for scheduler heartbeat misses, and the data team was rewriting DAGs to avoid the scheduler instead of solving real problems. This is the story of why we migrated to Argo Workflows, what we gained, what we lost, and the migration protocol we'd repeat.

The breaking point

The architecture problem

Side-by-side: Airflow vs Argo

from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from kubernetes.client import models as k8s

transform_task = KubernetesPodOperator(
    task_id="transform_events",
    namespace="dataeng",
    image="registry/transform:2.1",
    cmds=["python"],
    arguments=["/app/transform.py"],
    env_vars=[k8s.V1EnvVar(name="DATE", value="{{ ds }}")],
    resources=k8s.V1ResourceRequirements(
        requests={"cpu": "500m", "memory": "1Gi"},
        limits={"cpu": "2",     "memory": "4Gi"},
    ),
    is_delete_operator_pod=True,
    dag=dag,
)

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: transform-events
  namespace: dataeng-pipelines
spec:
  entrypoint: transform
  arguments:
    parameters:
      - name: date
  templates:
    - name: transform
      inputs:
        parameters:
          - name: date
      container:
        image: registry/transform:2.1
        command: [python]
        args: ["/app/transform.py"]
        env:
          - name: DATE
            value: "{{inputs.parameters.date}}"
        resources:
          requests: { cpu: 500m, memory: 1Gi }
          limits:   { cpu: 2,    memory: 4Gi }

What we gained

• DAG parse time eliminated — WorkflowTemplates are CRDs. There is no parser; kubectl apply is the parse step, and it costs nothing at runtime.
• Scheduler RAM 8 GB → ~200 MB (Argo controller is lightweight; metadata lives in etcd as CRDs).
• Failure isolation per workflow — one team's broken workflow no longer impacts other workflows. Resource quotas per namespace, no shared scheduler slot pool.
• Native retries + exit-handlers — Argo has these as first-class fields on the template. We deleted ~600 LOC of custom Airflow retry/notification code.
• Argo Events for triggers — S3 events, webhooks, cron, calendar, all wired the same way. Replaced 4 separate Airflow sensors with one event-driven CRD per source.

What we lost

• The Airflow UI — Argo's UI is functional, not beloved. Our analysts had built mental models around the Airflow tree view; we spent two weeks retraining them on Argo. The pain was real and worth doing once.
• The airflow.providers.* ecosystem — Airflow has an operator for every SaaS on earth. Argo expects you to build container images. Worth it long-term; painful in week one.
• Backfills are different — Airflow's backfill semantics (run the DAG for date X) translated to "submit a Workflow with parameter date=X." We wrote a thin CLI that wraps this; it's 80 lines of bash and does the job.
• Some teams' Cron habits — folks used to schedule via schedule_interval. Now they use Argo CronWorkflow. Same idea, different YAML — every team had to update their templates.

The migration protocol

• Week 1 — Argo in parallel, no migrations. Stood up the Argo controller in a new namespace. Wrote 3 trivial WorkflowTemplates (a hello-world, a Spark submit, a dbt run) to validate plumbing.
• Weeks 2–3 — Migrate non-critical DAGs first. Picked DAGs that already used KubernetesPodOperator + had idempotent tasks. Wrote a converter script that read Airflow DAG Python and produced WorkflowTemplate YAML. Manual review on every output; trust the script to draft, not to ship.
• Weeks 3–5 — Migrate the critical path. Order-of-magnitude harder. Required Argo Events configuration for upstream triggers (S3, Kafka). Ran both systems in parallel for 10 days; pages on Airflow ramped down as Argo took over each pipeline.
• Week 6 — Decommission Airflow. Disabled all DAGs, kept the instance up for 30 days as fallback, then deleted. The 30-day window cost ~$3,000; it was the cheapest insurance we ever bought.