Skip to content
Path 01 · Most in-demand

Build the pipelinesevery data team runs on.

Own Airflow DAGs, dbt models, Spark jobs, and the lakehouse they feed. Ship 6 production projects, each one interview-ready.

See the path
After this path

What you'll actually be able to do.

Not "you'll know about Airflow." What you'll ship, debug, and defend in an interview.

You’ll be the person who

  • Design pipelines from scratch, not follow tutorials
  • Debug Spark + Airflow on-call without panic
  • Model data in dbt that survives code review
  • Own bronze → silver → gold end-to-end
  • Explain your architecture in an interview

And the market pays you for

Design DAGsNot follow tutorials
Model in dbtThat pass review
Own pipelinesNot just contribute
Handle on-callWithout panic
System architecture

The system you'll build by the end.

A production reference architecture — not a toy demo. Every node maps to a course or project in this path.

01 · Source
Postgres
Stripe
Segment
02 · Ingest
Airbyte
Fivetran
Custom
03 · Storage
S3 + Iceberg
Snowflake
04 · Transform
dbt
Spark
05 · Serve
Metabase
API
Reverse ETL
Orchestration: AirflowEvery node → a course + project
Your path

From week one to capstone.

A realistic 5-stage timeline. Go faster if you already have pieces; slower if you're brand new.

  1. 01Week 1–3

    Foundations

    Docker, Postgres, SQL patterns, Git flow

    Study
    sql-masterypython-de
  2. 02Week 4–9

    Batch Pipelines

    Airflow, dbt incremental, Spark, partitioning

    Study
    airflowdbtdata-modelingspark
    Ship
    ecommerce-data-warehouseecommerce-metrics-layer
  3. 03Week 10–14

    Quality

    GE suites, contracts, incident response

    Study
    data-observabilitygovernanceapi-integration
    Ship
    dataguard-observabilityapi-data-ingestion
  4. 04Week 15–20

    Streaming intro

    Kafka basics, CDC, when NOT to use streaming

    Study
    streaming-fundamentalskafka-streamsevent-design
    Ship
    schema-evolution-contracts
  5. 05Week 21–24

    Ship capstone

    Medallion lakehouse, deployed, documented

    Study
    cicd-deploymentcost-optimization
    Ship
    iceberg-lakehouse
Capstone project

One project, endlessly talkable.

Every path ends with a flagship capstone you'll ship, write up, and walk through in every interview loop.

P04 · CapstoneIceberg lakehouse on S3

Your capstone — the system you'll talk through in every interview.

S3IcebergdbtSparkAirflowTerraform
Plus projects
ecommerce-data-warehouselogistics-batch-pipelinedataguard-observability

What you’ll ship

  • 01Ingest 50M rows/day from 3 sources
  • 02Bronze → silver → gold with dbt + Spark
  • 03Automated data contracts in CI
  • 04SLA dashboards and alerting
  • 05Cost optimization built-in
Proof

Questions you'll confidently answer.

These are real interview questions for Core Data Engineer roles. If you can answer all four with a story from your capstone, you're ready.

Q1

Walk me through an idempotent backfill for a 2B-row dimension

Q2

How would you design a data contract for an unreliable upstream?

Q3

When do you reach for streaming and when is it overkill?

Q4

How do you catch a bad load before it hits gold?

Why this matters: Most courses let you hide behind passive video-watching. ai-de projects force you into the exact failure modes interviewers probe for — so when you sit in the interview, you’ve already lived the answer.
Skills · syllabus

Stack you'll learn.

Not memorized — operated. Each tool is taught inside a project, not an isolated lecture.

SQLAirflowdbtSparkKafkaIcebergTerraformGreat ExpectationsGitDocker
Not quite the right fit? Explore the other 4 paths
Fastest-growing roleAI Data EngineerGo Senior / staff trackData Platform EngineerGo Closest to the businessAnalytics / Product Data EngineerGo Senior+ AI infraAI Platform EngineerGo
Your move

Start building your first system — today.

Module 01 is free. No card. Ship something real this weekend.

Compare all 5 paths
Press Cmd+K to open