The complete data engineer skills checklist (2026)
The complete data engineer skills checklist — SQL, Python, dbt, Spark, Kafka, Iceberg, LLM pipelines, and vector databases. The exact tech stack hiring managers look for.
To become a data engineer in 2026, you must master foundational languages (SQL, Python), data modeling, cloud platforms (Snowflake, AWS), orchestration (Airflow), modern transformation (dbt), and distributed processing (Spark, Kafka). The highest-paid engineers are also mastering AI data pipelines and vector databases. Start with /learn/sql-mastery and /learn/python-de — every other skill below depends on these two.
Core languages & foundations
The non-negotiable foundation. Every other skill on this page assumes fluency in SQL and Python. Skip these and the rest doesn't compile.
Advanced SQL
- Window functions, CTEs, recursive queries
- Query plans + index design
- OLAP query patterns (cubes, rollups)
- Snowflake / BigQuery / Postgres dialects
Python for data engineering
- Pandas / Polars / DuckDB for in-process data
- Idempotent ETL scripts + checkpointing
- API ingestion with pagination + retries
- pytest + mock for pipeline tests
Cloud fundamentals
- S3 / GCS object storage primitives
- IAM, KMS, VPC + private networking
- EC2 / Lambda for compute
- Terraform basics for reproducibility
Master SQL in 4 hours, hands-on.
Window functions, CTEs, query plans — the curriculum that turns analysts into engineers.
The modern data stack
The transformation + orchestration layer. dbt + Airflow + Snowflake/BigQuery is the canonical 2026 stack. Knowing this trio opens 80% of data engineering roles.
dbt
- Sources, staging, intermediate, marts
- Tests + dbt contracts
- Macros + Jinja templating
- Incremental models + snapshots
Apache Airflow
- DAGs, TaskFlow API, dynamic task mapping
- Sensors + retries + SLAs
- Kubernetes / Celery executors
- Connection + variable management
Dimensional modeling
- Star schema vs snowflake vs Data Vault
- Slowly-changing dimensions (Type 1/2/6)
- Grain definition + accumulating snapshots
- Conformed dimensions across marts
Big data & streaming
Once your data exceeds a single warehouse warehouse, you need distributed compute and event-driven systems. Skip these until you have a real volume or latency requirement; learn them when you do.
Apache Spark
- DataFrame API + Spark SQL
- Catalyst optimizer + AQE
- Partitioning + shuffle tuning
- Structured Streaming basics
Kafka + streams
- Topics, partitions, consumer groups
- Exactly-once semantics
- Kafka Streams stateful processing
- Schema Registry + Avro / Protobuf
Stateful streaming
- Event-time semantics + watermarks
- Stateful operators + checkpointing
- Windowed aggregations
- CDC patterns with Debezium + Flink
AI data systems & MLOps
The highest-growth area in data engineering. AI models are useless without structured, clean data to feed them — and the engineers who can build those data systems are the highest-paid in the field.
LLM pipelines
- Chunking + tokenization at scale
- Deduplication (MinHash / SimHash)
- Safety filtering + PII redaction
- Dataset versioning with DVC
Retrieval pipelines
- Embedding models + vector indexes
- Hybrid search (BM25 + dense)
- Reranking + permission filtering
- Eval harnesses (RAGAS, custom judges)
Feature stores
- Offline / online dual-store
- Point-in-time correctness
- Feast / Tecton patterns
- Train/serve skew detection
DataOps & engineering standards
The platform-engineering layer that separates senior from staff. Treat data systems like software: tests, CI, observability, governance.
CI/CD for data
- PR-gated dbt tests
- Environment promotion (dev → stage → prod)
- Schema drift checks
- GitOps deployment patterns
Data observability
- Freshness + volume anomaly detection
- Column-level lineage tracking
- SLO / SLA definition
- Monte Carlo / Bigeye / OpenLineage
Apache Iceberg
- Table format internals + manifest files
- Hidden partitioning + time travel
- Schema evolution patterns
- Comparison vs Delta Lake and Hudi
Build all 21 skills as one career path.
Pick the AI Data Engineer track to ship all 21 skills + 6 portfolio projects in a single 6-month progression. Mentor-reviewed at every milestone.
How to learn these skills
Pick one skill per phase and pair it with a project. Don't try to learn three at once — the depth required for production-grade work means you'll end up with shallow knowledge of three things instead of solid grounding in one.
The recommended order is in Data Engineer Roadmap 2026. Start there, then come back to this page as a checklist.
Frequently asked questions
Start shipping.
Three steps from a guide to a job-ready portfolio. Pick one and start now — the rest will follow.
Take the skill
Self-paced module with code, exercises, and a deliverable. Free preview, paid completion.
Start S0X · Sql Mastery →Ship the project
Production-grade build with starter kit + mentor code review. The artifact that gets you interviews.
Open P0X · Ecommerce Data Warehouse →Pick a career path
The full progression — skills + projects + interview prep — for the role you actually want.
See paths →