Apache Iceberg Table Format

Open table format mastery — time travel, schema evolution, and multi-engine analytics.

Hive is dying. Iceberg is the standard the entire open-data ecosystem (Spark, Trino, Flink, Snowflake, BigQuery, Dremio) is converging on. Engineers who own table format decisions own the lakehouse — and the lakehouse is where the data lives.

What you’ll be able to do

• Create and manage Iceberg tables with time travel, snapshots, and rollback
• Implement schema and partition evolution safely without breaking downstream consumers
• Configure REST / Hive / Glue / Nessie catalogs for multi-engine access from Spark, Trino, and Flink
• Run production Iceberg with compaction, streaming CDC, and zero-downtime Hive migrations

Curriculum

What you’ll build

• A working Iceberg lakehouse with a REST catalog, partitioned + clustered tables, time travel + rollback, and Spark + Trino reading the same source of truth
• A schema-evolution playbook covering the 5 safe changes (add nullable, rename, type-promote, add partition, drop column) and how to gate breaking changes in CI
• A streaming-CDC pipeline that lands Kafka events into Iceberg with exactly-once semantics, equality deletes, and < 1-minute end-to-end latency
• A production-ops runbook with compaction + snapshot-expiration cron, orphan-file cleanup, file-count + snapshot-age dashboards, and a Hive → Iceberg migration plan

Without Iceberg internals, your data lake silently turns into a data swamp.

WHAT GOES WRONG

• The 4 M file-handle incident — ingestion writes 10 K small files/hour (50 KB each); six months later a single SUM() query opens 4 million file handles and times out
• The 12 broken consumers — a column added to the events table; downstream pipelines with strict schemas fail immediately; on-call eats 4 hours hunting consumers
• The two-engine divergence — Spark says 10 M rows, Trino says 9.8 M; both pointing at 'the same' table via different catalog endpoints; the gap had been growing for 3 weeks
• The 2.7 M small-file table — built for a demo, no compaction policy, no retention, no monitoring; same root cause as every previous incident; production from day one is the only fix

What is Apache Iceberg Table Format?

Apache Iceberg is an open table format for large-scale analytics datasets. It provides time travel, schema evolution, and partition evolution on top of data lake storage like S3, enabling multi-engine access from Spark, Trino, Flink, and more. Adopted by Apple, Netflix, and LinkedIn, Iceberg is becoming the standard table format for modern data lakehouses.

Why this matters in production

Production data teams migrate from Hive to Iceberg for reliable schema evolution and ACID transactions on data lakes. Apple manages exabytes of data with Iceberg. Without proper table format management, data lakes become unreliable swamps with broken schemas and inconsistent reads.

Common use cases

• Building data lakehouses with ACID transactions and time travel
• Implementing schema evolution without breaking downstream consumers
• Enabling multi-engine analytics with Spark, Trino, and Flink on the same tables
• Running streaming CDC pipelines that merge into Iceberg tables
• Managing partition evolution for changing query patterns without data rewrites
• Migrating from Hive tables to Iceberg for improved reliability

Iceberg vs alternatives

Related skills

• Iceberg tables are most commonly read and written using Apache Spark.
• Streaming CDC into Iceberg tables builds on concepts from Streaming Fundamentals.
• Understanding storage optimization connects to Data Warehouse Internals.

Why this skill matters

Iceberg is the table format the next decade of data engineering is built on. Mid-to-senior data engineers at Apple, Netflix, LinkedIn, and Stripe are paid for exactly this skill — turning unreliable data lakes into ACID lakehouses that streaming, batch, and ML all read from the same source of truth.

Common questions about Iceberg