What is Apache Iceberg?

What is Apache Iceberg?

Iceberg was created at Netflix in 2017 to solve the limitations of the Hive table format at scale: slow partition discovery, unsafe concurrent writes, and the inability to change schemas without rewriting data. It was open-sourced and donated to the Apache Software Foundation in 2018.

Today Iceberg is the dominant open table format. It has native support in Spark, Flink, Trino, Presto, DuckDB, Snowflake, BigQuery, and Redshift. The emerging REST Catalog spec is making it the universal interface for multi-engine lakehouses.

The Hive table format treats partitions as directories — fragile, slow, and unsafe under concurrent writes. Iceberg replaces directory-based partitions with a three-layer metadata tree that records every snapshot, manifest, and data file. The result: O(1) query planning, ACID guarantees, and safe schema changes without touching the data files.

Why does Iceberg matter?

• ACID transactions via optimistic concurrency — concurrent writers no longer corrupt tables
• Add, drop, rename, or reorder columns with metadata-only operations — no data rewrite
• Hidden partitioning eliminates the manual WHERE partition_date = X filter on every query
• Time travel: query any past snapshot by timestamp or ID for audit, debug, and recovery
• Metadata tree gives O(1) query planning regardless of file count or table size
• One open table works across Spark, Flink, Trino, DuckDB, Snowflake — no format lock-in

How does Iceberg work?

Iceberg is a three-layer metadata hierarchy sitting on top of immutable data files in object storage:

1. Catalog — a pointer (Glue, Nessie, REST, JDBC) that maps a table name to its current metadata file. Engines start here.
2. Metadata file — JSON snapshot log + schema history + partition spec + sort order. One per table version.
3. Manifest list + manifests — Avro files listing the data files in each snapshot, with per-file column stats for pruning.
4. Data files — Parquet, ORC, or Avro objects in S3/GCS/ADLS. Immutable; deletes are separate delete files (MoR mode).

Every commit writes a new metadata file and atomically swaps the catalog pointer. Readers always see a consistent snapshot, even mid-write.

-- Create an Iceberg table with hidden partitioning
CREATE TABLE catalog.db.events (
    event_id     BIGINT,
    user_id      BIGINT,
    event_type   STRING,
    event_ts     TIMESTAMP
)
USING iceberg
PARTITIONED BY (days(event_ts));   -- hidden partition

-- Time travel: query yesterday's snapshot
SELECT * FROM catalog.db.events
TIMESTAMP AS OF '2026-05-22 00:00:00';

-- Schema evolution: add column (no rewrite, milliseconds)
ALTER TABLE catalog.db.events
ADD COLUMN session_id STRING;

Iceberg vs Delta Lake vs Hudi

Verdict: Choose Iceberg for multi-engine architectures and open interoperability — especially if Trino, Flink, DuckDB, or Snowflake share the same tables. Choose Delta Lake if you are all-in on Databricks and want Liquid Clustering. Choose Hudi for CDC-heavy workloads where MoR write speed matters more than read latency.

Catalogs, compaction, and the lifecycle you have to operate

A bare Iceberg table is incomplete — production deployments depend on three operational layers most tutorials skip:

• Catalog choice. Hadoop catalogs (file-based) work for a single engine but break cross-engine discovery. Use a REST catalog — AWS Glue, Project Nessie, Apache Polaris, or Tabular — to share tables across Spark, Trino, and DuckDB simultaneously with consistent metadata.
• Compaction. Streaming writes (Flink, Spark Structured Streaming) create many small files. Without a regular rewrite_data_files job, query latency degrades within days. Schedule compaction nightly on hot tables.
• Snapshot expiry. Every write creates a snapshot. Old snapshots keep data files alive in S3 and bloat the metadata tree. Run expire_snapshots weekly with a retention window (typically 7-30 days).

Done right, Iceberg becomes invisible infrastructure. Done wrong, you ship a lakehouse that gets slower every week.

Common mistakes (and what to do instead)

• Not running compaction — small files from streaming writes degrade query performance. Schedule CALL catalog.system.rewrite_data_files() on hot tables.
• Forgetting to expire snapshots — old snapshots keep data files alive and bloat metadata. Run expire_snapshots weekly with a 7-30 day retention window.
• Wrong write mode for CDC — copy-on-write rewrites files on every update (fast reads, slow writes). Merge-on-read appends delete files (fast writes, slower reads). Match to your update frequency.
• Choosing partition spec before profiling queries — partitioning by days(event_ts) is useless if every query filters by user_id. Profile first; partition evolution lets you change it later without a rewrite.
• Skipping the REST catalog — a Hadoop or file-based catalog locks you to one engine. Use Glue, Nessie, or Polaris to keep the multi-engine promise of Iceberg alive.
• Treating Iceberg as a query engine — Iceberg is just a table format. You still need Spark, Flink, Trino, or DuckDB to read and write it.

Who is Iceberg for?

Iceberg is built for data platform engineers and senior data engineers who own the lakehouse layer. If your data lives in S3, GCS, or ADLS and you need to share tables across multiple query engines without copying data, Iceberg is the right table format.

Teams that benefit most:

• Platform teams unifying batch and streaming on the same tables — Flink writes, Spark reads, Trino queries
• Analytics teams escaping Hive's partitioning pain and slow planning on multi-petabyte fact tables
• Compliance-heavy orgs that need immutable audit trails via time travel for GDPR, SOC 2, and incident forensics
• ML teams sharing feature tables across Spark training jobs and Snowflake serving without ETL between them