Apache Iceberg is an open table format for analytic datasets stored as files on object storage. It sits between the raw files (usually Parquet, ORC, or Avro) and the query engines that read them, adding a metadata layer that turns a directory of files into a real table with transactions, schema changes, and consistent reads. Netflix built it to escape the limits of Hive tables; it is now a top-level Apache project with adoption across Spark, Trino, Flink, Snowflake, BigQuery, and Dremio.

The problem it solves

A Hive table is really just "every file under this directory path." That definition breaks in predictable ways: listing millions of files on S3 is slow, two writers can corrupt each other's output, a schema change means rewriting data, and a reader can see a half-finished write. Iceberg replaces directory-listing with an explicit, versioned manifest of which files belong to the table at each point in time. Engines read the manifest instead of listing storage, so planning a query stays fast whether the table has a thousand files or ten million.

How the metadata is structured

Iceberg tracks a table through three layers:

  • Metadata file - a JSON document holding the current schema, partition spec, and a pointer to the current snapshot. Each commit writes a new metadata file.
  • Manifest list - one per snapshot; it names the manifest files and carries per-manifest statistics used to prune partitions.
  • Manifest files - Avro files listing the actual data files plus column-level stats (min/max, null counts) for each one.

Because every data file is enumerated with statistics, the engine can skip files that cannot match a filter before reading a single byte of Parquet. A query for event_date = '2026-06-01' reads only the manifests and data files whose stats overlap that value.

Snapshots, atomic commits, and time travel

Every write produces a new snapshot. A commit is a single atomic swap of the table's current-metadata pointer, which is how Iceberg gives serializable isolation without a separate lock service: a reader pinned to snapshot N keeps seeing snapshot N while a writer creates N+1. Old snapshots remain queryable until you expire them, so you can read the table as it was at a timestamp or roll a bad load back instantly.

-- Read the table as of a past snapshot (Spark SQL)
SELECT * FROM db.events VERSION AS OF 8273920138401;
SELECT * FROM db.events TIMESTAMP AS OF '2026-06-01 00:00:00';

-- Undo a bad write by pointing the table back at a good snapshot
CALL catalog.system.rollback_to_snapshot('db.events', 8273920138401);

Hidden partitioning

This is the feature that most separates Iceberg from Hive. In Hive you partition by a physical column (event_day) and every query has to filter on that exact column or it scans everything. Iceberg stores a partition transform in metadata - for example "partition by day(event_ts)" - and applies it automatically. Users filter on event_ts directly; Iceberg derives which day-partitions to read. You can also change the partitioning later (say, from daily to hourly) without rewriting old data, because the spec is versioned per snapshot.

Schema evolution

Columns are tracked by a stable integer ID, not by name or position. That means you can rename a column, reorder columns, add a nullable column, or widen an int to a long as a pure metadata change - no data rewrite, and no risk of reading the wrong column after a rename. Dropping a column hides it from reads without touching the files.

A minimal write

# PySpark with the Iceberg runtime configured against a catalog
df.writeTo("catalog.db.events") \
  .partitionedBy("days(event_ts)") \
  .createOrReplace()

# MERGE for upserts / CDC apply
spark.sql("""
  MERGE INTO catalog.db.events t
  USING staging.changes s ON t.id = s.id
  WHEN MATCHED THEN UPDATE SET *
  WHEN NOT MATCHED THEN INSERT *
""")

When to reach for it

  • You have large append-and-update datasets on a lake and want warehouse-style guarantees without loading into a warehouse.
  • Multiple engines (Spark for writes, Trino for ad-hoc, Flink for streaming) need one consistent view of the same tables.
  • You need CDC/GDPR-style row deletes, which Iceberg handles through merge-on-read delete files.

Tradeoffs and pitfalls

Iceberg adds operational duties that a pile of Parquet files does not have. Snapshots accumulate, so an expiration job is mandatory or storage and manifest counts grow without bound. Frequent small commits (common with streaming) create many small files and delete files that slow merge-on-read queries until compaction runs - schedule rewrite_data_files and rewrite_manifests. The catalog matters too: the table is only as consistent as the component that performs the atomic pointer swap (Hive Metastore, AWS Glue, Nessie, or the JDBC/REST catalog), and mixing catalogs against one table invites split-brain.

Iceberg vs Delta Lake vs Hudi

All three are open table formats with ACID commits and time travel. Delta Lake grew out of the Databricks/Spark world and uses a transaction log of JSON+Parquet checkpoints; its open-source engine support has widened but its center of gravity is Spark. Hudi optimizes for streaming upserts and record-level indexes. Iceberg's distinguishing bets are engine neutrality, hidden partitioning, and ID-based schema evolution, which is why warehouse vendors adopted its spec for external tables. The practical decision usually follows your primary engine and which format your warehouse reads natively.