Apache Iceberg vs Delta Lake vs Hudi: The Table Format War Explained

Marcus Tran|Dec 4, 2025|11 min read

The open table format competition between Apache Iceberg, Delta Lake, and Apache Hudi shapes how the next generation of data lakehouses stores, queries, and governs petabyte-scale datasets. All three are mature, ACID-capable, and run in production at scale. The differences are in how they implement core features and who supports them.

Key Takeaways (TL;DR)

  • Apache Iceberg has the broadest engine compatibility and the cleanest separation between compute and storage.
  • Delta Lake remains the default on Databricks, but the gap has narrowed since Delta went fully open-source.
  • Apache Hudi excels at incremental ingestion and CDC workloads, but its complexity and smaller community make it harder to justify for greenfield projects.
  • All three support time travel, schema evolution, ACID transactions, and (Iceberg only, natively) partition evolution.
  • Starting fresh, Iceberg is the safest bet. Already on Databricks, Delta Lake is fine. Heavy CDC requirements with Hudi expertise on the team, stick with Hudi.

What Problem Do Open Table Formats Solve?

Traditional data lakes built on Parquet or ORC files have no concept of transactions. If a Spark job writes 500 files and crashes at file 347, the table is corrupt. There is no consistent snapshot, no rollback, no schema enforcement. You end up writing custom cleanup scripts and hoping downstream consumers do not read half-written data.

Open table formats add a metadata layer on top of the existing file formats (still Parquet under the hood, usually) that provides:

  • ACID transactions for concurrent reads and writes
  • Time travel to query historical snapshots
  • Schema evolution without rewriting data
  • Partition evolution without data migration
  • Row-level updates and deletes (critical for GDPR compliance)

Architecture Deep Dive

Apache Iceberg: The Catalog-Centric Approach

Iceberg uses a three-layer metadata architecture. A catalog (Hive Metastore, AWS Glue, Nessie, or REST catalog) points to the current metadata file. That metadata file contains a list of manifest lists, each representing a snapshot. Each manifest list points to manifest files that track individual data files along with column-level statistics (min/max values, null counts).

This design means Iceberg can plan queries by reading only the metadata files it needs, skipping entire manifest groups based on partition pruning and column statistics. For a 10TB table, query planning might read 50KB of metadata instead of listing millions of files from object storage. Because Iceberg tracks data at the file level with rich statistics, it is engine-agnostic by design - any engine that can read the metadata format can query the table.

Delta Lake: The Transaction Log

Delta Lake centers on a JSON-based transaction log (the _delta_log directory). Every operation - write, delete, schema change, compaction - appends a new JSON file to this log. Periodically, Delta creates checkpoint files (Parquet format) that consolidate the log entries for faster reads. To reconstruct the current table state, a reader starts from the latest checkpoint and replays subsequent JSON log entries, conceptually similar to a write-ahead log in databases.

Delta's architecture was originally tightly coupled to Spark and the Databricks runtime. The open-sourcing of Delta Lake 3.0 and the UniForm initiative answer that criticism, but the Spark-first design is still visible in the codebase.

Apache Hudi: The Timeline

Hudi (Hadoop Upserts Deletes and Incrementals) organizes everything around a timeline of actions stored in the .hoodie metadata directory. Each action (commit, compaction, clean, rollback) is recorded as an instant on this timeline. What makes Hudi architecturally distinct is its two storage types:

  • Copy-on-Write (CoW): Updates rewrite entire files. Read performance is excellent because there are no merge operations at query time. Write amplification can be significant.
  • Merge-on-Read (MoR): Updates go to delta log files that get merged during reads or background compaction. Write performance is better, but reads require merging base files with deltas.

This dual-storage model gives Hudi fine-grained control over the read/write performance tradeoff, which is why it shines in near-real-time ingestion. It also adds operational complexity - compaction strategies, cleaning policies, and clustering configurations all need attention.

Feature Comparison

Feature Apache Iceberg Delta Lake Apache Hudi
ACID Transactions Yes (optimistic concurrency) Yes (optimistic concurrency) Yes (timeline-based)
Time Travel Snapshot-based, configurable retention Version-based, default 30 days Timeline-based, configurable
Schema Evolution Full (add, drop, rename, reorder columns) Add/rename columns; mergeSchema on write Add columns, limited type promotion
Partition Evolution Yes, without rewriting data No (requires full rewrite) No (requires full rewrite)
Hidden Partitioning Yes (year, month, day, hour, bucket, truncate) No (Liquid Clustering as alternative) No
Row-Level Deletes Copy-on-write and merge-on-read Deletion vectors (since 3.0) CoW and MoR native
Streaming Ingest Supported (Flink, Spark Structured Streaming) Strong (native Spark Structured Streaming) Excellent (DeltaStreamer, Flink)
Engine Support Spark, Flink, Trino, Presto, Dremio, Snowflake, BigQuery, Athena, StarRocks Spark, Flink (limited), Trino (via connectors), Databricks Spark, Flink, Presto, Trino, Hive
Cloud Catalog REST catalog, Glue, Nessie, Polaris, Unity (via UniForm) Unity Catalog, Glue (limited) Hive Metastore, Glue
Governance Apache Software Foundation Linux Foundation (Databricks-led) Apache Software Foundation

Code Examples: Common Operations

All examples use PySpark 3.5+.

Creating a Table

# Apache Iceberg
spark.sql("""
    CREATE TABLE catalog.db.events (
        event_id BIGINT,
        user_id BIGINT,
        event_type STRING,
        event_ts TIMESTAMP,
        payload STRING
    )
    USING iceberg
    PARTITIONED BY (days(event_ts))
""")

# Delta Lake
spark.sql("""
    CREATE TABLE delta_db.events (
        event_id BIGINT,
        user_id BIGINT,
        event_type STRING,
        event_ts TIMESTAMP,
        payload STRING
    )
    USING delta
    PARTITIONED BY (date_trunc('day', event_ts))
""")

# Apache Hudi — typically done via DataFrame write
df.write.format("hudi") \
    .option("hoodie.table.name", "events") \
    .option("hoodie.datasource.write.recordkey.field", "event_id") \
    .option("hoodie.datasource.write.precombine.field", "event_ts") \
    .option("hoodie.datasource.write.partitionpath.field", "event_ts") \
    .option("hoodie.datasource.write.table.type", "COPY_ON_WRITE") \
    .mode("overwrite") \
    .save("/data/hudi/events")

Iceberg's days(event_ts) is a hidden partition transform - users query event_ts directly and Iceberg figures out the partitioning. Hudi requires five Hoodie-specific options just to create a table.

Upserts (Merge)

# Apache Iceberg (Spark 3.4+)
spark.sql("""
    MERGE INTO catalog.db.events t
    USING updates s
    ON t.event_id = s.event_id
    WHEN MATCHED THEN UPDATE SET *
    WHEN NOT MATCHED THEN INSERT *
""")

# Delta Lake
spark.sql("""
    MERGE INTO delta_db.events t
    USING updates s
    ON t.event_id = s.event_id
    WHEN MATCHED THEN UPDATE SET *
    WHEN NOT MATCHED THEN INSERT *
""")

# Apache Hudi
updates_df.write.format("hudi") \
    .option("hoodie.table.name", "events") \
    .option("hoodie.datasource.write.recordkey.field", "event_id") \
    .option("hoodie.datasource.write.precombine.field", "event_ts") \
    .option("hoodie.datasource.write.operation", "upsert") \
    .mode("append") \
    .save("/data/hudi/events")

Iceberg and Delta have nearly identical MERGE syntax (both follow the SQL standard). Hudi's DataFrame API works but is harder to reason about for complex merge logic.

Time Travel

# Apache Iceberg — by snapshot ID or timestamp
spark.sql("SELECT * FROM catalog.db.events VERSION AS OF 123456789")
spark.sql("SELECT * FROM catalog.db.events TIMESTAMP AS OF '2025-11-01 00:00:00'")
spark.sql("SELECT * FROM catalog.db.events.snapshots")

# Delta Lake — by version or timestamp
spark.sql("SELECT * FROM delta_db.events VERSION AS OF 42")
spark.sql("SELECT * FROM delta_db.events TIMESTAMP AS OF '2025-11-01'")
spark.sql("DESCRIBE HISTORY delta_db.events")

# Apache Hudi — timestamp-based
spark.read.format("hudi") \
    .option("as.of.instant", "20251101000000000") \
    .load("/data/hudi/events")

Schema Evolution

# Apache Iceberg — full schema evolution
spark.sql("ALTER TABLE catalog.db.events ADD COLUMNS (device STRING)")
spark.sql("ALTER TABLE catalog.db.events RENAME COLUMN payload TO event_payload")
spark.sql("ALTER TABLE catalog.db.events DROP COLUMN device")
spark.sql("ALTER TABLE catalog.db.events ALTER COLUMN user_id TYPE BIGINT")

# Delta Lake — add and rename
spark.sql("ALTER TABLE delta_db.events ADD COLUMNS (device STRING)")
spark.sql("ALTER TABLE delta_db.events RENAME COLUMN payload TO event_payload")
# Column drop supported since Delta 3.0 with column mapping enabled

# Apache Hudi — add columns only (broad support)
spark.sql("ALTER TABLE hudi_db.events ADD COLUMNS (device STRING)")

Iceberg leads on schema evolution. The ability to rename, reorder, and drop columns without rewriting data is a genuine operational advantage. In Delta, a column rename can require a full table rewrite if column mapping was not enabled from the start - a hard mistake to undo.

Why Iceberg Has the Momentum

Engine Neutrality

Iceberg works well across engines. The same Iceberg table and catalog produce consistent results from Spark, Trino, Flink, and Athena. Delta Lake, by contrast, hits connector quirks, version mismatches, and features that only work on the Databricks runtime. Snowflake natively reads Iceberg tables, BigQuery added Iceberg support, and AWS made Iceberg the default for Athena and EMR. Apple, Netflix, and LinkedIn run Iceberg at large scale.

Partition Evolution

A Delta table partitioned by day that needs to switch to hourly partitioning requires a full backfill: create a new table, rewrite all historical data, swap downstream consumers, delete the old table. With Iceberg:

spark.sql("""
    ALTER TABLE catalog.db.events
    ADD PARTITION FIELD hours(event_ts)
""")
# New data uses hourly partitions
# Old data stays in daily partitions
# Queries work correctly across both — Iceberg handles it transparently

No data rewrite, no downtime. This alone justifies the format choice for any table that might change partitioning over its lifetime.

The REST Catalog Standard

Iceberg's REST catalog specification is becoming the standard API for table format catalogs - a vendor-neutral HTTP interface for catalog operations, so tooling does not need format-specific connectors. Snowflake's managed Iceberg tables, Tabular (now part of Databricks), and Dremio's Arctic all implement it. Databricks Unity Catalog exposes Iceberg-compatible metadata through UniForm.

The Caveats

Iceberg is not perfect. Small file management still requires regular compaction. The ecosystem for Iceberg-native CDC ingestion is not as mature as Hudi's DeltaStreamer. On Databricks, Delta Lake's integration is smoother for day-to-day operations because the optimizations are built into the runtime. Iceberg's documentation is comprehensive but dry, and many online tutorials are outdated or vendor-specific.

When Delta Lake Still Makes Sense

  • Databricks-native performance: Photon engine optimizations, predictive I/O, auto-compaction, and Liquid Clustering. On Databricks, Delta tables will be faster out of the box.
  • Liquid Clustering: Delta's answer to partition evolution, automatically reorganizing data based on query patterns without manual partition management.
  • Simpler mental model: The transaction log is easier to explain and debug than Iceberg's multi-layer metadata.
  • UniForm: Delta tables can expose Iceberg-compatible metadata, so you can write with Delta and read with Iceberg-compatible engines, reducing lock-in concerns.

If your entire stack runs on Databricks with no plan to change, Delta Lake is the pragmatic choice.

When Hudi Is the Right Choice

  • High-frequency CDC ingestion: Hudi's DeltaStreamer and native change-data-capture understanding make it excellent for streaming database replicas into a data lake. Syncing hundreds of MySQL/Postgres tables in near-real-time is what its tooling is built for.
  • Fine-grained write control: The CoW/MoR choice, plus configurable indexing (Bloom, HBase, bucket) and compaction strategies, exposes knobs that Iceberg and Delta do not.
  • Record-level indexing: Hudi maintains indexes on record keys, making point lookups and upserts on specific records faster than file-level scanning.

The tradeoff is complexity: more configuration, more monitoring, and more expertise than either Iceberg or Delta, with a smaller community.

Migration Considerations

From Raw Parquet/ORC

Start with a metadata-only migration if possible. Both Iceberg and Delta support in-place conversion that adds the metadata layer without rewriting data files - dramatically faster than a full rewrite.

# Iceberg: migrate existing Parquet table
spark.sql("""
    CALL catalog.system.migrate('db.existing_parquet_table')
""")

# Delta: convert existing Parquet
spark.sql("""
    CONVERT TO DELTA parquet.`/path/to/parquet/table`
    PARTITIONED BY (date STRING)
""")

Between Table Formats

There is no magic converter between formats. The practical approach:

  1. Create the target table with the desired schema and partitioning.
  2. Read from source format, write to target format using Spark.
  3. Validate row counts, schema, and partition structure.
  4. Update downstream consumers (always the hardest part).
  5. Keep the source table around for at least two weeks as a rollback option.

For large tables (50TB+), migrate partition by partition rather than all at once. This keeps the operation resumable and reduces blast radius.

Testing Your Migration

# Validate row counts after migration
source_count = spark.table("old_format.events").count()
target_count = spark.table("new_format.events").count()
assert source_count == target_count, f"Row count mismatch: {source_count} vs {target_count}"

# Validate schema compatibility
source_schema = spark.table("old_format.events").schema
target_schema = spark.table("new_format.events").schema

for field in source_schema:
    assert field.name in [f.name for f in target_schema], f"Missing field: {field.name}"

# Spot-check data quality on key columns
spark.sql("""
    SELECT
        (SELECT COUNT(DISTINCT user_id) FROM old_format.events) as old_users,
        (SELECT COUNT(DISTINCT user_id) FROM new_format.events) as new_users
""").show()

Performance: What the Benchmarks Don't Tell You

Query planning overhead: Iceberg's rich file-level statistics let the query planner skip irrelevant files without reading them. For tables with thousands of partitions, this is measurable - Iceberg can plan queries 3-5x faster than Delta on wide tables with 500+ columns.

Write amplification: For update-heavy workloads, Hudi's MoR mode has the lowest write amplification because it appends delta logs instead of rewriting files. Iceberg and Delta (with deletion vectors) have improved, but Hudi still edges them out for workloads with >20% update ratio.

Small file problem: All three suffer from it and all three have compaction mechanisms. Delta's auto-compaction on Databricks is the most hands-off. Iceberg requires explicit compaction jobs. Hudi's compaction is deeply configurable but demands attention.

Concurrent writers: Iceberg handles concurrent writers with optimistic concurrency and retry mechanisms. Delta's conflict resolution is tightly integrated with Spark. Hudi supports concurrent writes but requires careful configuration of lock providers.

The Convergence Trend

These formats are converging. Delta added deletion vectors (an Iceberg concept). Iceberg added merge-on-read (a Hudi concept). Hudi added metadata-based file listing (an Iceberg concept). Databricks acquired Tabular (the Iceberg company), and Delta's UniForm writes Iceberg-compatible metadata. The interoperability layers are getting better and the feature gaps are shrinking, so the format you choose matters less for future flexibility than it once did.

Practical Recommendations

  1. Choose Iceberg if: you use multiple query engines, want partition evolution, care about long-term vendor neutrality, or are starting a greenfield platform - especially with Snowflake, BigQuery, or Athena in the stack.
  2. Choose Delta Lake if: you are on Databricks and plan to stay. Native integration, auto-optimization, and Liquid Clustering make it the path of least resistance; UniForm mitigates lock-in.
  3. Choose Hudi if: your primary use case is streaming CDC ingestion from transactional databases, you need fine-grained write control, and your team has (or can build) Hudi expertise.
  4. Don't migrate for the sake of it: if your current format works, the cost of migration (engineering time, risk, downstream changes) often outweighs the theoretical benefits.

Leave a Comment