Delta Lake is an open table format that turns a directory of Parquet files into a transactional table. It originated at Databricks and is the storage layer behind the lakehouse architecture: warehouse-style guarantees - ACID transactions, schema enforcement, time travel, and efficient upserts - directly on data-lake files, with no separate database to load into.

The transaction log is the table

Delta's core is the transaction log (_delta_log/), a sequence of ordered JSON commits next to the Parquet data. Each commit records which files were added and removed by an operation. The current table state is the log replayed to the latest version; periodic Parquet checkpoints summarize the log so readers don't replay thousands of entries. Because a commit is an atomic addition to this log, concurrent writers get optimistic concurrency control and readers always see a consistent snapshot - never a half-written state.

What the log buys you

  • ACID transactions - appends, overwrites, merges, and deletes are all-or-nothing.
  • Time travel - query any past version by number or timestamp, and roll back bad writes.
  • Schema enforcement & evolution - writes that don't match the schema are rejected; evolution is explicit, preventing silent corruption.
  • Upserts and deletes - MERGE handles CDC apply and GDPR deletes that plain Parquet can't.
-- Time travel
SELECT * FROM events VERSION AS OF 42;
SELECT * FROM events TIMESTAMP AS OF '2026-06-01';

-- Upsert / CDC apply
MERGE INTO events t USING changes s ON t.id = s.id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Operational features

Delta adds maintenance commands that matter at scale: OPTIMIZE compacts small files (and with Z-ordering, co-locates data on chosen columns to improve skipping); VACUUM removes data files no longer referenced by the log after a retention window. As with any log-based format, skipping VACUUM lets storage grow, while running it too aggressively breaks time travel - keep a retention window that matches how far back you need to read.

Delta vs Iceberg vs Hudi

All three are open table formats with ACID commits, time travel, and merge-on-read deletes. Delta's center of gravity is Spark and Databricks, where its tooling is deepest; its open-source engine support has broadened (Trino, Flink, and others read it), and the UniForm feature can expose a Delta table with Iceberg-compatible metadata. Iceberg bets on engine neutrality and hidden partitioning and is the spec most warehouse vendors adopted for external tables; Hudi specializes in streaming upserts. If your platform is Databricks/Spark-first, Delta is the path of least resistance; if you need broad multi-engine neutrality, weigh Iceberg.

Note on the category

Delta Lake is technically a table format (a metadata layer over Parquet), not a file format like Parquet or ORC - it stores its data as Parquet underneath. It's grouped here alongside file formats because it occupies the same layer of the stack in practice: how your bytes live on the lake.