Apache Parquet is the standard columnar storage format for analytic data. Instead of writing a table row by row, Parquet stores each column together, which is what makes it fast and cheap for the queries data warehouses and lake engines actually run. It is the default file format behind Spark, Trino, Snowflake external tables, BigQuery exports, and table formats like Iceberg and Delta Lake.

Columnar layout, and why it matters

Analytic queries usually touch a few columns out of many - SELECT region, sum(amount) on a 60-column table. A row format reads every byte of every row to get at two columns. Parquet stores columns separately, so the engine reads only the two it needs and skips the rest. Because values in one column share a type and tend to repeat, they also compress far better than mixed row data. The result is less I/O and smaller files for the same data.

Row groups, column chunks, and statistics

A Parquet file is divided into row groups (horizontal slices, often ~128MB). Within each row group, every column is a column chunk broken into pages. Crucially, each chunk carries statistics - min, max, and null count. An engine filtering on event_date = '2026-06-01' reads the footer, checks each row group's min/max for that column, and skips entire row groups that can't match without decompressing them. This predicate pushdown is most of Parquet's query speed, and it only works if the data is sorted or clustered on the columns you filter - randomly ordered data has wide min/max ranges that prune nothing.

Encoding and compression

Parquet applies lightweight encodings before generic compression: dictionary encoding replaces repeated values with small integer codes, and run-length/bit-packing collapse repetition. On top of that you choose a codec - Snappy (fast, the common default), ZSTD (better ratio at similar speed, increasingly preferred), or gzip (smaller, slower). ZSTD is usually the best modern default for the storage-vs-CPU tradeoff.

df.write \
  .option("compression", "zstd") \
  .parquet("s3://lake/events/")

# pandas / pyarrow
df.to_parquet("events.parquet", compression="zstd")

The small-files problem

Parquet's design assumes reasonably large files. Streaming or frequent micro-batch writes produce thousands of tiny files, and the per-file footer-read and planning overhead then dominates query time - the classic "small files problem." The fix is compaction: periodically rewrite many small files into fewer large ones (table formats automate this with rewrite_data_files). Aim for files in the hundreds-of-MB range and row groups around 128MB.

Practical guidance

• Sort on your filter columns before writing so row-group statistics actually prune.
• Partition by low-cardinality columns you filter on (date, region) - not by high-cardinality ones (user_id), which explodes file counts.
• Don't store one giant column of JSON blobs - you lose columnar pruning; flatten fields into real columns.
• Pair it with a table format (Iceberg/Delta) when you need transactions, deletes, or schema evolution - Parquet alone is just files.

Parquet vs ORC vs Avro

ORC is the other major columnar format, strong in the Hive/Spark ecosystem with similar properties; the choice often follows your engine's native support. Avro is row-based, which makes it a poor fit for analytic scans but a good one for streaming and record-by-record write workloads (Kafka, schema registries). The common pattern is Avro on the wire, Parquet at rest.