Data formats are the most consequential topic data engineers think about least. The choice between CSV, Parquet, Arrow, and the rest quietly decides query speed, storage cost, and how cleanly tools interoperate - and most teams inherit it by accident. Two formats anchor the modern stack: Parquet on disk and Arrow in memory. Understanding what each is for, and how they fit together, explains most of why the analytics ecosystem works the way it does.

Parquet: columnar on disk

A Parquet file stores data by column, not by row. That single decision drives its advantages. A query reading three columns of a hundred-column table touches only those three. Values within a column are similar, so they compress far better than mixed-type rows. And Parquet keeps statistics (min/max, counts) per chunk, so a query engine can skip whole row groups that can't match a filter without reading them. Add partitioning by common filter columns and you get the format that underpins nearly every cloud warehouse and lakehouse. Parquet is what you write when data lands and rests.

Arrow: columnar in memory

Apache Arrow is a standard for how columnar data lives in memory. Its value is the shared standard part: when pandas, Polars, DuckDB, Spark, and a database all agree on the same in-memory layout, they can pass data between each other with zero copy - no serializing and re-parsing at every boundary. Before Arrow, every tool had its own internal format, and moving data between them meant constant, costly conversion. Arrow makes the in-memory representation a common currency.

Arrow Flight: moving it fast

Arrow Flight extends that idea to the network: a transport for shipping Arrow data between systems without flattening it into rows and rebuilding it on the other side. For large analytical result sets, keeping data in Arrow from server to client removes the serialization tax that traditional database drivers can't avoid - the same principle as in-memory zero-copy, applied to the wire.

How Parquet and Arrow work together

They're complements, not competitors. Parquet is the durable, compressed, on-disk format; Arrow is the fast, uncompressed, in-memory format. The natural flow is: data rests as Parquet in object storage, gets read into Arrow for processing, moves between tools as Arrow (in memory or over Flight), and is written back to Parquet to persist. Because both are columnar, the translation between them is efficient. Most modern data tools read Parquet into Arrow under the hood whether you notice or not.

Parquet vs ORC vs Avro

• Parquet - columnar, analytics-optimized, the broad default for warehouses and lakehouses.
• ORC - also columnar, strong in the Hive/Hadoop world; similar goals, narrower ecosystem today.
• Avro - row-oriented, built for streaming and record-by-record writes with rich schema evolution; the right choice for Kafka messages and write-heavy ingestion, not for analytical scans.

The rule of thumb: Avro for moving and writing records, Parquet for storing and scanning them analytically.

What's coming next

Parquet is dominant but aging in places, and a wave of newer formats targets its weak spots - better support for wide tables, machine-learning feature data, random access, and modern hardware. Names like Lance (ML and vector-friendly, fast random access), Nimble, and Vortex are early but worth watching. None displaces Parquet today; they signal where the pressure is - workloads Parquet wasn't designed for.

Practical recommendations

• Store analytical data as Parquet, partitioned by your common filters. It's the safe, fast, universally supported default.
• Let Arrow do the in-memory work - prefer tools (Polars, DuckDB, pyarrow) that keep data in Arrow and avoid needless conversion to pandas.
• Use Avro for streaming and write-heavy ingestion, then land it as Parquet for analytics.
• Watch the emerging formats for ML and vector workloads, but don't migrate off Parquet without a concrete reason.

Formats are infrastructure: invisible when right, expensive when wrong. Parquet at rest, Arrow in motion covers the overwhelming majority of analytics work - and knowing why is what lets you make the call deliberately instead of inheriting it.