Apache Arrow is a language-independent standard for representing columnar data in memory. Where Parquet defines how columns are laid out on disk, Arrow defines how they're laid out in RAM - and because every Arrow-aware tool agrees on that layout, they can share data without serializing and copying it. Arrow is the quiet plumbing underneath pandas 2.x, Polars, DuckDB, Spark's Python bridge, and many database drivers.

The problem Arrow removes: serialization tax

Historically, moving data between systems meant serializing it on one side and deserializing on the other - Spark to Python, database to client, process to process. For large datasets that copy-and-convert step often costs more than the actual computation. Arrow defines one canonical memory format so that two tools speaking Arrow can hand each other a buffer by pointer - zero-copy. A Spark-to-pandas transfer over Arrow skips the row-by-row pickling that made toPandas() notoriously slow.

Columnar and vectorization-friendly

Arrow stores each column as a contiguous block of memory with a separate validity bitmap for nulls. Contiguous typed memory is exactly what CPUs process fastest: it enables SIMD vectorized operations and cache-friendly scans. This is why engines like DuckDB and Polars, which compute directly on Arrow buffers, get their speed - there's no per-row object overhead.

What Arrow is not

Arrow is an in-memory and IPC format, not a storage format. You don't keep data "in Arrow" on disk for the long term - that's Parquet's job. (Arrow does define a file/IPC format for short-lived interchange and memory-mapping, but for durable analytic storage you write Parquet.) The mental split: Parquet at rest, Arrow in motion and in memory. The two are designed by overlapping communities to convert between each other cheaply.

Where you actually use it

  • Faster Spark ↔ Python - Arrow powers efficient toPandas() and pandas UDFs.
  • Engines - DuckDB and Polars read and return Arrow tables, so you can pass data between them with no copy.
  • Arrow Flight - a gRPC-based protocol for high-throughput data transfer between services, replacing slow ODBC/JDBC paths for bulk movement.
  • Database drivers (ADBC) - return query results as Arrow batches instead of row objects.
import pyarrow as pa, pyarrow.parquet as pq

# Read Parquet into Arrow, hand it to DuckDB and Polars with no copy
table = pq.read_table("events.parquet")          # Arrow table in memory
import duckdb, polars as pl
duckdb.sql("SELECT region, count(*) FROM table GROUP BY 1")  # queries the Arrow buffer
pl.from_arrow(table).group_by("region").len()               # same buffer, no conversion

Why it matters for data engineers

You rarely program Arrow directly, but it explains why modern tools interoperate so cheaply and why the pandas/Polars/DuckDB world feels fast on single-node data. When you choose tools, Arrow-native ones compose without conversion penalties - a pipeline that reads Parquet, transforms in Polars, and queries in DuckDB passes one memory representation throughout instead of paying a serialization tax at each hop.