pandas is the in-memory dataframe library that most Python data work passes through at some point - exploratory analysis, feature engineering, cleaning a few million rows before they go into a warehouse or a model. Its strength is expressiveness: labeled columns, a rich indexing model, and a deep set of vectorized operations over NumPy arrays. Its hard limit is that everything lives in a single process's RAM, which sets the boundary between "use pandas" and "reach for something distributed."

The dataframe as a working set

A pandas DataFrame holds columns as typed NumPy arrays (or, increasingly, Arrow-backed arrays). Operations are vectorized - you express a transform over a whole column rather than looping row by row, and the work runs in compiled C. The common engineering shape is read, reshape, aggregate, write: pull a slice of data, group it, compute summaries, and hand the result to the next stage.

import pandas as pd

df = pd.read_parquet("events.parquet")

# revenue per user per day, only completed orders
daily = (
    df[df["status"] == "completed"]
      .groupby(["user_id", df["ts"].dt.date])
      .agg(revenue=("amount", "sum"), orders=("order_id", "count"))
      .reset_index()
)
daily.to_parquet("daily_revenue.parquet", index=False)

The groupby().agg() pattern with named aggregations is the workhorse for rollups. Chaining keeps intermediate frames out of named variables, which makes the pipeline read top to bottom.

Memory is the constraint

pandas loads the entire frame into RAM, and the in-memory footprint is usually several times the on-disk size - object-dtype string columns and the default int64/float64 widths are the main culprits. A practical rule of thumb is that a frame wants 5-10x its CSV size in memory. Three levers push the ceiling higher before you have to leave pandas: downcast numeric columns, use category dtype for low-cardinality strings, and read in chunks when you only need to aggregate.

# downcast and categorize to cut memory
df["count"] = pd.to_numeric(df["count"], downcast="integer")
df["country"] = df["country"].astype("category")

# stream a large CSV instead of loading it whole
totals = pd.Series(dtype="float64")
for chunk in pd.read_csv("big.csv", chunksize=500_000, usecols=["country", "amount"]):
    totals = totals.add(chunk.groupby("country")["amount"].sum(), fill_value=0)

When the working set genuinely exceeds one machine's memory, that's the signal to move the heavy step to PySpark or push the aggregation down into the database with SQLAlchemy rather than fighting pandas with ever-larger boxes.

Where pandas fits in a pipeline

In an ETL or feature pipeline, pandas tends to sit at the edges and the middle-small steps: parsing semi-structured input, joining a few reference tables, computing features for a training set that fits in memory, and validating data before load. For repeated production jobs, watch two things - pin dtypes on read (don't let the parser guess differently across runs) and prefer Parquet over CSV so types and column selection survive the round trip. Copy-on-write semantics (default since pandas 3.0) removed most of the old SettingWithCopyWarning confusion by making chained-assignment behavior predictable.

Frequently asked questions

When should I stop using pandas and switch to Spark? When the data no longer fits in one machine's RAM with headroom for intermediates, or when a single job's runtime makes a single-process tool impractical. Below roughly tens of millions of rows on a normal box, pandas is usually faster than the overhead of a cluster.

Why does my DataFrame use far more memory than the file on disk? Object-dtype strings store Python objects with per-value overhead, and numeric columns default to 64-bit. Convert low-cardinality strings to category and downcast numerics; Arrow-backed string dtype also helps substantially.

How do I process a file too large to load at once? Read it in chunks with chunksize and aggregate incrementally, select only the columns you need with usecols, or convert to Parquet and read column subsets. If you still can't aggregate within memory, move the step to a distributed engine or the database.

Is iterating over rows with iterrows acceptable? Almost never for data of any size - it's orders of magnitude slower than vectorized column operations. Express the logic as operations over whole columns, or use groupby/merge/map instead.