PySpark is the Python API for Apache Spark, the distributed engine for data that doesn't fit on one machine. Where pandas holds a dataframe in a single process's RAM, PySpark spreads a logical DataFrame across partitions on a cluster and executes operations in parallel. You write transformations in familiar dataframe-style code; Spark builds a plan, optimizes it through Catalyst, and runs the work on executors. The trade-off is real overhead per job - PySpark earns its keep at the gigabyte-to-terabyte scale, not on a few thousand rows.

Distributed DataFrames and lazy execution

A Spark DataFrame is a partitioned, schema-typed collection. Transformations like select, filter, withColumn, and groupBy are lazy - they describe a computation but don't run it. Nothing executes until an action (write, count, collect, show) forces it. This lets Catalyst rearrange and prune the plan: pushing filters down to the scan, combining projections, and avoiding work whose output is never used.

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("daily_revenue").getOrCreate()

df = spark.read.parquet("s3://bucket/events/")

daily = (
    df.filter(F.col("status") == "completed")
      .groupBy("user_id", F.to_date("ts").alias("day"))
      .agg(F.sum("amount").alias("revenue"),
           F.count("order_id").alias("orders"))
)

daily.write.mode("overwrite").partitionBy("day").parquet("s3://bucket/daily_revenue/")

The code reads like the pandas version, but each step is a node in a plan that runs across the cluster only when write fires.

Partitions, shuffles, and skew

Performance in Spark is mostly a story about data movement. A groupBy or join generally triggers a shuffle - records are redistributed across the network so that matching keys land on the same executor. Shuffles are the expensive part of most jobs. Two problems dominate tuning: too many tiny partitions (scheduling overhead) and skew, where one key holds a disproportionate share of rows and a single task runs long after the rest finish. Spark's adaptive query execution mitigates some of this automatically, but lopsided keys often still need explicit handling.

# broadcast a small dimension to avoid a full shuffle join
from pyspark.sql.functions import broadcast

enriched = events.join(broadcast(dim_country), on="country_code", how="left")

# repartition before a wide aggregation to balance work
events.repartition(200, "user_id").groupBy("user_id").count()

Broadcasting a small table sends a full copy to every executor and skips the shuffle entirely - one of the highest-impact optimizations for star-schema-style joins.

When PySpark is the right tool

Reach for PySpark when a job's input outgrows a single machine, when you need to read and write partitioned data on object storage at scale, or when a step in a batch pipeline must run against a shared cluster. For smaller data the startup cost, serialization between the JVM and Python, and cluster coordination make it slower than pandas. A common pattern is to do large-scale filtering and aggregation in Spark, then pull the reduced result into pandas for the last-mile work or hand it to a model.

Frequently asked questions

How is PySpark different from pandas? pandas runs in one process and holds the whole frame in RAM; PySpark distributes the data across a cluster and executes lazily through a query optimizer. pandas wins on small data and interactivity; PySpark wins when the data exceeds one machine.

What is a shuffle and why does it matter? A shuffle moves records across the network so that rows with the same key end up on the same executor, which is needed for joins and aggregations. It's the most expensive operation in most jobs, so reducing or balancing shuffles is the core of Spark tuning.

Why is collect() risky? collect() pulls every row back to the driver's memory. On a large DataFrame it can exhaust the driver and crash the job. Use write to persist results, or limit/take when you only need a sample.

When should I broadcast a join? When one side is small enough to fit in each executor's memory (typically up to a few hundred MB). Broadcasting sends a full copy everywhere and avoids shuffling the large side, which is far cheaper for joining a big fact table against a small dimension.