Dask is a parallel computing library for Python that scales the familiar pandas, NumPy, and scikit-learn APIs across multiple cores or a cluster, and out to datasets larger than memory. It works by breaking a big array or dataframe into many smaller partitions, building a task graph of the operations you request, and executing that graph lazily on a scheduler. For a data engineer, that means a job written almost like pandas can process a few hundred gigabytes on one machine, or terabytes on a cluster, without rewriting the logic in a different framework.

Dask DataFrame and partitioning

A Dask DataFrame is a collection of pandas DataFrames split along the index into partitions. Operations are lazy: nothing runs until you call .compute() (which returns an in-memory pandas result) or write to disk. The partition count and the choice of index drive performance - a sorted, well-chosen index lets Dask do partition-aligned joins and groupbys without shuffling every row across the network.

import dask.dataframe as dd

# Read many parquet files lazily; each file becomes one or more partitions
df = dd.read_parquet("s3://bucket/events/*.parquet")

# pandas-style API, but nothing executes yet
daily = (
    df[df.amount > 0]
    .groupby("event_date")["amount"]
    .sum()
)

# Trigger the computation, return a pandas Series
result = daily.compute()

Larger-than-memory arrays

Dask Array mirrors NumPy for n-dimensional data that does not fit in RAM. You chop the array into a grid of NumPy-backed chunks and Dask streams them through the computation, holding only a few chunks in memory at once. The chunk shape is the main tuning knob: chunks that are too small flood the scheduler with tasks, while chunks that are too large defeat the point and blow past memory.

import dask.array as da

# A 100k x 100k array would be ~80 GB in float64; chunk it
x = da.random.random((100_000, 100_000), chunks=(10_000, 10_000))

# Reductions stream chunk by chunk instead of materializing the whole array
mean = x.mean().compute()

Schedulers and the cluster

Dask has a single-machine threaded and process scheduler for local work, and a distributed scheduler (dask.distributed) for clusters. The distributed client also powers the dashboard, which shows the task stream, memory per worker, and where time is spent - usually the fastest way to find a partition skew or a runaway shuffle. The same code runs against either scheduler; you change the deployment, not the job.

from dask.distributed import Client

# Connect to a scheduler (local cluster here; same API for a remote one)
client = Client(n_workers=4, threads_per_worker=2, memory_limit="4GB")

# Subsequent .compute() calls run on this cluster
# Dashboard at client.dashboard_link for live task and memory views

Frequently asked questions

When should I use Dask instead of pandas? When the dataset no longer fits comfortably in memory, or when a single-core pandas job is too slow and the work parallelizes cleanly across partitions. If everything fits in RAM and runs fast enough, plain pandas is simpler and usually faster because it avoids scheduling overhead.

How does Dask compare to PySpark? Dask is Python-native, keeps the pandas and NumPy APIs, and is lighter to set up - good when your team already lives in the Python data stack. PySpark runs on the JVM, has a more mature ecosystem for lake-scale batch jobs and Spark SQL, and is often the existing standard in big-data shops. For multi-terabyte jobs on shared infrastructure, Spark is frequently the safer default; for medium data and array-heavy science workloads, Dask fits better.

Why is my Dask job slower than expected? The usual culprits are too many tiny partitions (scheduler overhead), partitions that are too large (memory spilling to disk), or operations that force a full shuffle, such as a groupby or join on an unsorted column. Check the dashboard's task stream and tune partition size; aim for partitions in the low hundreds of megabytes.

Do I need a cluster to use Dask? No. The default schedulers run on a single machine using threads or processes, which is enough to parallelize across cores and handle larger-than-memory data backed by disk. You only add dask.distributed and worker nodes when one machine is no longer enough.