DuckDB is an analytical database that runs in-process - inside your application, like SQLite, rather than as a separate server you connect to over a network. There's no daemon, no cluster, no configuration; you pip install duckdb or link the library and immediately run fast analytical SQL against local files. It has become the default tool for the surprisingly large class of analytics problems that fit on a single machine, which is most of them.

Why it's fast

DuckDB is built like a modern cloud warehouse shrunk to one process. It stores and processes data in columnar format and uses a vectorized execution engine that processes batches of values at a time, so it saturates CPU caches and SIMD instructions instead of crawling row by row. The result is that aggregations and joins over tens of gigabytes run in seconds on a laptop - the kind of workload that used to require spinning up a Spark cluster or loading a warehouse.

Querying files in place

DuckDB's most-loved feature is that it reads Parquet, CSV, and JSON directly, including over HTTP and S3, without an import step. You point SQL at the files and it pushes down projections and filters so it only reads the columns and row groups it needs:

-- Query Parquet on S3 directly, no loading
SELECT country, SUM(revenue) AS total
FROM 's3://bucket/sales/2026/*.parquet'
WHERE event_date >= '2026-01-01'
GROUP BY country
ORDER BY total DESC;

It also speaks to pandas and Polars data frames in memory with zero copy, so it slots into a Python data workflow as the SQL engine over whatever you already have loaded.

Where it fits

DuckDB shines for local and embedded analytics: ad-hoc analysis of files, the transformation step in a small-to-mid pipeline, powering analytics features inside an application, and as a fast local development engine for logic you'll later run on a bigger warehouse. Its sweet spot is data that fits on one machine - which, with modern hardware, reaches into the hundreds of gigabytes. It's not built for high-concurrency multi-user serving or for datasets that genuinely require a distributed cluster; that's where a cloud data warehouse takes over. For the very common case where someone reaches for Spark to process 20GB, DuckDB usually does it faster, cheaper, and with far less machinery.

Frequently asked questions

Is DuckDB like SQLite? Architecturally yes - both are in-process, serverless, single-file-friendly databases. The difference is purpose: SQLite is row-oriented for transactional workloads, DuckDB is columnar and vectorized for analytics.

Can DuckDB query Parquet and CSV without importing? Yes - it reads Parquet, CSV, and JSON directly from local disk, HTTP, or S3, pushing down filters so it only reads what's needed.

How big can the data be? DuckDB targets data that fits on a single machine, which modern hardware pushes into the hundreds of gigabytes; beyond that, a distributed warehouse is the better fit.

Does it replace Spark? For single-node workloads it often does, faster and with less overhead. For genuinely distributed datasets and clusters, Spark or a cloud warehouse remains the right tool.