Distributed Data Processing

Distributed data processing is what you reach for when one machine can't hold the data or finish the work in time. The core idea is to split a dataset across many machines, run computation where each slice already lives, and combine the partial results - turning a job that would take days on a single server into one that finishes in minutes across a cluster. The engines in this category differ mostly in how they schedule that work, whether they process data in batches or as a continuous stream, and how much of the hard part (fault tolerance, shuffling, state) they hide from you.

Batch versus stream

The oldest split in the field is between batch and stream. Apache Hadoop and its MapReduce model defined the batch era: read a large bounded dataset from disk, process it in stages, write the result back. Stream processors like Apache Flink, Apache Storm, and Spark Streaming handle unbounded data that never stops arriving, producing results within seconds of an event. Apache Beam tries to erase the distinction with one programming model that runs on either kind of engine.

SQL on big data

Not every job needs hand-written code. Apache Hive put a SQL layer over Hadoop so analysts could query files in HDFS without writing MapReduce. Presto and Trino took that further with a fast, interactive query engine that federates across many sources at once. Apache Pig offered a middle path - a scripting language for data flows that compiled down to MapReduce.

Messaging and specialized engines

Apache Pulsar is a messaging and streaming platform that separates serving from storage, and Apache Samza is a stream processor built tightly around Kafka's log model. For workloads that don't fit a general engine, specialized ones exist: Apache Sedona adds geospatial types and spatial joins on top of Spark.

Choosing one

Large bounded batch jobs over files? Spark (the de-facto successor to Hadoop MapReduce) or Hive for SQL.
Low-latency stream processing with rich state and exactly-once? Apache Flink.
Interactive, federated SQL across many sources? Presto or Trino.
One pipeline that must run on multiple engines? Apache Beam.
Geospatial analytics at scale? Apache Sedona.

The practical decisions are latency (batch minutes vs. stream milliseconds), how much state your logic carries, what delivery guarantee you need, and which engine your team already operates. Most data platforms end up running more than one.

In this section

Apache Beam
A unified programming model and SDK for batch and streaming pipelines that run unchanged on Flink, Spark, or Google Cloud Dataflow, with first-class event-time windowing, watermarks, and triggers.
Apache Flink
A stateful stream processor with exactly-once semantics, event-time windowing with watermarks, and millisecond latency at high throughput - built stream-first, with batch as a bounded stream.
Apache Hadoop
The framework that launched big data: HDFS for replicated distributed storage, MapReduce for parallel processing that moves compute to the data, and YARN for cluster resource management.
Apache Hive
A SQL layer over Hadoop that compiles HiveQL into distributed jobs over HDFS files, built on schema-on-read tables and the Hive Metastore - the de-facto data-lake catalog Spark, Presto/Trino, and Flink all read.
Apache Pig
A Hadoop data-flow platform whose procedural Pig Latin language compiles load/filter/group/join scripts into MapReduce or Tez jobs - a middle ground between raw MapReduce and SQL, now largely superseded by Spark.
Apache Pulsar
A distributed messaging and streaming platform that separates stateless brokers from BookKeeper storage, scaling serving and storage independently and supporting both queue and stream subscriptions with tiered storage and geo-replication.
Apache Samza
A Kafka-centric stream processor that keeps fast local RocksDB state backed by a compacted changelog topic, runs one task per input partition, and runs on YARN or embedded as a library.
Apache Sedona
A cluster-computing system for large-scale geospatial data that extends Spark, Flink, and Snowflake with spatial types, PostGIS-style functions, spatial partitioning, and R-tree/quad-tree indexes for parallel spatial joins.
Apache Spark
Apache Spark is a distributed engine for large-scale data processing - batch, SQL, and streaming - built on in-memory computation and a query optimizer.
Apache Storm
An original distributed real-time computation system that processes unbounded streams one tuple at a time through topologies of spouts and bolts, with at-least-once delivery (exactly-once via Trident).
Presto / Trino
A distributed MPP SQL query engine for interactive analytics that federates across data lakes, warehouses, and databases through pluggable connectors - holding working data in memory, with no storage of its own. Trino is the actively developed fork.
Spark Streaming
Apache Spark's streaming engine - Structured Streaming - which runs DataFrame/SQL queries incrementally over an unbounded table using micro-batches, with event-time watermarks, stateful operators, and checkpoint-based exactly-once.