Apache Beam is a programming model and SDK for writing data-processing pipelines that run unchanged on multiple execution engines. You write the logic once - in Java, Python, or Go - and pick a runner at deployment time: the Flink runner, the Spark runner, or Google Cloud Dataflow. The promise is portability: the same pipeline that processes a bounded file today can process an unbounded stream tomorrow, and you can move between engines without rewriting business logic.

One model for batch and stream

Beam's central abstraction is the PCollection - a distributed, possibly unbounded dataset - and the PTransform, an operation applied to it. Because a PCollection can be bounded (batch) or unbounded (stream), the same transforms work for both. This is the "Beam" name itself: a contraction of Batch and stream.

with beam.Pipeline() as p:
    (p
     | beam.io.ReadFromText('events.json')
     | beam.Map(parse_event)
     | beam.WindowInto(beam.window.FixedWindows(60))
     | beam.CombinePerKey(sum)
     | beam.io.WriteToText('counts'))

Windowing and the data model

Beam treats event time as a first-class concept. Windows group elements by their event timestamp (fixed, sliding, or session windows), watermarks track how far event time has progressed, and triggers decide when to emit results for a window - including how to handle late-arriving data. This model came out of Google's work on the Dataflow papers and is what lets the same pipeline give correct results whether data arrives in order or hours late.

Runners and portability

The portability layer is real but imperfect. Each runner supports a different subset of Beam features, so a pipeline that relies on advanced state or timers may behave differently on the Flink runner than on Spark. The Direct Runner runs locally for testing. In practice, teams pick Beam when they want to avoid lock-in to a single engine or already run on Dataflow, which is the most complete runner.

Frequently asked questions

What problem does Beam solve? It decouples pipeline logic from the engine that runs it, so the same code runs on Flink, Spark, or Dataflow and handles both batch and streaming data.

Is Beam an execution engine? No. Beam is a model and SDK; it has no engine of its own. A runner (Flink, Spark, Dataflow) does the actual distributed execution.

What are windows, watermarks, and triggers? Windows group events by event time, watermarks estimate event-time progress, and triggers control when each window's results are emitted - together they define how streaming aggregations and late data are handled.