The Apache Beam Python SDK lets you write a single data-processing pipeline and run it on multiple execution engines - Google Cloud Dataflow, Apache Flink, Apache Spark, or a local direct runner - without rewriting the logic. Beam's premise is one programming model for both batch and streaming: the same transforms process a bounded file and an unbounded Kafka stream, with windowing and triggers handling the difference in how results are grouped and emitted. The Python SDK exposes this model through a pipeline object and a set of composable transforms.

PCollections and transforms

A pipeline is a graph of PCollections (distributed, possibly unbounded datasets) connected by PTransforms. The pipe operator applies transforms in sequence; ParDo with a DoFn is the general-purpose per-element transform, while Map, Filter, and GroupByKey cover common cases.

import apache_beam as beam

with beam.Pipeline() as p:
    (p
     | "Read"   >> beam.io.ReadFromText("gs://bucket/events/*.json")
     | "Parse"  >> beam.Map(json.loads)
     | "Key"    >> beam.Map(lambda e: (e["user_id"], 1))
     | "Count"  >> beam.CombinePerKey(sum)
     | "Write"  >> beam.io.WriteToText("gs://bucket/out/counts"))

The same graph runs on the direct runner locally for tests and on Dataflow in production by changing the pipeline options, not the transforms.

Windowing and the batch/streaming unification

For unbounded data, Beam groups elements into windows - fixed, sliding, or session - and triggers decide when a window's aggregate is emitted, including handling late data via watermarks. This is the part of Beam that justifies its complexity: the same GroupByKey works on streams because windowing scopes the grouping in time.

Concept	Role
PCollection	Distributed dataset, bounded (batch) or unbounded (stream)
PTransform	An operation: Map, Filter, ParDo, GroupByKey, Combine
Runner	Execution engine: Dataflow, Flink, Spark, or DirectRunner
Window	Time-based grouping for unbounded streams

Runners and where it fits

The Python SDK's most-used runner is Dataflow, Google's managed autoscaling service, where Beam is the native programming model. The Flink and Spark runners let you keep Beam code while running on your own cluster. Note the SDK's portability has limits: Python streaming features and connector coverage lag the Java SDK, and cross-language transforms add operational overhead. For pure-Python batch on a single engine, a lighter tool is often enough; Beam earns its keep when you need one model across batch and streaming or portability across runners.

Frequently asked questions

Is Beam an orchestrator like Airflow? No. Beam defines and runs one data-parallel pipeline; Airflow schedules and sequences many jobs. A common pattern is Airflow triggering a Beam/Dataflow job as one task.

Does the Python SDK match the Java SDK? Not fully. Java has broader connector support and more mature streaming. Check that your sources, sinks, and streaming features are available in Python before committing.

What runner should I use locally? The DirectRunner runs the pipeline in-process for development and testing. It is not for production scale - switch to Dataflow, Flink, or Spark for real workloads.

How does Beam handle late data? Through watermarks and triggers. A window can fire on the watermark and again when late elements arrive, with accumulation or discarding modes controlling how the updated result is emitted.