Apache Samza is a stream-processing framework built tightly around the log. It came out of LinkedIn alongside Kafka and shares Kafka's worldview: a stream is a partitioned, replayable log, and processing is a job that consumes from input logs and produces to output logs. Samza's design choices - local state, partition-aligned parallelism, and pluggable execution - reflect that origin and make it a natural fit for Kafka-centric pipelines.

Local state and the changelog

Samza keeps task state in a local key-value store (RocksDB) on the same machine as the processing task, which makes state access fast because it avoids a network hop to a remote database. Durability comes from a changelog: every state update is also written to a compacted Kafka topic, so if a task fails and restarts elsewhere, it rebuilds its local store by replaying the changelog. This pattern - fast local state backed by a replayable log - is now common across stream processors.

Partition-aligned parallelism

A Samza job runs one task per input partition, and each task processes its partition independently. Parallelism therefore tracks the number of partitions in the input stream, and ordering is preserved within a partition. This direct mapping keeps the model simple and predictable, at the cost of tying your maximum parallelism to how the input is partitioned.

Execution and where it fits

Samza can run on YARN or as an embedded library (Samza standalone) inside your own application, and it pairs almost exclusively with Kafka as its messaging layer. Its closest conceptual relative is Kafka Streams, which embeds the same log-and-local-state ideas as a library. For teams not already centered on Kafka, more general engines like Apache Flink are usually the broader choice; Samza shines where Kafka is already the backbone.

Frequently asked questions

How does Samza make state durable? State lives in a local RocksDB store for speed, and every update is mirrored to a compacted Kafka changelog topic so a restarted task can rebuild its state by replaying it.

What sets Samza's parallelism? One task per input partition - parallelism scales with partition count and ordering holds within each partition.

When would I pick Samza? When your pipeline is built on Kafka and you want local-state stream processing tightly aligned with the log; otherwise a general engine like Flink covers more ground.