Spark Streaming is Apache Spark's answer to processing unbounded data. The current API, Structured Streaming, lets you write a streaming query with the same DataFrame and SQL code you would use for a batch table - Spark runs it incrementally as new data arrives. The appeal is that one engine and one programming model cover both batch and streaming, so a team already running Spark for ETL and ML can add streaming without adopting a separate system.

Micro-batch and the unbounded table

Structured Streaming models a stream as an unbounded table that grows over time; your query runs continuously and updates its result as rows append. Under the hood the default execution is micro-batch: Spark collects data for a short interval and processes it as a small batch job. This gives latency in the hundreds-of-milliseconds-to-seconds range - higher than a true per-event engine, but with Spark's throughput and exactly-once guarantees. A continuous-processing mode exists for lower latency at some feature cost.

(spark.readStream.format("kafka")
  .option("subscribe", "events").load()
  .groupBy(window(col("ts"), "5 minutes"), col("action"))
  .count()
  .writeStream.format("delta")
  .option("checkpointLocation", "/chk").start())

State, watermarks, and exactly-once

Structured Streaming supports event-time windowing with watermarks to bound how long it waits for late data, and stateful operations like aggregations and stream-stream joins. Fault tolerance comes from checkpointing offsets and state to durable storage plus idempotent sinks, giving end-to-end exactly-once when the source is replayable (like Kafka) and the sink supports it.

Where it fits

Spark Streaming is the natural choice when streaming sits alongside large batch jobs and ML in the same Spark platform, especially on Databricks with Delta Lake as the sink. When the priority is the lowest possible per-event latency and the richest streaming state model, Apache Flink is usually the stronger fit. Both read from and write to Kafka.

Frequently asked questions

Is Spark Streaming real-time per event? The default is micro-batch - short batches every interval - giving sub-second to few-second latency; a continuous mode trades features for lower latency. Flink is the more per-event engine.

What is the unbounded table model? Structured Streaming treats a stream as a table that keeps appending rows, so the same DataFrame/SQL code used for batch produces a continuously updated streaming result.

How does it guarantee exactly-once? By checkpointing source offsets and operator state to durable storage and using idempotent or transactional sinks with a replayable source like Kafka.