Apache Flink is a stream processor built for stateful computation over unbounded data with low latency and strong correctness guarantees. Where earlier systems bolted streaming onto a batch engine, Flink treats the stream as the primitive and batch as a special case of a bounded stream. It is the engine teams reach for when they need real exactly-once state, event-time correctness, and millisecond-scale latency at high throughput.

Stateful processing and checkpoints

Flink's defining feature is managed, fault-tolerant state. Operators can keep keyed state (counters, windows, machine-learning features) that survives failures because Flink periodically takes a consistent checkpoint of all operator state using a distributed snapshot algorithm. On failure it restores from the last checkpoint and replays from the corresponding source offsets, which is how it delivers exactly-once state semantics. State can live on the heap or in RocksDB on local disk for state too large for memory.

Event time and watermarks

Flink processes data by event time - when an event actually happened - not when it arrived. Watermarks flow through the pipeline marking event-time progress, so windowed aggregations produce correct results even when events arrive out of order or late. This separation of event time from processing time is what makes Flink results stable and reproducible regardless of network delays.

stream
  .keyBy(e -> e.userId)
  .window(TumblingEventTimeWindows.of(Time.minutes(5)))
  .aggregate(new CountAggregate());

APIs and where it fits

Flink offers layered APIs: the low-level DataStream API for full control, the Table API and Flink SQL for declarative queries over streams, and the ProcessFunction for custom timers and state. It competes most directly with Spark Streaming; the rough rule is that Flink leads on low-latency, per-event stateful streaming, while Spark leads when the same engine also runs heavy batch and ML. It pairs naturally with Kafka as a source and sink.

Frequently asked questions

How does Flink achieve exactly-once? Distributed checkpoints snapshot all operator state consistently; on failure it restores state and rewinds sources to the matching offsets, so each event affects state exactly once.

Flink or Spark Streaming? Flink is a true per-event streaming engine with lower latency and richer state; Spark Streaming uses micro-batches and shines when you also need large-scale batch and ML in one engine.

Where does large state live? In the RocksDB state backend on local disk, checkpointed to durable storage, so state can exceed available memory.