Apache Flink – Data/ML Engineer Blog

Apache Flink for Data Engineers: Real-Time Streams Without the Hand-Waving

If your dashboards are always “five minutes behind,” batch jobs keep stepping on each other, or your “real-time” pipeline is just micro-batching in disguise, you need Flink. Apache Flink gives you true event-time processing, large managed state, and exactly-once guarantees—so fraud rules trigger now, not next hour.

Why Flink matters (and when it doesn’t)

Flink shines when you need:

Low latency (10s of ms to seconds), not minute-level batches.
Event-time correctness with late data handling.
Huge stateful ops (joins, aggregations, CEP) at scale.
End-to-end exactly-once with Kafka, object stores, and lakehouse tables.

Don’t pick Flink if all you need is a nightly rollup, or if your team can’t support always-on streaming ops yet.

Core concepts (architectural map)

Event time, watermarks, and windows

Event time = when an event actually happened.
Watermarks model “how far along” you are in event time; control lateness.
Windows (tumbling, sliding, session) group events by event time.

State and checkpoints

Flink keeps operator state (e.g., keyed aggregates) in a state backend (RocksDB or heap).
Checkpoints (frequent, incremental) enable fault recovery.
Savepoints (manual) let you upgrade jobs without data loss.

Exactly-once

Implemented via checkpoint barriers and two-phase commits to sinks.
Works with Kafka, many JDBC sinks, and lakehouse formats via connectors (Iceberg/Hudi/Delta).

APIs

SQL/Table API for declarative analytics and CDC joins.
DataStream API for full control of state, timers, and custom operators.
PyFlink exposes SQL, Table, and (increasingly) DataStream for Python users.

Deployment

Standalone/Kubernetes/YARN, or managed (e.g., Amazon Kinesis Data Analytics for Apache Flink, Ververica Platform).
Scale via parallelism; rescale with savepoints.

Quick start: “from Kafka to Iceberg with exactly-once” (SQL)

-- 1) Source: Kafka orders
CREATE TABLE orders_raw (
  order_id STRING,
  user_id  STRING,
  amount   DECIMAL(10,2),
  ts       TIMESTAMP(3),
  WATERMARK FOR ts AS ts - INTERVAL '10' SECOND
) WITH (
  'connector' = 'kafka',
  'topic' = 'orders',
  'properties.bootstrap.servers' = 'kafka:9092',
  'value.format' = 'json',
  'scan.startup.mode' = 'latest-offset'
);

-- 2) Sink: Iceberg table (append or upsert w/ primary key)
CREATE TABLE dim_daily_revenue (
  day DATE,
  revenue DECIMAL(18,2)
) PARTITIONED BY (day)
WITH (
  'connector' = 'iceberg',
  'catalog-name' = 'hadoop_prod',
  'catalog-type' = 'hadoop',
  'warehouse' = 's3://lake/warehouse/'
);

-- 3) Transform: event-time windowed aggregation
INSERT INTO dim_daily_revenue
SELECT
  CAST(TUMBLE_START(ts, INTERVAL '1' DAY) AS DATE) AS day,
  SUM(amount) AS revenue
FROM orders_raw
GROUP BY TUMBLE(ts, INTERVAL '1' DAY);

Watermark tolerates 10s of out-of-order events.
Iceberg keeps snapshots and ACID guarantees for downstream analytics.

DataStream example: keyed state + timers (Java)

DataStream<Event> events = env
  .fromSource(kafkaSource, WatermarkStrategy
    .<Event>forBoundedOutOfOrderness(Duration.ofSeconds(10))
    .withTimestampAssigner((e, t) -> e.getTs()), "orders");

events
  .keyBy(Event::getUserId)
  .process(new KeyedProcessFunction<String, Event, Alert>() {
    private ValueState<BigDecimal> running;
    private ValueState<Long> timer;

    public void open(Configuration c) {
      running = getRuntimeContext().getState(
        new ValueStateDescriptor<>("sum", BigDecimal.class));
      timer = getRuntimeContext().getState(
        new ValueStateDescriptor<>("timer", Long.class));
    }

    public void processElement(Event e, Context ctx, Collector<Alert> out) throws Exception {
      BigDecimal sum = Optional.ofNullable(running.value()).orElse(BigDecimal.ZERO).add(e.getAmount());
      running.update(sum);
      if (timer.value() == null) {
        long fireAt = ctx.timerService().currentProcessingTime() + 60_000;
        ctx.timerService().registerProcessingTimeTimer(fireAt);
        timer.update(fireAt);
      }
      if (sum.compareTo(new BigDecimal("5000")) > 0) {
        out.collect(new Alert(e.getUserId(), "Spending spike"));
      }
    }

    public void onTimer(long ts, OnTimerContext ctx, Collector<Alert> out) throws Exception {
      running.clear(); timer.clear(); // reset every minute
    }
  })
  .sinkTo(alertsSink);

Shows keyed state, processing-time timers, and a rolling threshold alert.

PyFlink snippet: simple SQL job

from pyflink.table import EnvironmentSettings, TableEnvironment

t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
t_env.execute_sql("""CREATE TEMPORARY VIEW metrics AS
  SELECT window_start, window_end, COUNT(*) AS cnt
  FROM TABLE(TUMBLE(TABLE source_table, DESCRIPTOR(ts), INTERVAL '1' MINUTE))
  GROUP BY window_start, window_end
""")

Use PyFlink when your team is Python-heavy but still wants Flink SQL/Table power.

Flink vs the usual suspects

Capability	Flink	Spark Structured Streaming	Kafka Streams
Latency	ms–seconds	seconds–minutes (micro-batch by default)	ms–seconds
State size	Very large (RocksDB)	Good (state store), but micro-batch overhead	Moderate (RocksDB)
Event-time / watermarks	First-class	Good	Basic
APIs	SQL/Table, DataStream, CEP	SQL/DataFrame	Java/Scala DSL
Exactly-once end-to-end	Yes (2PC)	Yes (to many sinks)	Yes (within Kafka)
Connectors	Broad (Kafka, JDBC, S3, Iceberg, Hudi, Delta, Pulsar, etc.)	Broad	Kafka-centric
Ops model	Always-on jobs	Micro-batch or continuous	Embedded apps

If you need rich event-time semantics + huge state + many sinks, Flink is the pragmatic choice.

Best practices (that save you outages)

Design around access patterns
- Model streams and keys according to the joins/aggregations you must run.
- Avoid “catch-all” topics that force expensive shuffles.
Pick the right watermark strategy
- Start with bounded out-of-orderness (e.g., 10–120s).
- Measure late event rate; tune allowed lateness and side outputs.
Manage state deliberately
- Default to RocksDB for large state; heap state for low latency + small state.
- Size pod memory & disk IOPS for RocksDB; watch compaction metrics.
Checkpoints, not afterthoughts
- Interval: 10–120s; exactly-once sinks need stable checkpointing.
- Use externalized checkpoints and practice recovery.
Backpressure visibility
- Enable backpressure sampling; widen bottleneck operators (parallelism) first.
- Watch busy time, mailbox throughput, GC, and network buffers.
Schema & CDC discipline
- Use Debezium/Flink-CDC for source-of-truth changes.
- Enforce schema evolution rules; validate in a staging job.
Safe upgrades with savepoints
- Take a savepoint, deploy new code, resume from savepoint.
- Keep operator UIDs stable to preserve state mapping.
Partition wisely
- Avoid hot keys (e.g., celebrity user). Salt keys or use load-aware partitioning.
- Align Kafka partitions with Flink parallelism to minimize skew.
Test with deterministic replays
- Keep a small Kafka topic (or files) as a replayable fixture for CI.
- Validate watermarking, late data behavior, and exactly-once idempotency.
Choose the simplest API

Prefer Flink SQL/Table for analytics and joins.
Drop to DataStream only when you need custom state/timers/CEP.

Common pitfalls (learned the hard way)

Treating processing time as event time. Your alerts will be wrong during spikes.
Over-wide windows. Massive state + slow checkpoints = timeouts.
Ignoring RocksDB I/O. Under-provisioned disks → backpressure chain reactions.
Unstable operator UIDs. You’ll “lose” state on redeploy.
Non-idempotent sinks. Exactly-once in Flink won’t save a side-effect-only HTTP sink.

Real-world pattern: dedup + sessionization → Iceberg

Use-case: Web events arrive out of order; you need per-user sessions and deduped pageviews.

Plan:

Kafka → Flink SQL
Dedup by (user_id, page_id, ts) using primary key table or LAST_VALUE over window.
Session windows with SESSION(ts, INTERVAL '30' MINUTES).
Write to Iceberg (partition by date(ts)), then query with Trino/Spark for BI.

Why Flink: Event-time sessions with late data + exactly-once into lakehouse.

Internal link ideas (add these on your site)

“Kafka Exactly-Once vs At-Least-Once: What Actually Breaks”
“Designing Watermarks: Tuning Late Data Without Dropping Revenue”
“Iceberg vs Delta vs Hudi: Choosing Your Lakehouse Table Format”
“CDC with Debezium + Flink: Blueprints and Failure Modes”
“Streaming Joins 101: When to Pre-Aggregate vs Full State Joins”

Conclusion & takeaways

Flink = real real-time: event time, watermarks, and large state with exactly-once.
Start with SQL/Table; reach for DataStream when you need precise state/timers.
Engineer for state + I/O: RocksDB tuning, checkpoint stability, and backpressure are table stakes.
Upgrade safely with savepoints and stable operator UIDs.
If you only need hourly aggregates, don’t over-engineer—batch is fine.

Call to action: Want a production-ready Flink template (Kafka → Flink SQL → Iceberg) with CI replay tests and Terraform/K8s manifests? Say “Flink starter” and I’ll drop it in.

Image prompt

“A clean, modern data architecture diagram illustrating an Apache Flink streaming job reading from Kafka, performing event-time window aggregations with watermarks, and writing exactly-once to an Iceberg table in a data lake — minimalistic, high contrast, 3D isometric style.”

Data/ML Engineer Blog