A data pipeline architecture is the shape of how data moves from source to warehouse: how often it runs, how it handles late or corrected data, and how the stages connect. The choice is driven by one question - how fresh does the data need to be, and what are you willing to pay in complexity to get there? Three processing modes and two reference architectures cover most real systems.

Batch, streaming, micro-batch

The processing mode sets the latency floor and the operational cost.

Mode	Latency	Fits
Batch	minutes to hours	Daily reporting, backfills, large reprocessing
Micro-batch	seconds to minutes	Near-real-time dashboards, Spark Structured Streaming
Streaming	sub-second to seconds	Fraud detection, alerting, live counters

Batch is the cheapest to operate and the easiest to reason about - a run either finishes or it doesn't, and you can rerun it. Streaming buys freshness at the cost of always-on infrastructure, harder failure recovery, and the need for windowing and watermarks to handle late events. Micro-batch is the middle ground: it processes small slices on a tight schedule, keeping batch's replayability while cutting latency to seconds.

Lambda vs. Kappa

Lambda runs two paths in parallel: a batch layer that recomputes accurate results over all history, and a speed layer that serves low-latency approximate results until the batch catches up. The serving layer merges both. It's robust but you maintain the same logic twice, in two codebases that drift apart.

Kappa drops the batch layer. Everything is a stream; reprocessing means replaying the log from the start through the same streaming code. One codebase, but it demands a durable, replayable log (Kafka with long retention) and streaming jobs that can reprocess history at speed. Kappa pairs naturally with log-based change data capture, since CDC already turns source state into a replayable event stream.

Orchestrated DAG pipelines

Most batch and micro-batch systems are expressed as a directed acyclic graph (DAG) of tasks in an orchestrator like Airflow, Dagster, or Prefect. The DAG encodes dependencies - extract before transform, load before tests - so the scheduler runs tasks in order, parallelizes independent branches, and retries failures.

extract_orders >> stage_orders >> [run_dbt_models, dq_checks] >> publish

The orchestrator owns scheduling, retries with backoff, and alerting; the failure semantics it gives you are the subject of error handling. Streaming pipelines are usually long-running jobs rather than DAGs, but a DAG often orchestrates their deploy, monitoring, and backfill.

Frequently asked questions

When should I choose batch over streaming? When the business consumes the data on a daily or hourly cadence anyway. Streaming adds always-on cost and recovery complexity that only pays off when seconds of freshness change a decision.

What's the practical difference between Lambda and Kappa? Lambda maintains two code paths (batch + speed); Kappa maintains one streaming path and reprocesses by replaying the log. Kappa is simpler to keep correct but needs a durable, replayable event log.

Is micro-batch the same as streaming? No. Micro-batch processes data in small fixed intervals and keeps batch-style replayability; true streaming processes each event as it arrives, with windowing and watermarks for late data.

Why model pipelines as DAGs? The DAG makes task dependencies explicit, which lets the orchestrator parallelize, retry, and resume from the point of failure instead of rerunning everything.