Real-time data warehousing replaces the nightly batch load with continuous, incremental ingestion so analytical data is fresh within seconds to minutes of being produced. The classic warehouse answered "what happened yesterday"; a real-time warehouse answers "what's happening now" - live operational dashboards, fraud scoring on in-flight transactions, inventory that reflects the last sale rather than last night's snapshot.

Streaming ingestion instead of batch

The defining change is the pipeline. Instead of extracting a day's data and loading it in one job, events flow through a streaming backbone - typically Apache Kafka - and are continuously applied to warehouse tables. Source databases are often captured with change data capture (Debezium reading the database log), so inserts, updates, and deletes propagate as they happen rather than being re-scanned in bulk. A stream processor (Flink, Kafka Streams, Spark Structured Streaming) can enrich, join, and aggregate the events in flight before they land.

Engines built for fresh data

Serving low-latency analytics on constantly-arriving data is its own problem, and a class of engines targets it: ClickHouse, Apache Pinot, Apache Druid, and streaming databases like Materialize and RisingWave that keep materialized views incrementally up to date. These ingest high-volume streams and answer sub-second aggregation queries on data that's seconds old - something a batch-tuned warehouse, optimized for large periodic loads, isn't built to do.

What it costs

Real-time is more expensive and more complex than batch, and not every workload needs it. Streaming pipelines run continuously, so there's no quiet overnight window to absorb failures; late and out-of-order events, exactly-once delivery, and reprocessing all become live concerns. The pragmatic pattern is to apply real-time only where data freshness has clear value and keep batch for the rest - many platforms run both, with streaming feeding the hot serving layer and batch handling heavy historical transforms. It draws directly on polyglot data processing, since batch and streaming engines coexist over the same tables.

Frequently asked questions

What counts as "real-time" here? Data available for analytics seconds to a few minutes after it's produced, via continuous ingestion - as opposed to hours-old data from a nightly batch.

What technologies enable it? A streaming backbone (Kafka), change data capture (Debezium) for database sources, stream processors (Flink, Spark Structured Streaming), and low-latency serving engines (ClickHouse, Pinot, Druid) or streaming databases (Materialize, RisingWave).

Does real-time replace batch? Rarely entirely. Batch remains cheaper and simpler for heavy historical processing; most platforms run real-time for the data that needs freshness and batch for everything else.

What's the hardest part? Handling late and out-of-order events, guaranteeing delivery semantics, and reprocessing - all without an overnight maintenance window, since the pipeline never stops.