For years, Uber's data lake ingestion worked the way most companies' ingestion works: batch jobs that ran on a schedule, collected data from various sources, and wrote it to the lake in bulk. The system handled thousands of datasets and petabytes of data. It worked. But "worked" increasingly meant "data analysts waited hours for fresh data" and "compute costs grew faster than data volume."

In early 2026, Uber launched IngestionNext - a streaming-first data lake ingestion platform that replaced the batch-heavy architecture with a system built on Apache Kafka, Apache Flink, and Apache Hudi. The results: data latency dropped from hours to minutes, and compute usage decreased by 25%. Those are not benchmark numbers from a proof of concept. They are production metrics from a platform supporting thousands of datasets at one of the most data-intensive companies in the world.

This article breaks down the architecture of IngestionNext, the specific engineering decisions that make it work, and what data teams at smaller organizations can learn from Uber's approach.

Why Batch Ingestion Hits a Wall

Batch ingestion is simple, reliable, and well-understood. A scheduled job wakes up, reads from a source, transforms the data, and writes it to the lake. For decades, this pattern has been the backbone of data warehousing and analytics.

The problem is not that batch ingestion breaks. It is that it creates latency that compounds through the entire data stack. When raw data lands in the lake once per hour, every downstream transformation, model, and dashboard inherits that latency. A dashboard that queries a model built on three layers of transformed data - each layer running hourly - shows data that is potentially three hours stale.

At Uber's scale, this latency had real business consequences. Pricing models that depend on demand patterns need fresh data to work accurately. Fraud detection systems that flag suspicious transactions need to act within minutes, not hours. Operational dashboards that monitor driver supply and rider demand need real-time visibility to make dispatch decisions.

The batch system also had a cost problem. Each batch job provisioned compute resources, processed all data accumulated since the last run, and then released those resources. This pattern creates spiky resource utilization - idle most of the time, then demanding massive compute for short bursts. The result was either over-provisioning (paying for idle capacity) or contention during peak processing windows.

The IngestionNext Architecture

IngestionNext replaces batch collection with continuous streaming while maintaining the batch system's reliability guarantees. The architecture has three core components: a Kafka-based event transport layer, a Flink-based processing layer, and a Hudi-based storage layer.

Event transport: Kafka

All data enters the system through Apache Kafka topics. Source systems - databases, application services, IoT devices, third-party APIs - publish events to Kafka as they happen. This is a fundamental architectural shift from "pull data on a schedule" to "receive data as it arrives."

Kafka serves two purposes in this architecture. First, it decouples producers from consumers. Source systems write events and move on - they do not need to know or care about downstream processing. Second, it provides a durable, replayable log. If a processing job fails, it can rewind to any point in the Kafka log and reprocess. This replay capability is what makes streaming ingestion as reliable as batch.

# Conceptual: How change data capture feeds Kafka
# (Uber uses internal tooling; this illustrates the pattern)

from confluent_kafka import Producer
import json

class DatabaseCDCProducer:
    """Captures database changes and publishes to Kafka."""

    def __init__(self, kafka_config: dict, topic: str):
        self.producer = Producer(kafka_config)
        self.topic = topic

    def on_change(self, change_event: dict):
        """Called by CDC connector (e.g., Debezium) on each DB change."""
        message = {
            "table": change_event["table"],
            "operation": change_event["op"],  # INSERT, UPDATE, DELETE
            "before": change_event.get("before"),
            "after": change_event.get("after"),
            "timestamp": change_event["ts_ms"],
            "transaction_id": change_event.get("txn_id"),
        }

        self.producer.produce(
            topic=self.topic,
            key=str(change_event["primary_key"]).encode(),
            value=json.dumps(message).encode(),
            callback=self.delivery_report,
        )

    def delivery_report(self, err, msg):
        if err:
            logger.error(f"Delivery failed: {err}")
        else:
            logger.debug(f"Delivered to {msg.topic()} [{msg.partition()}]")

Processing: Apache Flink

Flink reads from Kafka topics and performs the transformations that batch jobs previously handled: schema validation, data type conversions, deduplication, late-arriving data handling, and partitioning. The critical difference is that Flink processes records continuously rather than in scheduled batches.

Uber's Flink jobs use event-time processing with watermarks - a mechanism that handles late-arriving data gracefully. In a ride-sharing platform, events frequently arrive out of order. A driver's GPS signal from 5 minutes ago might arrive after a more recent signal due to network latency. Watermarks tell Flink how long to wait for late data before closing a processing window.

-- Flink SQL: Streaming ingestion with watermark handling
-- (Simplified illustration of the pattern)

CREATE TABLE raw_ride_events (
    ride_id        STRING,
    driver_id      STRING,
    rider_id       STRING,
    event_type     STRING,
    event_time     TIMESTAMP(3),
    latitude       DOUBLE,
    longitude      DOUBLE,
    fare_amount    DECIMAL(10, 2),
    -- Watermark allows 5 minutes for late-arriving events
    WATERMARK FOR event_time AS event_time - INTERVAL '5' MINUTE
) WITH (
    'connector' = 'kafka',
    'topic'     = 'ride-events',
    'properties.bootstrap.servers' = 'kafka:9092',
    'format'    = 'json'
);

-- Continuous aggregation: updates every time a new event arrives
CREATE TABLE ride_metrics_5min AS
SELECT
    window_start,
    window_end,
    city_id,
    COUNT(*)                                     AS total_events,
    COUNT(DISTINCT ride_id)                      AS unique_rides,
    SUM(CASE WHEN event_type = 'completed'
        THEN fare_amount ELSE 0 END)             AS total_fares,
    AVG(CASE WHEN event_type = 'completed'
        THEN fare_amount END)                    AS avg_fare
FROM TABLE(
    TUMBLE(TABLE raw_ride_events, DESCRIPTOR(event_time), INTERVAL '5' MINUTE)
)
GROUP BY window_start, window_end, city_id;

Storage: Apache Hudi

The processed data lands in Apache Hudi tables stored on the data lake (in Uber's case, HDFS and cloud object storage). Hudi was originally created at Uber specifically to solve the problem of efficiently updating data lake tables - a problem that batch ingestion avoids by always writing full partitions but streaming ingestion faces constantly.

Hudi supports two table types that are central to IngestionNext:

Copy-on-Write (CoW) tables for read-heavy workloads. Data is written in optimized columnar format (Parquet). Reads are fast because there is no merge overhead. Writes are slower because every update rewrites the affected file.

Merge-on-Read (MoR) tables for write-heavy workloads. Updates are written to a delta log and merged during reads. Writes are fast because only the delta is written. Reads incur a merge cost but still perform well for most analytics queries.

The choice between CoW and MoR depends on the dataset's access pattern - how frequently it is written versus read, how large the updates are, and how latency-sensitive the readers are. IngestionNext automatically selects the table type based on observed access patterns.

The 25% Compute Reduction

The compute savings come from a fundamental difference in how batch and streaming systems use resources.

A batch job processes all data accumulated since the last run. If the job runs hourly, it processes one hour's worth of data in a burst. This requires provisioning enough compute to handle the peak - the busiest hour's data processed as quickly as possible. During off-peak hours, much of this capacity sits idle.

A streaming system processes data continuously in small increments. The compute resources required at any moment are proportional to the current data rate, not the peak accumulated volume. This smooths out the resource utilization curve dramatically.

At Uber's scale, this difference is significant. Their data volumes vary by 5-10x between peak hours (Friday evening in major cities) and off-peak hours (early morning). A batch system provisioned for peak needs wastes 60-80% of capacity during off-peak. A streaming system with autoscaling adjusts continuously, using resources proportional to actual demand.

The 25% compute reduction is a net figure - it accounts for the overhead of running Kafka and Flink infrastructure continuously. The gross savings on batch compute are larger, but the always-on streaming infrastructure has its own costs.

Handling the Hard Parts

Streaming ingestion is straightforward when everything works. The engineering challenge is handling the cases when things go wrong.

Exactly-once semantics

In a batch system, exactly-once processing is simple: if the job fails, rerun it. Idempotent writes ensure no duplicates. In a streaming system, achieving exactly-once semantics requires coordination between Kafka (source), Flink (processing), and Hudi (storage). IngestionNext uses Flink's checkpoint mechanism combined with Hudi's transaction log to ensure that every record is processed exactly once, even across failures and restarts.

Schema evolution

Source schemas change. Columns get added, types get modified, tables get restructured. In a batch system, these changes are handled during the next run - the batch job reads the new schema and adjusts. In a streaming system, schema changes arrive mid-stream and must be handled without stopping the pipeline.

IngestionNext uses schema registries (compatible with Confluent Schema Registry) to manage schema evolution. Producers register schemas before publishing events. Consumers validate incoming events against the registry. Backward-compatible changes (adding nullable columns, widening types) are handled automatically. Breaking changes trigger alerts and require manual intervention.

Backfill and replay

Streaming systems need to handle historical data reprocessing - whether due to bug fixes, schema changes, or new downstream requirements. IngestionNext maintains Kafka retention long enough to support replays (typically 7-14 days) and can fall back to batch processing from the data lake for longer historical periods. This hybrid capability means the system is streaming-first but not streaming-only.

Lessons for Smaller Data Teams

Uber operates at a scale that most organizations will never reach. But the architectural principles behind IngestionNext apply to data teams of any size.

Start with CDC, not full streaming

The highest-impact, lowest-risk entry point to streaming ingestion is Change Data Capture (CDC). Tools like Debezium can capture database changes and publish them to Kafka with minimal configuration. This gives downstream systems access to near-real-time data without modifying the source applications.

{
  "name": "orders-cdc-connector",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.hostname": "db-primary",
    "database.port": "3306",
    "database.user": "cdc_reader",
    "database.server.id": "1001",
    "database.include.list": "production",
    "table.include.list": "production.orders,production.payments",
    "topic.prefix": "cdc",
    "schema.history.internal.kafka.topic": "schema-changes",
    "transforms": "route",
    "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.route.regex": "cdc\\.production\\.(.*)",
    "transforms.route.replacement": "lake-ingest.$1"
  }
}

Use managed services to reduce operational overhead

Running Kafka, Flink, and Hudi is operationally complex. For teams that do not have Uber's infrastructure engineering capacity, managed services - Confluent Cloud for Kafka, AWS Kinesis Data Analytics or managed Flink for processing, and Delta Lake or Apache Iceberg on managed storage - provide the same architectural benefits with significantly less operational burden.

Keep batch as a fallback

IngestionNext does not eliminate batch processing - it relegates batch to a fallback role for historical reprocessing and disaster recovery. This is the right approach for most organizations. Build the streaming path for latency-sensitive data, keep the batch path for backfills and corrections, and use the same storage layer (Hudi, Delta Lake, or Iceberg) for both.

Measure latency end-to-end

The most common mistake in streaming migrations is measuring only ingestion latency while ignoring the full pipeline. Data landing in the lake in 30 seconds is meaningless if the downstream dbt models still run hourly. End-to-end latency - from source event to dashboard update - is the metric that matters.

What Comes Next

Uber's IngestionNext is part of a broader industry trend toward real-time data infrastructure. Across the data engineering landscape, the distinction between "batch" and "streaming" is dissolving. Modern table formats (Hudi, Delta Lake, Iceberg) handle both modes. Modern processing engines (Flink, Spark Structured Streaming) unify batch and stream processing under a single API. Modern orchestrators are learning to manage both patterns.

The question for most data teams is no longer whether to adopt streaming but when and where. The answer, based on Uber's experience, is to start with the datasets where latency has the most business impact, use CDC as the entry point, and keep batch as a reliable fallback while the streaming infrastructure matures.

Hours to minutes is not just a latency improvement. It changes what questions the business can ask and how quickly it can act on the answers. That is the real value of streaming-first architecture.