"Sub-second analytics over billions of events" sounds like a problem only the largest companies have. In practice, any product with real user traffic generates enough events that a traditional warehouse starts answering dashboard queries in seconds, then tens of seconds, then "let it run and check back." The architecture that fixes this has settled into a recognizable pattern: a fast columnar engine like ClickHouse for the serving layer, an open lakehouse format like Delta Lake for durable history, and a clear division of labor between them. Here's how the pieces fit and why.

Why traditional setups break down

General-purpose warehouses optimize for flexibility and broad SQL, not for the specific shape of real-time analytics: high-rate event ingestion plus fast aggregations over recent data. Push enough events through one and you hit the same walls - ingestion lags, dashboard queries slow as data grows, and costs climb because you're scanning more than you need. The data lake alone has the opposite problem: cheap and durable, but not built for the sub-second interactive queries a dashboard demands. Neither half does the whole job well.

The two-layer pattern

The effective design splits responsibilities. ClickHouse serves the queries: a column-oriented engine that ingests millions of rows per second and aggregates over them in milliseconds, holding the recent, hot data that dashboards hit constantly. Delta Lake (or another open table format) holds the durable, complete history in cheap object storage, with ACID guarantees and time travel - the system of record. Recent data lives in both; older data ages out of ClickHouse to the lake, queried less often and far more cheaply. You get interactive speed on the data people actually look at and durable, affordable retention on everything else.

Architecture in motion

Events flow from the application into a streaming buffer (commonly Kafka), which feeds two sinks: ClickHouse for immediate serving and the Delta Lake tables for durable storage. Dashboards query ClickHouse for anything recent and interactive; analytical or historical jobs read the lake. The buffer also solves the ingestion problem - it lets ClickHouse receive large batches rather than a firehose of tiny inserts, which is exactly what keeps its storage engine healthy. A retention policy moves data out of ClickHouse as it cools, so the serving layer stays small and fast.

Scaling from millions to billions

The pattern stretches because each layer scales on its own terms. ClickHouse scales out with sharding and replication, and stays fast by leaning on its strengths: partition by time, pre-aggregate hot metrics with materialized views so dashboards read small summaries, and keep the hot window narrow. Delta Lake scales with object storage, which is effectively unbounded and cheap. The key discipline is keeping ClickHouse focused on recent, frequently-queried data - its speed comes partly from not holding everything.

Optimization that matters

• Pre-aggregate with materialized views so common dashboard queries read rolled-up data instead of scanning raw events.
• Partition by time and let queries prune to the window they need.
• Batch ingestion through the buffer to avoid the "too many parts" trap that small, frequent inserts cause.
• Tier storage by temperature - hot in ClickHouse, cold in the lake - so you pay for speed only where it's used.

Cost: where the savings come from

The large savings teams report from this architecture aren't magic - they come from not storing everything in the expensive fast layer. Keeping only the hot window in ClickHouse and the long tail in cheap object storage means you pay for interactive performance on a small fraction of the data and pennies-per-gigabyte retention on the rest. The dramatic percentages you'll see quoted are mostly this tiering, applied at scale.

The takeaway

Sub-second analytics at scale isn't one product; it's a division of labor. Let ClickHouse do what it's great at - fast aggregations over recent, hot data - and let an open lakehouse format hold the durable, complete history cheaply. Connect them with a streaming buffer, batch your ingestion, pre-aggregate the hot metrics, and tier data by temperature. The result is dashboards that stay instant as events grow into the billions, without paying warehouse prices to store every event in the fast lane. The ClickHouse course covers the MergeTree and materialized-view mechanics this relies on.