Capacity Planning for Data Warehouses and Lakehouses

Cloud platforms were supposed to make capacity planning obsolete - just scale on demand. In practice they moved the problem rather than removing it. You no longer size servers, but you do size warehouses, set autoscaling bounds, choose cluster configurations, and decide what to pay for performance. Get it wrong and you either throttle users or hand the cloud provider a blank check. Capacity planning didn't die in the cloud; it changed from "how much hardware do I buy" into "how do I configure elasticity so it serves my workloads without bankrupting me."

Why it still matters

Elasticity is not a substitute for planning - it's a tool that planning aims. Auto-scaling without sensible bounds doesn't optimize cost; it just bills you for whatever the workload demands, including the runaway query and the idle cluster nobody turned off. The teams who control cloud data spend are the ones who decided, deliberately, how much capacity each workload should get and under what conditions it scales. The platform handles the mechanism; you still own the policy.

Warehouses vs lakehouses

The two architectures expose capacity differently. In a cloud warehouse (Snowflake-style), capacity is the compute cluster - its size and how many run concurrently - decoupled from storage, which scales on its own and cheaply. In a lakehouse (Databricks-style), capacity is cluster configuration: instance types, worker counts, and autoscaling ranges over open storage. The planning questions rhyme - how much compute, for which workloads, scaling how - but the levers differ, so plan to each platform's actual controls.

Start with SLOs, not hardware sizes

The most common mistake is starting from "how big a warehouse should we buy." Start instead from service-level objectives: how fast must this dashboard load, how soon must this pipeline finish, how many concurrent users must this support. Capacity is then whatever meets those targets - derived from requirements, not guessed and tuned by complaint. Without SLOs you have no definition of "enough," so you oscillate between over-provisioning (expensive) and under-provisioning (slow) forever.

Know your workload types

Different workloads drive capacity differently, and lumping them together leads to sizing for the wrong thing:

  • Interactive / BI - many concurrent, latency-sensitive queries; capacity is driven by concurrency, so isolating it (a dedicated warehouse) keeps one heavy user from starving others.
  • Batch ETL - large, scheduled, throughput-oriented jobs; capacity is driven by volume and deadline, and a bigger short-lived cluster can be cheaper than a small slow one.
  • Ad-hoc / data science - spiky and unpredictable; capacity wants elasticity and tight idle-suspension.

Separating these onto right-sized, independently-scaled compute is most of good capacity planning.

A practical approach

For a Snowflake-style warehouse: assign separate virtual warehouses per workload type, size each to its SLO (start conservative, observe, adjust), set aggressive auto-suspend so idle compute stops, and use multi-cluster scaling for concurrency spikes within bounds you set. For a Databricks-style lakehouse: match instance types to the work (memory-heavy vs compute-heavy), set autoscaling min/max from baseline and peak, use job clusters that spin up and tear down for scheduled work, and keep interactive clusters from idling. In both, the pattern is the same: isolate workloads, size to objectives, bound the autoscaling, and reclaim idle capacity.

The takeaway

Capacity planning in the cloud is design, not procurement. Start from SLOs so "enough" has a definition, separate workloads so each gets right-sized compute, set autoscaling bounds so elasticity serves you instead of surprising you, and turn off what's idle. Whether it's a warehouse's virtual clusters or a lakehouse's instance configs, the discipline is identical - decide deliberately how much capacity each workload deserves, and let the platform's elasticity execute that decision rather than make it for you. Stop guessing; start designing.

Comments (1)

  1. binance referral
    Your article helped me a lot, is there any more related content? Thanks!

Leave a Comment