Choosing Between DUPLICATE, UNIQUE, and AGGREGATE in VeloDB

When you create a table in VeloDB (Apache Doris), one early decision quietly shapes how the table behaves for its entire life: the data model. Doris offers three - DUPLICATE, UNIQUE, and AGGREGATE - and they're not interchangeable. Choose wrong and you get duplicate rows you didn't want, updates that don't apply, or pre-aggregation that destroys the detail you needed. Choose right and the table does exactly what the workload requires with no fighting. Here's how to pick deliberately instead of by default.

The three models at a glance

  • DUPLICATE - stores every row exactly as inserted, keeping all duplicates. It's the raw, append-only model: nothing is merged or deduplicated.
  • UNIQUE - enforces uniqueness on the key; a new row with an existing key replaces the old one. This is the model for mutable records that get updated.
  • AGGREGATE - pre-aggregates rows that share a key according to a function you define (SUM, MAX, MIN, REPLACE), so the table stores rolled-up results rather than raw rows.

The mental model

Think about what should happen when two rows share the same key. In DUPLICATE, both are kept - the table is a faithful log of events. In UNIQUE, the newer one wins - the table is a current-state view where each key holds its latest value. In AGGREGATE, they're combined by your aggregation function - the table is a running summary that never stores the individual rows at all. That single question - keep both, replace, or combine - is the whole decision.

A quick decision rubric

  • Raw event logs, detail analytics, need every row? DUPLICATE. Logs, clickstreams, anything where you'll slice the raw data later.
  • Records that change over time, need the latest state per key? UNIQUE. Orders with a status, user profiles, any upsert workload.
  • Only ever query aggregates, don't need individual rows? AGGREGATE. Pre-summed metrics, counters, rollups feeding dashboards.

If you're unsure and might need the raw data later, default to DUPLICATE - you can always aggregate at query time, but you can't recover detail AGGREGATE threw away.

Minimal SQL

-- Raw events: keep everything
CREATE TABLE events (ts DATETIME, user_id BIGINT, action VARCHAR(64))
DUPLICATE KEY(ts, user_id) ...;

-- Mutable state: latest row per key wins
CREATE TABLE orders (order_id BIGINT, status VARCHAR(32), updated DATETIME)
UNIQUE KEY(order_id) ...;

-- Pre-aggregated metrics: sum on insert
CREATE TABLE daily_revenue (day DATE, region VARCHAR(32),
                            revenue BIGINT SUM)
AGGREGATE KEY(day, region) ...;

The SUM on the revenue column is what tells AGGREGATE how to combine rows that share (day, region).

A worked example: a streaming dashboard

Suppose events stream in and you want a real-time revenue-by-region dashboard. You have two reasonable designs. Store raw events in a DUPLICATE table and aggregate at query time - maximum flexibility, you keep every event for other analyses, at the cost of heavier queries. Or store an AGGREGATE table keyed by (day, region) with revenue SUM - the dashboard reads tiny pre-summed rows and is extremely fast, but you've given up the individual events. If the events feed only this dashboard, AGGREGATE wins on speed; if they feed many analyses, DUPLICATE keeps your options open. The choice encodes a real tradeoff, which is exactly why it's worth making consciously.

Best practices

  • Decide the model from the access pattern, not habit - ask "keep, replace, or combine" before creating any table.
  • Prefer DUPLICATE when in doubt - you can aggregate later; you can't un-aggregate.
  • Use UNIQUE for upserts rather than bolting deduplication onto a DUPLICATE table.
  • Reserve AGGREGATE for write-once-summarize workloads where individual rows truly don't matter.

The takeaway

DUPLICATE, UNIQUE, and AGGREGATE encode three different answers to one question: what happens when rows share a key - keep both, replace, or combine. Match the model to how the data is written and read, default to DUPLICATE when the future is uncertain, and you'll avoid the quiet bugs that come from a table doing something other than what you assumed. It's a small decision at CREATE TABLE time and a permanent one in effect - so make it on purpose.

Leave a Comment