Colocation vs. Bucket Shuffle in Apache Doris: When Local Joins Win (and How to Enforce Them)

The problem you actually have

Your BI queries join a few fat tables all day. Broadcast is too big, full shuffle is too slow, and your PM still wants sub-second metrics. Good news: Apache Doris can make joins effectively “local” so the network stops being your bottleneck. The two heavy hitters are Colocation Join and Bucket Shuffle Join. Use them right and your dashboards feel instant; use them wrong and you’re back to shuffling mountains of data. (Apache Doris)

Quick mental model: four ways Doris moves data for joins

Shuffle method	Network cost	When it applies
Broadcast	`N × T(R)`	Small right side, many join nodes
Partition Shuffle	`T(S) + T(R)`	General hash joins
Bucket Shuffle	`T(R)`	Join key hits the left table’s bucket col; left is single partition
Colocation	`≈ 0`	Both sides already hash-distributed identically; same colocate group

As Doris’ distribution requirements get stricter, performance usually improves. But beware: too few buckets can reduce parallelism and make Bucket Shuffle/Colocation slower than a regular shuffle. (Apache Doris)

What each actually means (no fluff)

Bucket Shuffle (fast, flexible)

If the join condition includes the left table’s bucket column, Doris keeps the left table in place and moves only the right side to the corresponding buckets. That’s why the network cost looks like T(R) instead of T(S)+T(R). This is chosen automatically when conditions match. Key constraints: equality join, both sides include bucket columns, and for physical tables the left side should be a single partition to guarantee correctness. (Apache Doris)

Colocation (fastest, but strict)

If both tables are already hash-distributed on the same keys with the same bucket count and replica count, Doris can do the join locally with no shuffle. You enforce this by putting tables into the same Colocation Group via PROPERTIES("colocate_with"="..."). If the group is unstable (e.g., replica repair or balancing), Doris will degrade to a normal shuffle plan. (Apache Doris)

How to enforce each strategy

1) Enforce Colocation (DDL you can copy)

-- Fact
CREATE TABLE fact_orders (
  dt DATE, user_id BIGINT, sku BIGINT, qty INT, price BIGINT
)
DUPLICATE KEY(dt, user_id)
PARTITION BY RANGE(dt) (
  PARTITION p2025_11 VALUES LESS THAN ("2025-12-01")
)
DISTRIBUTED BY HASH(user_id) BUCKETS 32
PROPERTIES (
  "replication_num" = "3",
  "colocate_with"   = "sales_cg"   -- put in the same colocate group
);

-- Dimension
CREATE TABLE dim_users (
  user_id BIGINT, city VARCHAR(64), segment VARCHAR(32)
)
DUPLICATE KEY(user_id)
DISTRIBUTED BY HASH(user_id) BUCKETS 32
PROPERTIES (
  "replication_num" = "3",
  "colocate_with"   = "sales_cg"   -- must match: key(s), bucket count, replicas
);

Verify it’s really local:

-- Shows colocate: true in the plan when group is stable
EXPLAIN SELECT /* check plan */ 1
FROM fact_orders f
JOIN dim_users d ON f.user_id = d.user_id;

In a colocated plan, the explain string includes colocate: true. If the group is unstable, you’ll see it degrade (e.g., a broadcast/shuffle appears). (Apache Doris)

Monitor the group:

-- Requires admin privileges
SHOW PROC '/colocation_group';
-- check IsStable=true; if false, joins degrade until balancing/repair completes

You can also drill into a specific group’s bucket-to-BE placement. (Apache Doris)

Move tables in/out of a group later:

ALTER TABLE fact_orders SET ("colocate_with" = "sales_cg");  -- add/move
ALTER TABLE fact_orders SET ("colocate_with" = "");          -- remove

And if you must change replication for a colocate group, do it at the group level:
ALTER COLOCATE GROUP group_name SET ("replication_num" = "3"); (Apache Doris)

Cross-database joins? Use a global group name with __global__ prefix in colocate_with. (Apache Doris)

2) Encourage Bucket Shuffle (SQL patterns that “nudge” the planner)

Bucket Shuffle is about query shape more than DDL. Aim for the left side to keep its bucket distribution and be single-partition:

-- Left side stays put (bucketed by user_id). We prune to one partition.
EXPLAIN
SELECT SUM(f.qty * f.price) AS revenue, d.segment
FROM fact_orders f
JOIN dim_users d
  ON f.user_id = d.user_id      -- join uses the left bucket column
WHERE f.dt = '2025-11-15'       -- single partition on the left
GROUP BY d.segment;

In a fitting plan you’ll see join op: ... (BUCKET_SHUFFLE) and the right side being exchanged to the left. If you can’t meet the constraints, Doris falls back to partition shuffle. (Apache Doris)

If you must override distribution explicitly: Doris supports distribute hints to pick [broadcast] or [shuffle] for the right table, useful when colocation isn’t possible and the planner didn’t choose what you want:

SELECT COUNT(*) 
FROM big_left l
JOIN [broadcast] small_dim r ON r.k = l.k;  -- force broadcast

(There is no explicit “bucket shuffle hint”; Doris selects it automatically when conditions are met.) (Apache Doris)

Choosing between Colocation and Bucket Shuffle

Use Colocation when:

The same keys drive most joins (e.g., user_id or order_id).
You control table design (you can align keys, bucket count, and replication).
You need lowest latency and can live with stricter layout rules.
Why: true local joins (zero shuffle) when the group is stable. (Apache Doris)

Use Bucket Shuffle when:

Joins vary, or only one side is aligned.
You can filter the left to a single partition and the join column is the left’s bucket column.
Why: move only the right side; easier to satisfy than full colocation. (Apache Doris)

Production checklist (do these, avoid pain)

Design

Pick stable join keys; set DISTRIBUTED BY HASH(...) BUCKETS N consistently.
For colocated joins, set PROPERTIES("colocate_with"="...") on both tables. (Apache Doris)
Use enough buckets for concurrency; too few limits parallelism (even for colocated/bucket shuffle). (Apache Doris)

Operate

Watch SHOW PROC '/colocation_group' and keep IsStable=true; degraded groups lose colocated plans.
Change replication/bucket counts consistently across the group (use ALTER COLOCATE GROUP for replicas). (Apache Doris)

Query

Prefer predicates that prune partitions on the left side for Bucket Shuffle.
Use hints only when necessary ([broadcast] or [shuffle]), and verify with EXPLAIN. (Apache Doris)

Common pitfalls (and the fix)

Group is “unstable” → plan shuffles anyway. That’s normal during balancing/repair. Wait until stable or pause auto-balance temporarily if it hurts SLAs. (Apache Doris)
Different bucket counts or replica numbers. You won’t get colocated joins until these match in the group. (Apache Doris)
Left table isn’t single-partition. Bucket Shuffle may not trigger; add a date filter or refactor partitions. (Apache Doris)
Too few buckets. You killed parallelism; increase buckets and rebalance. (Apache Doris)

Minimal, end-to-end example you can run

-- 1) Two tables aligned for colocation
CREATE TABLE a (k BIGINT, v BIGINT) DISTRIBUTED BY HASH(k) BUCKETS 16
PROPERTIES("replication_num"="3", "colocate_with"="cg_demo");

CREATE TABLE b (k BIGINT, v BIGINT) DISTRIBUTED BY HASH(k) BUCKETS 16
PROPERTIES("replication_num"="3", "colocate_with"="cg_demo");

-- 2) Verify group status
SHOW PROC '/colocation_group';

-- 3) Expect colocate: true
EXPLAIN SELECT /* colocated */ SUM(a.v+b.v) FROM a JOIN b USING(k);

-- 4) Bucket Shuffle pattern (left is single partition)
EXPLAIN
SELECT /* bucket shuffle */ COUNT(*)
FROM fact_orders f  -- partitioned by dt, bucketed by user_id
JOIN dim_users d    -- bucketed by user_id
  ON f.user_id = d.user_id
WHERE f.dt = CURRENT_DATE();  -- prune to one partition on the left

Check the plans: colocate: true vs. join op: ... (BUCKET_SHUFFLE). (Apache Doris)

TL;DR

Colocation = best latency, strict layout (same keys, buckets, replicas, group stable).
Bucket Shuffle = great latency, looser constraints (left’s bucket col in join, single-partition left).
Always verify with EXPLAIN and watch group stability. (Apache Doris)

Call to action: Pick your two most expensive joins. For one, add a colocate group; for the other, shape the SQL to hit Bucket Shuffle. Measure plan + latency before/after.

Internal link ideas (official docs)

Join (Bucket Shuffle vs Colocate, conditions & examples). (Apache Doris)
Colocation Join (DDL, groups, stability, global groups). (Apache Doris)
Optimizing Join with Colocate Group (troubleshooting stability). (Apache Doris)
ALTER TABLE PROPERTY (modify colocate_with, distribution). (Apache Doris)
ALTER COLOCATE GROUP (change replication for a group). (Apache Doris)
Adjusting Join Shuffle Mode with Hint (force broadcast/shuffle). (Apache Doris)

Image prompt

“A clean, modern diagram: two panels side-by-side. Left panel shows Bucket Shuffle — left table stays in place; right table buckets are routed to matching nodes. Right panel shows Colocation — both tables aligned in the same buckets on the same nodes; join executes locally with no data movement. Minimalist, high contrast, 3D isometric style.”

Data/ML Engineer Blog