A ClickHouse cluster runs fine until a routine deploy spikes load, a node blips, and schema changes stall. Dashboards still read, but writes and ON CLUSTER DDL hang. The cause usually isn't storage or CPU - it's coordination. ClickHouse Keeper is the service that makes replication and distributed DDL reliable, and it behaves well in production only when you treat it like a control plane rather than a sidecar.

Keeper in one minute

Keeper is ClickHouse's built-in, ZooKeeper-compatible coordination service. It runs RAFT for consensus and coordinates two things: replicated tables (Replicated*MergeTree metadata and replication queues under /clickhouse/tables/…) and distributed DDL (ON CLUSTER tasks queued so every node applies the change). Writes through Keeper are linearizable; reads are ZooKeeper-like and non-linearizable by default. It runs either as a stand-alone service or embedded in ClickHouse, with the same <keeper_server> config at its core. The useful mental model is etcd in Kubernetes: small, redundant, boring, and protected.

Production topologies

• Dedicated Keeper hosts, not co-located with ClickHouse servers, to isolate I/O and latency.
• Odd-node RAFT quorum - start with 3. ClickHouse's reference "replication + scaling" example coordinates two shards × two replicas with a 3-node Keeper.
• Modest sizing - Keeper nodes are light; roughly 4 GB RAM is enough until clusters grow large.
• Separate disks from ClickHouse data. Coordination is sensitive to disk latency.

Cluster	Tolerates	When to choose
3-node	1 node loss	Most production clusters - simplest ops, lowest overhead
5-node	2 node loss	Stricter SLAs, multi-AZ, or maintenance headroom - more ops cost

A minimal Keeper config

Stand-alone Keeper cluster, 3 nodes (each with a unique server_id):

<!-- /etc/clickhouse-keeper/keeper_config.xml -->
<clickhouse>
  <keeper_server>
    <tcp_port>9181</tcp_port>
    <server_id>1</server_id>
    <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
    <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
    <raft_configuration>
      <server><id>1</id><hostname>keeper-1</hostname><port>9444</port></server>
      <server><id>2</id><hostname>keeper-2</hostname><port>9444</port></server>
      <server><id>3</id><hostname>keeper-3</hostname><port>9444</port></server>
    </raft_configuration>
  </keeper_server>
</clickhouse>

Point the ClickHouse servers at Keeper through the ZooKeeper-compatible block:

<!-- /etc/clickhouse-server/config.d/keeper.xml -->
<clickhouse>
  <zookeeper>
    <node><host>keeper-1</host><port>9181</port></node>
    <node><host>keeper-2</host><port>9181</port></node>
    <node><host>keeper-3</host><port>9181</port></node>
  </zookeeper>
</clickhouse>

Observability: daily checks

Run these from clickhouse-client:

-- Keeper connectivity (exists only if ZK/Keeper configured)
SELECT * FROM system.zookeeper LIMIT 5;

-- Replication health per table on the local node
SELECT database, table, is_leader, is_session_expired, absolute_delay
FROM system.replicas;

-- What is stuck in replication?
SELECT database, table, replica_name, position, source_replica, required_quorum
FROM system.replication_queue ORDER BY position LIMIT 20;

-- Keeper call latency and errors
SELECT * FROM system.zookeeper_log ORDER BY event_time DESC LIMIT 20;

You can also poke Keeper directly with its client utility:

clickhouse-keeper-client -h keeper-1 -p 9181 --query "ls /"
clickhouse-keeper-client -h keeper-1 -p 9181 --query "create /ops_sanity 'ok'"

Failover behavior

Leader loss: RAFT elects a new leader from the remaining nodes (majority required). Writes that rely on Keeper pause during the election; reads continue from local parts. Keeper unavailable (quorum lost): metadata operations for replicated tables and ON CLUSTER DDL fail or queue while the server retries connecting. The translation: losing quorum means no changes to replicated metadata, but queries over existing local parts still run, and synchronization resumes when Keeper recovers.

Disaster drills

Rehearse these in a staging cluster that mirrors the production topology.

1. Leader loss (single node). Stop the current leader (systemctl stop clickhouse-keeper on keeper-1). Watch the election stabilize via clickhouse-keeper-client -h keeper-2 --query "stat", then confirm system.zookeeper reads succeed and system.replication_queue advances again.
2. Follower loss (no quorum loss). Stop a non-leader on a 3-node cluster, insert into a replicated table, and confirm new parts appear on other replicas while the queue stays small.
3. Quorum loss (majority down). Stop 2 of 3 nodes, attempt a CREATE TABLE … ON CLUSTER or an insert into a ReplicatedMergeTree table, and expect errors or queuing. Restore quorum, then SYSTEM SYNC REPLICA db.tbl and check the replication queue drains.
4. Replica rebuild (node replacement). On a dead node, SYSTEM DROP REPLICA 'replica_name' FROM db.tbl to clear its Keeper path, then recreate the table on the new node with the same Keeper path and macros - it fetches parts automatically.

Keeper's own backups

Keeper maintains snapshots and logs under the configured paths. These are not format-compatible with ZooKeeper (a converter exists for ZK→Keeper migration), so put those directories on reliable storage and include them in backups.

Real-world snippets

A replicated table uses macros for unique Keeper paths:

CREATE TABLE sales.events ON CLUSTER prod_cluster
(
  ts DateTime,
  user_id UInt64,
  amount Float32
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/sales.events', '{replica}')
ORDER BY (ts, user_id);

Distributed DDL fans out to every node through Keeper's DDL queue:

ALTER TABLE sales.events ON CLUSTER prod_cluster
ADD COLUMN payment_method LowCardinality(String);

Common pitfalls

Pitfall	Symptom	Fix
Co-locating Keeper with hot ClickHouse storage	Keeper timeouts, flapping sessions	Dedicated hosts or separate disks - it's latency-sensitive
Even number of Keeper nodes	No majority when half go down	Use odd-node RAFT clusters (3 or 5)
Ignoring DDL orchestration	Drifted schemas, stuck queries	Always use ON CLUSTER; monitor the DDL queue
No visibility into replication	Surprise lag	Watch system.replicas and system.replication_queue
Never testing quorum loss	Outage during an AZ failure	Run the drills above; practice SYSTEM SYNC REPLICA recovery

The short version

ClickHouse performance depends on coordination health. Put Keeper on dedicated nodes, use odd-node RAFT quorums, watch the replication and DDL queues, and rehearse failure in staging. Three things to do today: add dashboards for system.replicas and system.replication_queue; run the leader-loss drill and record recovery time; audit configs to confirm Keeper isn't sharing disks with data. Done well, topology hiccups become non-events.