Consumer Groups & Rebalance

How partitions get handed out — and why your last outage was a rebalance storm.

0/3 done

Pick cooperative-sticky

Cooperative-sticky beats eager

A consumer group is a set of consumers that subscribe to the same topics; the group coordinator (a broker) assigns each partition to exactly one member. When a member joins/leaves, the group rebalances.

Two protocols:

  • Eager (range / roundrobin) — every consumer revokes everything, then reassigns. Stop-the-world. Painful at scale.
  • Cooperative-sticky (partition.assignment.strategy=CooperativeStickyAssignor) — only the partitions that actually need to move are revoked. Use this.

Visualization

Click a node to focus its neighbourhood · drag to pan · scroll to zoom

A 3-consumer group reading a 6-partition topic — and what happens when one consumer leaves.

Cooperative-sticky consumer in Go

Author a Go consumer that joins group orders-fanout with the cooperative-sticky assignor and commits offsets only after successful side-effects.

Reading in progress · 0 of 3 activities done