Brokers, Topics, Partitions

Brokers, topics, partitions

• A broker is a Kafka server process. A cluster is a set of brokers.
• A topic is a named log; producers write to it, consumers read from it.
• A partition is an ordered sub-log; the unit of parallelism and ordering.

Ordering is guaranteed within a partition, never across partitions. Therefore the key you pick for a record decides which partition it lands in (by hash) — and therefore which records are ordered relative to each other.

Rule of thumb: co-partition by the entity whose causal ordering matters (order_id, account_id, device_id).

Producers → broker → partitions of a topic → consumers in a group.

Lab: a single-node KRaft + Schema Registry + Connect stack

ZooKeeper is being phased out; modern Kafka deployments run in KRaft mode where a Raft-elected controller quorum owns metadata. The compose file below boots a single combined controller+broker node, the Confluent Schema Registry and a Connect worker so you can iterate locally.