Pub/Sub vs Message Queue

Two delivery contracts, one log

The classic messaging dichotomy:

• Queue (competing consumers) — each message is delivered to exactly one of N workers. Used to share load. Think RabbitMQ work queue, SQS.
• Pub/Sub (fan-out) — each message is delivered to every interested subscriber independently. Used to broadcast facts. Think SNS, a topic exchange.

Kafka unifies both with one primitive: the consumer group.

• Put all your workers in the same group → the partitions are split among them → queue semantics (each record handled once by the group).
• Give each independent service its own group on the same topic → every group reads every record → pub/sub semantics (fan-out).

So 'queue vs topic' isn't a Kafka deployment choice — it's a group.id choice made per consumer. The log is retained either way, which is the superpower legacy brokers lack: a new subscriber can join tomorrow and replay everything from offset 0.

A newspaper printing press produces one run (the log). A queue is the bundle of papers split between delivery drivers — each house gets one paper, drivers share the round. Pub/sub is every household holding its own subscription — everyone receives the same edition independently. Same press; the subscription model differs per reader.

One topic, two groups: 'billing' (queue across its workers) and 'search-indexer' (its own group) — each group sees every record, members within a group share partitions.