Idempotent & Transactional Producers

Idempotent vs transactional

• Idempotent producer — sequence numbers per (producer-id, partition); broker drops duplicates on retry. Per-partition guarantee.
• Transactional producer — transactional.id + initTransactions() + beginTransaction() / commitTransaction() / abortTransaction(). Atomic writes across multiple partitions and the consumer-offsets topic. This is what unlocks exactly-once stream processing (read → process → write).

Pair with consumers set to isolation.level=read_committed so they skip aborted records.

Read from orders, enrich, and write to orders.enriched and orders.audit atomically. If anything throws, the transaction is aborted and no downstream consumer ever sees the partial output.

Mirror the same shape in Go. segmentio/kafka-go exposes idempotence via Writer.AllowAutoTopicCreation/RequiredAcks and the underlying transport. Key requirements: RequiredAcks: RequireAll, balancer keyed on OrderId.