Schema Evolution

Schema Evolution

Add columns easily, change types painfully, drop columns dangerously.

Why it matters

Forward / backward compatibility decides whether you can deploy producers and consumers independently. Schema registries (Confluent, Apicurio) enforce it at the broker.

Going deeper

Compatibility modes you'll see in a schema registry:

Mode	Producer can…	Consumer constraint
Backward	Add optional fields; remove fields with defaults	New consumers can read old + new data
Forward	Add fields with defaults	Old consumers can still read new data (ignoring new fields)
Full	Only changes safe in both directions	Producers and consumers deploy in any order
None	Anything	Caveat emptor; every change is a coordinated release

Three concrete tactics that keep evolution cheap:

1. Default values for every new field — unlocks both directions of compat.
2. Deprecate before drop — mark a field deprecated for one release cycle, alert on read access, then drop only after the read counter hits zero.
3. Versioned topics / endpoints for breaking changes — orders.v2 alongside orders.v1, with a documented sunset date for v1.

Schema evolution is the etiquette of changing a shared document.

Imagine a hundred people printing the same form. Adding an optional field to the form is polite — the old printers keep working, anyone who wants to fill in the new field can. Renaming a field is rude — every printer that referenced the old name now prints garbage. Removing a field is hostile — every reader that depended on it silently breaks, often weeks later.

Schema registries (Confluent, Apicurio, AWS Glue) are the meeting chair that rejects rude or hostile changes before they hit the broker, so producers and consumers stay independently deployable.