Change Data Capture (CDC) — the OLTP→OLAP Bridge

Read the WAL, not the tables

Nightly SELECT * FROM orders dumps are the dark age. They miss deletes, miss intra-day changes, hammer the OLTP DB, and scale linearly with table size.

Log-based CDC reads the database's write-ahead log (Postgres WAL, MySQL binlog, SQL Server CDC tables) and publishes every row change as an event — typically into Kafka via Debezium. The result:

• Captures inserts, updates and deletes.
• Near-zero load on the OLTP DB (read from the log, not the tables).
• Naturally event-driven: every downstream system gets the same stream.

Query-based CDC (poll WHERE updated_at > last_seen) still has its place for sources without log access, but it cannot see deletes and cannot guarantee no missed changes under concurrent writes.

Query-based CDC is checking your bank balance once a day and guessing what happened: if it's the same at 9am and 9pm you assume nothing moved — but you'd never see the deposit and withdrawal that cancelled out, and you'd miss a closed account entirely. Log-based CDC is reading the itemised statement instead: every single transaction, in order, including the ones that net to zero and the account closures (deletes). And you read it from the bank's own ledger (the write-ahead log) without ever bothering the teller — so the production database barely notices you're watching.