What is the Change Data Capture pattern?
A CDC connector, such as Debezium, reads the database's write-ahead log or binlog and publishes an event for each committed insert, update and delete. Because it reads the log, it only sees committed changes, in commit order, and adds almost no load to the database. Consumers use the stream to update caches, search indexes, read models or other services.
What problem does the Change Data Capture pattern solve?
Polling for changes is slow, misses intermediate states and loads the database. Making application code publish events next to every write risks the dual-write problem. The transaction log already records every change exactly once, so reading it is both reliable and cheap.
How do you implement the Change Data Capture pattern?
- Enable the log format your connector needs (logical replication in PostgreSQL, row-based binlog in MySQL).
- Run a connector that publishes changes to a broker, one topic per table or per aggregate.
- Decide whether consumers get raw row changes or curated events, for example by capturing an outbox table instead.
- Plan the initial snapshot and how consumers handle replays and duplicates.
- Monitor connector lag and the log retention it needs, so the log is not truncated under it.
When should you use it?
- You need to keep caches, search indexes or read models in sync with a database.
- You are publishing events from a legacy system you cannot change.
- You are implementing a transactional outbox and want a log-tailing relay.
When should you avoid it?
- Consumers would be coupled to another team's table layout. Publish domain events instead.
- You have no one to operate the connector and broker.
What do you gain?
- Captures every committed change, in order
- No changes to application code
- Low overhead on the database
What does it cost?
- Raw table changes expose the internal schema
- Another piece of infrastructure to run
- Schema changes need coordination with consumers
What are common mistakes with the Change Data Capture pattern?
- Publishing raw table rows as a public contract, so every column rename breaks consumers.
- Letting the log fill the disk when the connector falls behind.
- Assuming exactly-once delivery when the connector can redeliver after a restart.
Change Data Capture in practice
Glimmer Streaming
Caches and search indexes were refreshed by polling the database every few seconds. Polling made up a large share of database load at peak and still missed short-lived changes.
A CDC connector now tails the database's write-ahead log and publishes each committed change to a topic. Polling was removed, and consumers see changes shortly after commit.
Example decision record
- Context
- Polling the database to refresh the search index uses a large share of its connections.
- Decision
- Stream committed changes from the write-ahead log to the broker with a CDC connector.
- Consequences
- Polling load on the database goes away.
- Consumers depend on the table schema unless changes are mapped to stable events.