What is the Transactional Outbox pattern?
Instead of updating the database and then publishing to a broker, write the event into an outbox table in the same transaction as the business change. A separate relay, either a poller or change-data-capture such as Debezium, reads the outbox and publishes the events, marking them as sent.
What problem does the Transactional Outbox pattern solve?
Updating a database and publishing a message are two separate systems, and no single transaction covers both. If the process crashes between them you either lose the event or publish one for a change that never committed. The outbox makes the two atomic.
How do you implement the Transactional Outbox pattern?
- Add an outbox table with an event ID, aggregate ID, type, payload and timestamp.
- Insert the event row inside the same transaction as the business change.
- Run a relay that polls the table or tails the database log and publishes to the broker.
- Accept at-least-once delivery. Include the event ID so consumers can drop duplicates.
- Delete or archive published rows so the table does not grow forever.
When should you use it?
- A state change must reliably produce an event for other services.
- Sagas or read models depend on those events arriving.
When should you avoid it?
- Events are best-effort notifications where an occasional loss is fine.
- You already use event sourcing, where the event store is the source of truth and the log itself.
What do you gain?
- No lost or phantom events
- No distributed transaction
- Works with any database that has transactions
What does it cost?
- Delivery is at-least-once, so duplicates happen
- An extra table and relay process to run
- Adds some publishing latency
What are common mistakes with the Transactional Outbox pattern?
- Publishing to the broker inside the transaction anyway, which brings the dual write back.
- Consumers that are not idempotent.
- Several relay instances publishing the same aggregate's events out of order.
Transactional Outbox in practice
Cobble Commerce
The order service committed orders to MySQL and then published to Kafka. When the publish failed after the commit, merchants never heard about the order and their stock levels drifted.
Order events are now written to an outbox table in the same transaction as the order, and a relay publishes them. Events can be delivered more than once, so downstream consumers were made idempotent.
Example decision record
- Context
- Writing an order to the database and then publishing to the broker sometimes loses events when the publish fails.
- Decision
- Write events to an outbox table in the same transaction as the order, and publish them with a log-tailing relay.
- Consequences
- Events are not lost when the broker is unavailable.
- Delivery is at-least-once, so every consumer must be idempotent.