Transactional Outbox Lab
An Order service saves each order and tells Shipping about it through a message broker. Two systems have to change, and no transaction covers both. Crash the service at the worst moment, take the broker down, and count the orders that never ship, the shipments for orders that do not exist, and the ones shipped twice.
Background: Transactional Outbox, Idempotent Consumer, Change Data Capture.
Run these in order
Write twice and hope
Commit the order, then publish the event. A crash between the two, or a broker outage, leaves a saved order that Shipping never hears about. Click a red tile to see what happened to it.
Swap the order of the writes
Publish first, then commit. Now a crash ships an order that was never saved. Broker outages fail the request instead, which is consistent but turns every outage into errors for customers.
Use an outbox
Write the order and the event in one local transaction. A relay publishes from the outbox table later. Nothing is lost and nothing is invented: broker outages only delay events. But a relay that crashes after publishing sends the event again.
Make the consumer idempotent
Shipping records each event id and skips repeats. The outbox guarantees at-least-once publishing, and idempotency turns that into exactly-once effects. Raise the crash and outage rates as high as you like: it stays consistent.
This lab simplifies reality on purpose. Each order meets at most one crash and one outage, and the relay always comes back. Real relays poll the outbox table or read the database log with change data capture (for example Debezium). Either way they publish at least once, so consumers still need to handle duplicates.