Dead Letter Queue Lab
The Order service publishes an OrderPlaced event for every order, and one consumer takes them from an ordered queue and books each delivery with the Warehouse. At 0:05 a new app release starts sending the odd order with a malformed address, and at 0:20 the Warehouse slows down for three minutes. Choose how the consumer retries and what happens to orders that keep failing, then play the hour and count the orders that are stuck, lost, late or booked twice.
Background: Dead Letter Queue, Asynchronous Messaging, Retry with Backoff and Jitter, Idempotent Consumer.
60 orders a minute. From 0:05, about 1 in 100 has a malformed address. From 0:20 to 0:23 the Warehouse is slow: calls time out after 3 s, though half of them are booked anyway.
Run these in order
Retry for ever
The consumer retries a failed order until it works, and the queue is ordered, so nothing overtakes it. At 0:05 the first order with a malformed address arrives. It will never work, and the consumer tries it about 13,000 times. Every order behind it waits: by the end of the hour more than 3,000 are stuck and the oldest has waited 54 minutes. One bad message stopped the whole flow.
Give up after three tries
Retry three times without waiting, then move the order to a dead letter queue. The queue keeps moving and the 45 malformed orders are set aside. But the Warehouse is slow from 0:20 to 0:23, and three quick retries are over in 12 seconds, so 15 good orders are dead-lettered too. Nobody is alerted: 60 orders sit in the DLQ at the end of the hour and no customer gets them. Untick the DLQ: the same 60 orders are thrown away instead.
Alert and redrive
Page the team when the DLQ holds 10 orders, and move the dead letters back once the fix is live. The alert fires at 0:16, a fix that reads the malformed address is live at 0:31, and the redrive puts 34 orders back on the queue. Every order is delivered now. But some of the orders that failed in the slowdown had been booked: the Warehouse took the booking and then timed out. The redrive books them again, and 15 orders ship twice.
Make the consumer idempotent
Book with the order ID as an idempotency key, so the Warehouse ignores a booking it has already taken. Nothing ships twice, from a retry or from the redrive. The redriven orders still arrived up to 25 minutes late, though: 15 of them were good orders that only needed to wait out a three-minute slowdown.
Back off between retries
Retry up to eight times, waiting 1, 2, 4, 8 seconds and so on, up to a minute. The retries now span three minutes and ride out the slowdown: no good order is dead-lettered. But every malformed order now holds the queue for three minutes before it gives up, and one arrives every minute and a half. The backlog climbs past 1,500 and orders wait over half an hour.
Do not retry what cannot succeed
Keep the backoff for errors that might go away, but send an order that fails validation to the DLQ at once. Malformed orders leave the queue in a quarter of a second, the slowdown is ridden out, and no order waits more than three minutes. The alert, the fix and the redrive deliver the rest, once each. Retries are for transient errors; a DLQ is for the ones that are not.
This lab simplifies reality on purpose. There is one consumer and one ordered queue, so a retry holds up everything behind it; with many consumers and no ordering, a poison message wastes capacity instead of stopping the flow. Traffic is flat, there is one kind of malformed order with one fix, and the fix always takes fifteen minutes. Real DLQs also need a retention period, a way to inspect and discard single messages, and an alert on the age of the oldest dead letter as well as the count. Redriving can also deliver messages out of order, which this lab does not show.