What is the Dead Letter Queue pattern?
When a consumer fails to process a message a set number of times, the message moves to a separate queue (the dead letter queue) with information about why it failed. The main queue keeps flowing. Someone inspects the dead letters, fixes the cause, and replays or discards them.
What problem does the Dead Letter Queue pattern solve?
Some messages will never succeed: malformed payloads, references to deleted data, bugs in the consumer. Retrying them forever either blocks an ordered queue or wastes capacity. Moving them aside keeps the system running while keeping the evidence.
How do you implement the Dead Letter Queue pattern?
- Set a retry limit with backoff for each consumer, separating errors worth retrying from errors that never will succeed.
- On the final failure, publish the message to the DLQ with the error, stack trace, attempt count and original topic.
- Alert on DLQ depth and age, and route alerts to the owning team.
- Build a replay tool that sends selected messages back to the original queue after a fix.
- Decide how long dead letters are kept, and what happens to ones nobody claims.
When should you use it?
- You process messages from any queue or topic.
- One bad message must not stop the others.
When should you avoid it?
- Strict ordering per key matters and skipping a message would corrupt later ones. Pause that key's partition instead.
What do you gain?
- Bad messages stop blocking good ones
- Failures are kept for diagnosis
- Fixed messages can be replayed
What does it cost?
- Someone has to own and work the queue
- Replays can arrive out of order
- Easy to fill and forget
What are common mistakes with the Dead Letter Queue pattern?
- A DLQ that nobody monitors, so failures pile up silently for weeks.
- Sending transient failures (timeouts) straight to the DLQ without retrying.
- Replaying messages to a consumer that is not idempotent.
Dead Letter Queue in practice
Ledgerline Pay
A webhook payload with an unexpected character made the consumer throw on every attempt. The broker redelivered it forever, and valid notifications queued up behind it.
After a few failed attempts a message now moves to a dead-letter queue with the error attached, and on-call is alerted. The main consumer keeps flowing, and the team replays fixed messages from the DLQ.
Example decision record
- Context
- Malformed messages have repeatedly stalled consumers.
- Decision
- Give every consumer a dead-letter queue and move messages there after a fixed number of failed attempts.
- Consequences
- One bad message no longer blocks the rest.
- Someone must own the DLQ: alert on it, inspect it and replay fixed messages.