MicroservicePatterns.com
Communication · Core

Idempotent Consumer

The Idempotent Consumer pattern makes processing the same message more than once have the same effect as processing it once, usually by recording each message ID with its side effects.

Last updated

What is the Idempotent Consumer pattern?

Brokers and relays deliver at least once, so a consumer will sometimes see the same message again after a crash, a timeout or a rebalance. An idempotent consumer detects the duplicate and skips it, or applies changes that are naturally safe to repeat. The same idea applies to APIs through idempotency keys.

What problem does the Idempotent Consumer pattern solve?

Exactly-once delivery across a network is not something you can rely on. Without idempotency, a redelivered PaymentRequested message charges the customer twice. Making consumers safe to repeat is cheaper and more reliable than trying to prevent duplicates.

How do you implement the Idempotent Consumer pattern?

  1. Give every message a stable unique ID, set by the producer.
  2. In the consumer, record the ID in a processed-messages table in the same transaction as the side effect, with a unique constraint.
  3. If the insert fails because the ID exists, acknowledge the message and do nothing else.
  4. Where possible, make the operation itself repeatable, such as 'set status to shipped' rather than 'add one'.
  5. Keep processed IDs for longer than the longest redelivery window.

When should you use it?

  • You consume from any broker or relay with at-least-once delivery, which is almost all of them.
  • Duplicate side effects would cost money or trust: charges, emails, stock changes.

When should you avoid it?

  • The operation is already naturally idempotent and has no external side effects.

What do you gain?

  • Safe retries and redeliveries
  • Lets producers and brokers favour reliability over exactly-once
  • Simple to implement with a unique constraint

What does it cost?

  • Extra write per message
  • Needs a retention policy for processed IDs
  • External side effects (emails, third-party calls) need their own keys

What are common mistakes with the Idempotent Consumer pattern?

  • Checking for the ID in one transaction and doing the work in another, which leaves a gap.
  • Generating the ID in the consumer, so a redelivery gets a new ID.
  • Forgetting that a third-party call made before the crash still happened.

Idempotent Consumer in practice

Real-world example

Stripe

A client that times out cannot tell whether its charge request failed or succeeded and only the response was lost. Retrying blindly risks charging a customer twice.

Stripe's API accepts an idempotency key on each mutating request. A retry with the same key is matched to the original operation, and if that operation already succeeded the stored result is returned instead of running it again. Stripe pairs this with exponential backoff and jitter in its client libraries. The same idea applies to message consumers that see redelivered messages.

Source: Designing robust and predictable APIs with idempotency (Stripe, 2017)

Example decision record
Context
Consumer rebalancing causes invoice events to be processed more than once.
Decision
Record each processed message ID under a unique constraint in the same transaction as the side effect.
Consequences
  • Duplicates are acknowledged without changing state.
  • Processed IDs must be kept for longer than the broker's redelivery window.