MicroservicePatterns.com

Suggestions

  • PagePattern catalogAll patterns by category.
  • LabCircuit Breaker and Retries labCrash a dependency and tune the breaker.
  • LabSaga labFail a step and watch the compensations run.
  • PagePractice questionsScenario questions with explanations.
  • PageDesign reviewCheck your own architecture for anti-patterns.
Resilience

Rate Limiting Lab

Many users share an API with room for 100 requests a second. For twenty seconds one client sends far more than its share. Watch what that does to everyone else, then give each client a limit and see how the noisy client's retries, and the size of the limit, change the outcome.

Background: Rate Limiting, Retry with Backoff and Jitter, API Gateway.

Scenarios
Rate limiter
Clients

The API can serve 100 requests a second. Normal users send 50 a second between them, each far below any per-client limit.

0:00 · normal traffic
Normal users50 req/sNoisy client20 req/sRate limiteroffAPIroom for 100 req/s
–Normal users' requests succeeding
–API load, req/s (room for 100)
–Noisy client served, req/s
–Turned away with 429, per s
noisy client spiking1001,037API load, req/s (dashed line: capacity)0%100%Normal users succeeding
Guided experiments

Run these in order

  1. Let one client flood the API

    No limiter, no retries. From 0:15 one client sends 250 requests a second to an API with room for 100. The API cannot tell who is who, so normal users lose two thirds of their requests too.

  2. Add naive retries

    Every client retries failures at once. Load climbs to about ten times capacity and normal users drop to around 10%. Retrying into an overloaded service makes the overload worse.

  3. Limit each client

    Give each client 30 requests a second. The noisy client is turned away with a cheap 429 and the API stays under capacity, so normal users are fine. The noisy client still hammers the limiter with hundreds of retries a second.

  4. Honour Retry-After

    The noisy client waits as the 429 tells it to. It now sends only what its limit allows, so almost nothing is refused. But its own work piles up in a growing backlog: it has to slow down at the source, or ask for a higher limit.

  5. Set the limit too high

    Allow 80 requests a second. 50 from users plus 80 from one client is more than the API can serve, so the limit no longer protects anyone. Per-client limits must add up to what the service can actually handle.

  6. Allow a big burst

    Let a quiet client save up 400 requests. The first second of the spike gets through all at once and overloads the API before the limit kicks in. Size the burst to what the service can absorb, not to what clients would like.

This lab simplifies reality on purpose. Normal users are pooled into one stream that never reaches its limit, an overloaded API fails requests evenly instead of slowing down, and the limiter itself is free. Real limiters run in a gateway or a shared store such as Redis, often per client and per endpoint, and also choose between token buckets, leaky buckets and sliding windows.