Bulkhead Lab
A Checkout service uses its worker threads to call Payments, which every checkout needs, and Recommendations, which the page can live without. For thirty seconds Recommendations gets slow. Watch whether that stays its own problem or takes checkouts down too.
Background: Bulkhead, Timeout, Circuit Breaker.
Run these in order
Share one pool
Recommendations slows down from 0:15 to 0:45. Its calls hold threads for seconds, the shared pool fills up, and checkouts fail too, even though Payments is perfectly healthy.
Shorten the timeout
A 500 ms timeout frees threads sooner. Recommendations now holds about 30 calls/s × 0.5 s = 15 threads, which still fits in 20, so checkouts mostly survive.
Add more traffic
Raise Recommendations to 45 calls/s. Now it needs about 22 threads during the slowdown, more than the whole pool, and checkouts fail again. A timeout limits how long each call holds a thread, not how many calls hold threads at once.
Put up a bulkhead
Give Recommendations its own 6 threads. When it slows down it fills only its own pool and the rest of its calls are turned away at once. The page shows without recommendations, and checkouts stay at 100%.
Size it from the load
With only 2 threads, recommendations fail even when nothing is wrong. Normal load needs about 30 calls/s × 0.1 s = 3 threads at once (Little's law: threads = rate × time per call). Size each pool from that, plus headroom.
This lab simplifies reality on purpose. A call that finds no free thread is rejected at once instead of waiting in a queue, there are no retries, and traffic does not change during the minute. Real bulkheads are thread pools, semaphores, connection pools or separate instances, and libraries such as Resilience4j and Polly provide them. A circuit breaker in front of the slow dependency would also stop the calls sooner.