Circuit Breaker Lab
Send traffic through a client to a dependency you control. Break it, slow it down, overload it, then watch how the breaker and the retry settings decide whether the rest of your system survives.
Background: Circuit Breaker, Retry with Backoff, Bulkhead.
- No events yet. Crash the dependency to see the breaker react.
Run these in order
Watch an outage with no protection
Load the setup. Every request and every retry still hits the dead dependency. Watch the load counter: it is several times your traffic.
Switch the breaker on
Tick “Breaker enabled”. After 5 failures in a row the bar turns red. Calls now fail in a few milliseconds and the dependency stops receiving traffic.
Heal it and watch recovery
Click “Heal dependency”. When the open time ends the breaker goes amber, lets one probe through, and closes on success. Try a longer open time and see how recovery slows.
Compare a hang with a crash
Load the hanging dependency. Each call burns the whole timeout, so average response time climbs. Turn the breaker on and compare. Then lower the timeout.
Trigger a retry storm
Load the storm. Retries push load above capacity, which causes more failures and more retries. Then tick jitter, reduce retries, or enable the breaker, one at a time.
See what a breaker will not catch
Load the flaky setup. At 40% random errors the breaker only trips when five failures happen in a row by chance (about 1% of calls), so it opens at odd moments and says little about the real error rate. Raise retries from 0 to 3 and compare the success rate. Then lower the threshold to 2 and see how often it trips.
This lab simplifies reality on purpose. It models one dependency, trips the breaker on consecutive failures, lets a single probe through when half-open, and retries with exponential backoff. Real breakers often trip on a failure rate over a sliding window, and libraries such as Resilience4j, Polly and Envoy expose those options.