MicroservicePatterns.com
Resilience · Core

Circuit Breaker

The Circuit Breaker pattern stops calling a dependency that keeps failing, fails fast for a while, then sends a few probe calls to see whether it has recovered before resuming traffic.

Last updated

What is the Circuit Breaker pattern?

A breaker wraps remote calls and tracks their failures. After too many failures it opens and rejects calls immediately. After a wait it goes half-open and lets a few probe calls through. If they succeed it closes and normal traffic resumes, and if they fail it opens again.

What problem does the Circuit Breaker pattern solve?

When a dependency is slow or down, callers pile up waiting on timeouts, exhaust their own threads and connections, and the failure spreads upstream. Failing fast protects the caller and gives the struggling dependency room to recover.

How do you implement the Circuit Breaker pattern?

  1. Wrap every call to a remote dependency, with a timeout on each call.
  2. Track failures as consecutive errors or a failure rate over a sliding window.
  3. Open when the threshold is crossed, and reject calls immediately with a clear error.
  4. After a wait, go half-open and let a limited number of probe calls through.
  5. Close on success, reopen on failure, and provide a fallback such as cached data, a default or a degraded feature.

When should you use it?

  • You make synchronous calls to services that can fail or slow down.
  • A failure in one dependency must not take the whole request path with it.

When should you avoid it?

  • The call is local and in-process.
  • You have no sensible fallback and the user must see an error anyway. A breaker still helps the dependency, but it will not rescue the user.

What do you gain?

  • Fails fast instead of piling up
  • Gives dependencies time to recover
  • Contains the blast radius

What does it cost?

  • Thresholds need tuning
  • Can open on a blip if set too tight
  • Fallbacks add code to maintain

What are common mistakes with the Circuit Breaker pattern?

  • No timeout on the call, so the breaker never sees a failure.
  • Counting consecutive failures against a dependency that fails 40% of calls at random. The breaker trips only on chance streaks, so it says little about the real error rate. Use a failure rate over a window.
  • One breaker shared across unrelated endpoints.

Circuit Breaker in practice

Real-world example

Netflix

The Netflix API depended on dozens of backend systems. Without isolation, a single slow dependency could tie up request threads on every API server within seconds and take the whole API down with it.

From 2011 the Netflix API team wrapped dependency calls in Hystrix, which isolates each dependency, fails fast while its circuit is open and serves a fallback. Netflix open-sourced Hystrix in 2012. It is no longer in active development, and its README points new projects to libraries such as Resilience4j.

Sources: Hystrix wiki (Netflix, GitHub); Hystrix README (Netflix, GitHub)

Example decision record
Context
Payment and inventory dependencies have intermittent latency spikes that tie up callers.
Decision
Put a circuit breaker on every outbound HTTP and gRPC client, with fallbacks for non-critical calls.
Consequences
  • Callers fail fast while a dependency is unhealthy.
  • Breaker state changes are exported as metrics and alerted on.