Circuit Breakers
After enough failures, stop calling a broken dependency and fail immediately instead of queueing work behind it.
err := breaker.Do(func() error { return call(ctx) })Timeouts bound one call. They do not help when a dependency is down, because every call still waits the full timeout before failing. A thousand requests a second against a dead service means a thousand goroutines a second each sitting for two seconds. You run out of goroutines, connections or memory, and now two services are down instead of one.
A circuit breaker is a small piece of state in front of a dependency. It counts failures. Once there are too many it stops making calls at all and returns an error straight away. The name comes from household wiring: the breaker opens the circuit so nothing downstream is energised.
Three states
| State | Behaviour |
|---|---|
| Closed | Calls go through. Failures are counted. |
| Open | Calls fail instantly without touching the dependency. |
| Half-open | After a cooldown, one probe call is allowed. Success closes the breaker, failure opens it again. |
Counting the right things
Only count failures that mean the dependency is unhealthy: timeouts, refused connections, 5xx responses. A 404 or a 400 is your caller's mistake, and a cancelled context means your own caller walked away.
What trips people up
Keep one breaker per dependency, not one for the whole program, or a broken search service will cut off your database. Go 1.27 has no breaker in the standard library, so you either write the thirty lines yourself or take a third-party one. Either way the breaker is shared state read and written by many goroutines, so guard it with a mutex or the sync/atomic types.
Example
package main
import (
"errors"
"fmt"
"sync"
)
// errOpen is returned when the breaker refuses to make the call.
var errOpen = errors.New("circuit open")
// Breaker trips after threshold consecutive failures, then fails fast.
type Breaker struct {
threshold int
mu sync.Mutex
failures int
}
func (b *Breaker) Do(work func() error) error {
b.mu.Lock()
tripped := b.failures >= b.threshold
b.mu.Unlock()
if tripped {
return errOpen // the dependency is never dialled
}
err := work()
b.mu.Lock()
defer b.mu.Unlock()
if err != nil {
b.failures++
return err
}
b.failures = 0 // one success clears the count
return nil
}
func main() {
b := &Breaker{threshold: 2}
down := func() error { return errors.New("connection refused") }
for i := 1; i <= 4; i++ {
fmt.Printf("call %d: %v\n", i, b.Do(down))
}
}
// Output:
// call 1: connection refused
// call 2: connection refused
// call 3: circuit open
// call 4: circuit openWhen to use it
- Keep a dead payment provider from exhausting your request goroutines during an outage.
- Give a struggling downstream service room to recover instead of hammering it with retries.
- Fail a non-essential recommendation call in microseconds so the page still renders.
- Surface a dependency's health as a metric by exporting the breaker's state.
More examples
The half-open probe
A breaker that never closes again is just an outage. This adds the cooldown: after it expires, exactly one call is let through to test the water. Taking now as an argument rather than calling time.Now inside makes the method testable, because a table test can pass any instant it likes. The iota block gives the three states readable names.
type state int
const (
closed state = iota
open
halfOpen
)
// allow reports whether a call may proceed, and moves open to halfOpen
// once the cooldown has passed.
func (b *Breaker) allow(now time.Time) bool {
b.mu.Lock()
defer b.mu.Unlock()
switch b.state {
case open:
if now.Sub(b.openedAt) < b.cooldown {
return false
}
b.state = halfOpen // let exactly one probe through
return true
case halfOpen:
return false // a probe is already in flight
default:
return true
}
}Deciding what counts as a failure
This is the function that decides whether the breaker learns anything from a call. A context.Canceled error means your own caller hung up, which says nothing about the dependency's health, so it is excluded. 429 is included because a dependency shedding load wants fewer calls, which is exactly what an open breaker gives it. Write one of these per client and unit-test it.
// countsAsFailure reports whether this result means the dependency is unhealthy.
func countsAsFailure(resp *http.Response, err error) bool {
if err != nil {
// Our own caller giving up is not the dependency's fault.
return !errors.Is(err, context.Canceled)
}
// 5xx is the server failing; 429 is the server asking for less traffic.
return resp.StatusCode >= 500 || resp.StatusCode == http.StatusTooManyRequests
}One breaker per dependency
A single global breaker means any one broken dependency blocks calls to all the healthy ones. Keying breakers by name keeps the failure contained. The map is created lazily under a mutex because handlers reach for breakers from many goroutines at once. Reach for this when a service talks to more than two or three backends.
type breakers struct {
mu sync.Mutex
m map[string]*Breaker
}
// get returns the breaker for one named dependency, creating it on first use.
func (bs *breakers) get(name string) *Breaker {
bs.mu.Lock()
defer bs.mu.Unlock()
b, ok := bs.m[name]
if !ok {
b = &Breaker{threshold: 5, cooldown: 10 * time.Second}
bs.m[name] = b
}
return b
}
Discussion