Circuit Breakers and Bulkheads
Fail fast when a dependency is sick, and cap each dependency's concurrency so one slow call cannot fill the process.
CLOSED --(error rate over threshold in window)--> OPEN --(sleep window elapses)--> HALF_OPEN --(trial call)--> CLOSEDA circuit breaker wraps a remote call, counts recent outcomes, and stops calling when too many fail. It exists because waiting on a sick dependency costs more than failing: every in-flight call holds a connection and a worker, so a dependency taking 30 seconds instead of 30 milliseconds does not slow your service down, it fills it up and takes it with it.
Three states
Closed: calls pass through, outcomes are counted over a rolling window. Open: calls fail immediately, without touching the network, for a sleep window. Half-open: one trial call decides whether to close or re-open. Netflix's Hystrix, which most implementations copy, shipped a 10-second rolling window (metrics.rollingStats.timeInMilliseconds), tripping at 50% errors (circuitBreaker.errorThresholdPercentage) but only once the window holds 20 requests (circuitBreaker.requestVolumeThreshold), with a 5-second circuitBreaker.sleepWindowInMilliseconds. The volume threshold is the one people omit, and without it two failures out of two trip the breaker at 4 a.m. when traffic is nil.
Bulkheads
A breaker reacts after failures. A bulkhead stops a failure spreading at all, by giving each dependency a fixed slice of concurrency so a slow one can only ever consume its own allowance. Hystrix used a thread pool per dependency (coreSize default 10, maxQueueSize default -1, meaning no queue at all); a semaphore per dependency does the same job without the threads.
Note the vocabulary clash: Envoy calls its concurrency caps circuit breakers (max_connections, max_pending_requests, max_requests, all defaulting to 1024) and keeps error-rate logic in a separate outlier_detection block. Those are bulkheads by the definition above.
Example
// A circuit breaker with Hystrix's semantics: a rolling window, a minimum
// request volume before it may trip, and a jittered sleep window.
class CircuitBreaker {
constructor(opts = {}) {
this.windowMs = opts.windowMs ?? 10_000; // rollingStats.timeInMilliseconds
this.errorThresholdPct = opts.errorThresholdPct ?? 50;
this.requestVolume = opts.requestVolume ?? 20; // requestVolumeThreshold
this.sleepWindowMs = opts.sleepWindowMs ?? 5_000;
this.timeoutMs = opts.timeoutMs ?? 1_000; // Hystrix default is 1000ms
this.state = 'CLOSED';
this.events = []; // { at, ok } within the rolling window
this.openUntil = 0;
this.probing = false;
}
async call(fn, fallback) {
if (this.state === 'OPEN') {
if (Date.now() < this.openUntil) return this.refuse(fallback);
this.state = 'HALF_OPEN';
}
// In HALF_OPEN exactly one call goes through. Letting the whole backlog
// through is how a recovering dependency gets knocked straight back down.
if (this.state === 'HALF_OPEN') {
if (this.probing) return this.refuse(fallback);
this.probing = true;
}
try {
const out = await fn(AbortSignal.timeout(this.timeoutMs));
this.record(true);
return out;
} catch (err) {
// A timeout is a failure. Slowness is the symptom that matters here.
this.record(false);
if (fallback) return fallback(err);
throw err;
} finally {
this.probing = false;
}
}
record(ok) {
const now = Date.now();
if (this.state === 'HALF_OPEN') {
if (ok) { this.state = 'CLOSED'; this.events = []; } else { this.open(now); }
return;
}
this.events.push({ at: now, ok });
const cutoff = now - this.windowMs;
while (this.events.length && this.events[0].at < cutoff) this.events.shift();
const total = this.events.length;
if (total < this.requestVolume) return; // too few samples to judge
const failed = this.events.reduce((n, e) => n + (e.ok ? 0 : 1), 0);
if ((failed / total) * 100 >= this.errorThresholdPct) this.open(now);
}
open(now) {
this.state = 'OPEN';
// Jitter the sleep window. Without it every instance in the fleet probes
// in the same millisecond, every 5 seconds, indefinitely.
this.openUntil = now + this.sleepWindowMs * (0.5 + Math.random());
this.events = [];
}
refuse(fallback) {
const err = Object.assign(new Error('circuit open'), { code: 'CIRCUIT_OPEN', status: 503 });
if (fallback) return fallback(err);
throw err;
}
}
// One breaker PER DEPENDENCY, not one per process: a shared breaker lets a
// broken recommendations service stop your calls to the payment gateway.
const breakers = {
recommendations: new CircuitBreaker({ timeoutMs: 150 }),
shipping: new CircuitBreaker({ timeoutMs: 800, errorThresholdPct: 25 }),
};When to use it
- Stopping a slow recommendations service from exhausting the worker pool that also serves checkout.
- Serving a cached last-known-good search result while the search cluster reindexes.
- Explaining in an interview why you would still put a breaker in front of a dependency that has no fallback.
More examples
Bulkhead: a semaphore per dependency
This is the pattern that survives the case a circuit breaker does not cover: a dependency that is slow but not failing, so the error rate never crosses the threshold and the breaker stays closed while every worker sits in a socket read. Reach for it when one page calls several services of different importance. Sizing is not guesswork, it is Little's Law, and the deliberate choice to let the pools sum to less than the process capacity is what keeps headroom for the dependency you cannot shed.
// A bulkhead is a hard cap on concurrency per dependency. Little's Law
// gives the size: concurrency = target QPS x expected latency, so 200 rps
// at 50ms needs 10 slots. Anything above that is queue, not throughput.
class Bulkhead {
constructor({ slots, maxQueue }) {
this.free = slots;
this.maxQueue = maxQueue; // Hystrix's maxQueueSize default is -1: no queue
this.waiting = [];
}
async run(fn) {
if (this.free === 0) {
// REJECT rather than queue without bound. A full bulkhead returning an
// error in microseconds is the whole point of the pattern.
if (this.waiting.length >= this.maxQueue) {
throw Object.assign(new Error('bulkhead full'), { status: 503 });
}
await new Promise((resolve) => this.waiting.push(resolve));
}
this.free--;
try {
return await fn();
} finally {
this.free++;
this.waiting.shift()?.();
}
}
}
// Sized from each dependency's own latency, and deliberately summing to less
// than the process can handle, so no single dependency can own the process.
const pools = {
recommendations: new Bulkhead({ slots: 10, maxQueue: 0 }), // optional
search: new Bulkhead({ slots: 25, maxQueue: 50 }),
payments: new Bulkhead({ slots: 40, maxQueue: 100 }), // never shed
};
await pools.recommendations.run(() => breakers.recommendations.call(fetchRecs));A fallback worth having, and when there is none
The question an interviewer follows up with is 'and then what?', and these are the only three answers. The third is the one worth saying out loud: a breaker in front of your primary database does not improve availability, it just fails quickly, and that is still a win because the freed workers can serve the requests that do not need that dependency. The anti-pattern is the fourth option nobody should write, returning a default that the caller cannot distinguish from real data, which turns an outage into silent data corruption.
// A breaker only helps if something sensible happens when it is open.
// Three honest answers, in descending order of preference.
// 1. Stale data. Keep the last good value with a long TTL, separate from the
// short TTL used for freshness, and serve it while the breaker is open.
async function getRecommendations(userId) {
return breakers.recommendations.call(
async (signal) => {
const recs = await recsClient.get(userId, { signal });
await cache.set(`recs:${userId}`, recs, { ttlSeconds: 86_400 }); // stale store
return recs;
},
// Fallback: yesterday's recommendations beat an empty carousel.
async () => (await cache.get(`recs:${userId}`)) ?? [],
);
}
// 2. A static default, and a flag the page can render around.
async function getShippingEstimate(cart) {
return breakers.shipping.call(
(signal) => shippingClient.quote(cart, { signal }),
() => ({ estimate: null, degraded: true }), // UI shows "calculated at checkout"
);
}
// 3. No fallback. For a dependency you genuinely cannot do without, the
// breaker still earns its place: it converts a slow failure into a fast
// one and gives the workers back. Do NOT invent a fake success here.
async function chargeCard(order) {
return breakers.payments.call((signal) => gateway.charge(order, { signal }));
// no fallback argument: a 503 the client can retry, not a fabricated receipt
}
Discussion