Circuit Breakers and Bulkheads

Reject calls fast while a dependency is down, cap how many requests can wait on it, and return something useful instead.

Syntaxbuilder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions { FailureRatio = 0.1, MinimumThroughput = 100, SamplingDuration = TimeSpan.FromSeconds(30), BreakDuration = TimeSpan.FromSeconds(5) });

A circuit breaker is a counter sitting in front of a dependency. While calls succeed it does nothing. Once enough of them fail inside a sampling window it opens: further calls are rejected immediately, without touching the dependency. After a break duration it goes half-open and lets one call through as a probe. Success closes it again; failure reopens it.

Why rejecting beats trying

While a dependency is down, each attempt costs a full timeout and holds a request open for that whole time. Failing in microseconds instead frees your service to serve everything that does not need that dependency, and it stops you adding load to something already in trouble. In Polly, a rejected call throws BrokenCircuitException.

Bulkheads

A bulkhead is a cap on how many of your requests may be waiting on one dependency at any moment. The name comes from a ship: a flood fills one compartment, not the hull. Polly calls this AddConcurrencyLimiter, and the standard HTTP handler includes one with a permit limit of 1000 and a queue limit of 0. Without a cap, a single slow dependency can absorb every in-flight request in the process.

Fallbacks, and the threshold that bites

A fallback is what you return instead of an error; AddFallback supplies it. The circuit-breaker defaults are a failure ratio of 0.1 over a 30-second sampling window with a minimum throughput of 100 calls, and a 5-second break. That minimum is the part people miss.

Example

Example · csharp
// A breaker in its simplest form: count consecutive failures, and once the
// limit is reached stop calling the dependency at all.
Breaker breaker = new(failuresBeforeOpening: 3);

for (int attempt = 1; attempt <= 5; attempt++)
{
    if (!breaker.AllowsCall())
    {
        // No timeout, no socket, no load on the dependency.
        Console.WriteLine($"call {attempt}: rejected, circuit open");
        continue;
    }

    try
    {
        await CallDependencyAsync();
        breaker.RecordSuccess();
        Console.WriteLine($"call {attempt}: ok");
    }
    catch (HttpRequestException)
    {
        breaker.RecordFailure();
        Console.WriteLine($"call {attempt}: failed");
    }
}

// The dependency is down for the whole run.
static Task CallDependencyAsync()
{
    return Task.FromException(new HttpRequestException("connection refused"));
}

// Counts failures in a row and refuses calls once the limit is reached.
internal sealed class Breaker(int failuresBeforeOpening)
{
    private int _consecutiveFailures;

    public bool AllowsCall() => _consecutiveFailures < failuresBeforeOpening;

    public void RecordSuccess() => _consecutiveFailures = 0;

    public void RecordFailure() => _consecutiveFailures++;
}

// Output:
// call 1: failed
// call 2: failed
// call 3: failed
// call 4: rejected, circuit open
// call 5: rejected, circuit open

When to use it

  • Stopping a dead payment provider from holding every request thread in a checkout service.
  • Keeping a product page working when the recommendations service is returning 503s.
  • Capping concurrent queries against a search cluster that falls over above ten.
  • Exposing which dependency is currently broken through a health check endpoint.

More examples

Bulkhead, breaker and fallback on one HttpClient

This is the shape to use when a dependency is genuinely optional. A 204 No Content from the fallback lets the calling code render the page without the recommendations panel, so a broken dependency costs a missing panel instead of a failed page. Note the ordering: Polly executes strategies from the outside in, in the order you add them, so a fallback added last would never see BrokenCircuitException. Use AddResilienceHandler rather than stacking a second handler on top of AddStandardResilienceHandler, which the docs explicitly warn against.

Example · csharp
using System.Net;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;
using Polly;
using Polly.CircuitBreaker;
using Polly.Fallback;

ServiceCollection services = new();

services.AddHttpClient("recommendations")
    .AddResilienceHandler("recommendations", static pipeline =>
    {
        // Strategies run outermost first, in the order they are added, so
        // the fallback must come first to catch what the breaker throws.
        pipeline.AddFallback(new FallbackStrategyOptions<HttpResponseMessage>
        {
            ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
                .Handle<BrokenCircuitException>(),
            FallbackAction = static args =>
                Outcome.FromResultAsValueTask(
                    new HttpResponseMessage(HttpStatusCode.NoContent))
        });

        // Bulkhead: at most 20 requests may wait on this dependency, and
        // nothing queues behind them. The 21st is rejected straight away.
        pipeline.AddConcurrencyLimiter(20, 0);

        // Open once a quarter of at least 20 calls in 30 seconds have failed.
        pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
        {
            FailureRatio = 0.25,
            MinimumThroughput = 20,
            SamplingDuration = TimeSpan.FromSeconds(30),
            BreakDuration = TimeSpan.FromSeconds(10)
        });
    });

Watching the breaker change state

A breaker that opens silently turns into a mystery: requests start failing fast and no log says why. Wire OnOpened, OnHalfOpened and OnClosed to your logger and a counter, and the transitions become the clearest signal you have about a dependency. Expose CircuitState through a health check so an operator can see which dependency is broken without reading code. Breaking a circuit on purpose during an incident goes through a separate option, ManualControl, which takes a CircuitBreakerManualControl object; the state provider only reports state, it does not control it.

Example · csharp
using Microsoft.Extensions.Http.Resilience;
using Polly.CircuitBreaker;

// A state provider lets other code read the breaker without calling it.
CircuitBreakerStateProvider state = new();

HttpCircuitBreakerStrategyOptions options = new()
{
    FailureRatio = 0.5,
    MinimumThroughput = 10,
    SamplingDuration = TimeSpan.FromSeconds(30),
    BreakDuration = TimeSpan.FromSeconds(15),
    StateProvider = state,

    OnOpened = static args =>
    {
        // Returning default is the shorthand for a completed ValueTask.
        Console.WriteLine("payments circuit opened");
        return default;
    },
    OnHalfOpened = static args =>
    {
        Console.WriteLine("payments circuit half-open: probing with one call");
        return default;
    },
    OnClosed = static args =>
    {
        Console.WriteLine("payments circuit closed");
        return default;
    }
};

// state.CircuitState is Closed, Open, HalfOpen or Isolated, so a health
// check or a dashboard can report "payments degraded" without a test call.
Console.WriteLine(state.CircuitState);

A bulkhead by hand, for a dependency that is not HTTP

Polly's concurrency limiter only covers what goes through its pipeline. For a dependency reached some other way, a gRPC channel, a message broker, a native client library, a SemaphoreSlim does the same job in ten lines. The WaitAsync overload that takes a TimeSpan returns false instead of waiting indefinitely, which is the part that makes it a bulkhead rather than a queue. The Release call must be in a finally block; a permit lost to an exception never comes back, and the gateway slowly stops working.

Example · csharp
namespace Shop.Search;

// One semaphore per dependency, sized to what the dependency can take
// rather than to how much traffic you happen to have.
internal sealed class SearchGateway(HttpClient client) : IDisposable
{
    // The search cluster handles 10 concurrent queries from this service.
    private readonly SemaphoreSlim _slots = new(10, 10);

    // Returns null when there was no slot, so the caller can decide.
    public async Task<string?> SearchAsync(string term, CancellationToken token)
    {
        // Wait briefly for a slot, then give up rather than queue forever.
        if (!await _slots.WaitAsync(TimeSpan.FromMilliseconds(50), token))
        {
            return null;
        }

        try
        {
            using HttpResponseMessage response = await client.GetAsync(
                $"/search?q={Uri.EscapeDataString(term)}", token);
            response.EnsureSuccessStatusCode();
            return await response.Content.ReadAsStringAsync(token);
        }
        finally
        {
            // Release in a finally, or one exception leaks a permit forever.
            _slots.Release();
        }
    }

    public void Dispose() => _slots.Dispose();
}

Related

dotnet http resiliencedotnet rate limitingdotnet degradationdotnet timeouts everywheredotnet metrics

Discussion

  • Be the first to comment on this lesson.