Retries and Circuit Breakers

Add the standard resilience handler for retries with jittered backoff and a circuit breaker, and know which calls are safe to repeat.

Syntaxservices.AddHttpClient("name").AddStandardResilienceHandler(options => { ... });

Networks drop packets and services restart. A call that fails for that kind of reason, a transient failure, often succeeds a moment later. The Microsoft.Extensions.Http.Resilience package adds that second attempt for you. Call AddStandardResilienceHandler() on the builder that AddHttpClient returned and every request through that client goes through a pipeline of five strategies.

What the standard handler does

From outermost to innermost: a rate limiter capping concurrent requests, a total timeout of 30 seconds for the whole operation, a retry, a circuit breaker, then a 10-second timeout on each individual attempt. The retry makes up to three extra attempts with exponential backoff — each wait roughly double the last — plus jitter, a small random offset. Jitter matters: without it, every instance that failed at the same moment retries at the same moment and hits the recovering service together.

The circuit breaker

Retrying a service that is genuinely down only adds load. The breaker watches a 30-second window and, once 10% of at least 100 calls have failed, opens: for the next 5 seconds it rejects calls immediately with BrokenCircuitException instead of trying them. Then it lets one through to test the water.

What is safe to retry

By default the handler retries HTTP 408, 429, any 5xx, HttpRequestException and a timeout — but it retries them for every HTTP method. A repeated GET is harmless. A repeated POST can charge a card twice, because the first attempt may well have succeeded and only the response got lost. Call options.Retry.DisableForUnsafeHttpMethods() unless the endpoint is idempotent.

Example

// dotnet add package Microsoft.Extensions.Http.Resilience
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;

ServiceCollection services = new();

services.AddHttpClient("catalog", client =>
{
    client.BaseAddress = new Uri("https://api.example.com/");
})
.AddStandardResilienceHandler(options =>
{
    // Two extra attempts, the first after about 1 second, the next after about 2.
    options.Retry.MaxRetryAttempts = 2;
    options.Retry.Delay = TimeSpan.FromSeconds(1);

    // Never repeat POST, PATCH, PUT, DELETE or CONNECT.
    options.Retry.DisableForUnsafeHttpMethods();

    // Open the breaker once a fifth of the sampled calls fail.
    options.CircuitBreaker.FailureRatio = 0.2;
    options.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(10);
});

// The values you get when you pass no delegate at all.
HttpStandardResilienceOptions defaults = new();
Console.WriteLine($"retries: {defaults.Retry.MaxRetryAttempts}");
Console.WriteLine($"backoff: {defaults.Retry.BackoffType}");
Console.WriteLine($"jitter: {defaults.Retry.UseJitter}");
Console.WriteLine($"total timeout: {defaults.TotalRequestTimeout.Timeout}");
Console.WriteLine($"attempt timeout: {defaults.AttemptTimeout.Timeout}");

// Output:
// retries: 3
// backoff: Exponential
// jitter: True
// total timeout: 00:00:30
// attempt timeout: 00:00:10

When to use it

Stop a deploy of a downstream service from turning into errors on your own API.
Back off politely from a rate-limited third-party API that sends Retry-After.
Fail fast while a dependency is down instead of queueing requests behind 30-second timeouts.
Make a payment POST safe to retry by sending an idempotency key.

More examples

The standard handler and the longhand it stands for

Reach for AddResilienceHandler only when the standard pipeline genuinely does not fit, for example when you want no rate limiter at all. Add exactly one resilience handler per client: stacking two means a retry inside a retry, and nine attempts where you meant three. Note that the timeout strategy throws Polly's TimeoutRejectedException, not TimeoutException, which matters if you write your own ShouldHandle.

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;
using Polly;

ServiceCollection services = new();

// Shorthand: one line, sensible defaults, the order already chosen for you.
services.AddHttpClient("short").AddStandardResilienceHandler();

// Longhand: build the pipeline yourself. Strategies run in the order added,
// outermost first, so retry wraps the breaker and the breaker wraps the timeout.
services.AddHttpClient("long").AddResilienceHandler("custom", builder =>
{
    builder.AddRetry(new HttpRetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true
    });

    builder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
    {
        FailureRatio = 0.1,
        MinimumThroughput = 100,
        SamplingDuration = TimeSpan.FromSeconds(30),
        BreakDuration = TimeSpan.FromSeconds(5)
    });

    builder.AddTimeout(TimeSpan.FromSeconds(10));
});

Deciding for yourself which responses to retry

Use this when the service you call has its own conventions. ShouldHandle is a predicate over the outcome, so Outcome.Result is the HttpResponseMessage when one came back and null when an exception was thrown instead. ShouldRetryAfterHeader is the polite option against a rate-limited API: the server has told you when to come back, so waiting exactly that long beats an exponential guess.

using System.Net;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;

ServiceCollection services = new();

services.AddHttpClient("search").AddStandardResilienceHandler(options =>
{
    // Replace the default predicate: only 429 and 503 are worth repeating here.
    // A 500 from this service means a bad query, and repeating it wastes time.
    options.Retry.ShouldHandle = args => ValueTask.FromResult(
        args.Outcome.Result?.StatusCode is HttpStatusCode.TooManyRequests
            or HttpStatusCode.ServiceUnavailable);

    // When the service sends a Retry-After header, obey it instead of guessing.
    options.Retry.ShouldRetryAfterHeader = true;
});

Retries only help if the call is idempotent

This is the reason the safe-method rule exists and the way out of it. Once the server promises that two requests carrying the same Idempotency-Key produce one charge, retrying a POST stops being dangerous and you can leave retries enabled for it. Generate the key once, before the first attempt, and reuse it; generating one per attempt defeats the whole mechanism.

using System.Net.Http.Json;

public sealed class PaymentsClient(HttpClient http)
{
    // The caller supplies a key that identifies this attempt. If a retry sends
    // the same request twice, the server recognises the key and does not
    // charge the card again. That makes the POST safe to repeat.
    public async Task<Receipt?> ChargeAsync(Guid idempotencyKey, decimal amount)
    {
        using HttpRequestMessage request = new(HttpMethod.Post, "charges")
        {
            Content = JsonContent.Create(new { amount })
        };

        request.Headers.Add("Idempotency-Key", idempotencyKey.ToString());

        using HttpResponseMessage response = await http.SendAsync(request);
        response.EnsureSuccessStatusCode();

        return await response.Content.ReadFromJsonAsync<Receipt>();
    }
}

public record Receipt(string Id, decimal Amount);

Discussion

Be the first to comment on this lesson.

What the standard handler does

The circuit breaker

What is safe to retry

Example

When to use it

More examples

The standard handler and the longhand it stands for

Deciding for yourself which responses to retry

Retries only help if the call is idempotent

Related

Discussion