Proving It

Test the timeout and the retry, inject failure on purpose in a safe environment, and attach a runbook to every alert.

Syntaxpipeline.AddChaosLatency(new ChaosLatencyStrategyOptions { InjectionRate = 0.05, Latency = TimeSpan.FromSeconds(5) });

Resilience code only runs when something is broken, which is exactly when nobody is watching it. A timeout that was never tested, a retry that silently retries zero times, a fallback that throws on its own: all of these look fine in production until the day they matter. So this behaviour has to be exercised on purpose.

Test the timeout and the retry

Both are testable with no network at all. HttpMessageHandler is the abstract class HttpClient sends every request through, and you can write your own. A handler that never answers proves the timeout fires. A handler that counts calls and fails twice proves the retry runs exactly three times. Set retry delays to TimeSpan.Zero in tests so the suite does not spend real seconds waiting.

Inject failure on purpose

Chaos engineering means causing failures yourself, in a controlled place, so you learn how the system behaves before a real failure teaches you. Polly ships chaos strategies: AddChaosLatency adds delay, AddChaosFault throws an exception, AddChaosOutcome returns a bad result. Each takes an InjectionRate between 0 and 1. Register them only outside production, and place them innermost in the pipeline so your own retry and timeout get a chance to react.

The runbook

A runbook is a short written procedure attached to an alert. Without one, an alert tells somebody at 03:00 that something is wrong and nothing more. A useful one names the symptom, the dashboard, the exact check to run, the action to take, and when to escalate.

Example

// A fake dependency that fails a fixed number of times and then succeeds.
// Fixed, not random, so the result is the same on every run.
FlakyDependency dependency = new(failuresBeforeSuccess: 2);

string result = await WithRetryAsync(dependency.GetAsync, maxAttempts: 3);

Console.WriteLine($"result: {result}");
Console.WriteLine($"calls made: {dependency.CallCount}");

// Calls the operation again after a failure, up to maxAttempts times.
static async Task<string> WithRetryAsync(Func<Task<string>> operation, int maxAttempts)
{
    for (int attempt = 1; ; attempt++)
    {
        try
        {
            return await operation();
        }
        catch (HttpRequestException) when (attempt < maxAttempts)
        {
            // The filter is what keeps this honest: the last failure is not
            // swallowed, it is thrown to the caller.
            await Task.Delay(TimeSpan.FromMilliseconds(10));
        }
    }
}

internal sealed class FlakyDependency(int failuresBeforeSuccess)
{
    public int CallCount { get; private set; }

    public Task<string> GetAsync()
    {
        CallCount++;
        return CallCount <= failuresBeforeSuccess
            ? Task.FromException<string>(new HttpRequestException("injected failure"))
            : Task.FromResult("ok");
    }
}

// Output:
// result: ok
// calls made: 3

When to use it

Proving a newly configured 500ms timeout actually fires before shipping it.
Checking that a retry policy retries the number of times you think it does.
Running a staging load test with 5% of calls artificially slowed to find the breaking point.
Writing the runbook entry for a new "payments circuit open" alert.

More examples

A test that proves the timeout fires

This test takes 200 milliseconds and needs no server. The second assertion is the valuable one: it pins down the exception shape that your logging and metrics code relies on, so if someone later wraps the call differently and loses the inner TimeoutException, the test fails rather than the dashboard quietly going wrong. Write one of these for every timeout you configure; a timeout that was never observed firing is an assumption, not a behaviour.

using System.Net;
using Xunit;

namespace Shop.Tests;

// Never answers until the token is cancelled. HttpMessageHandler is the
// abstract class HttpClient sends requests through, so replacing it
// replaces the network.
internal sealed class HangingHandler : HttpMessageHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        await Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken);
        return new HttpResponseMessage(HttpStatusCode.OK);
    }
}

public class TimeoutTests
{
    [Fact]
    public async Task Request_gives_up_after_the_configured_timeout()
    {
        using HttpClient client = new(new HangingHandler())
        {
            BaseAddress = new Uri("https://rates.test"),
            Timeout = TimeSpan.FromMilliseconds(200)
        };

        TaskCanceledException error = await Assert.ThrowsAsync<TaskCanceledException>(
            () => client.GetAsync("/rates/GBP"));

        // A timeout carries an inner TimeoutException. A caller who went
        // away does not, which is how production code tells them apart.
        Assert.IsType<TimeoutException>(error.InnerException);
    }
}

Injecting latency, in staging only

Run load against staging with this in place and you find out, cheaply, what 5% slow calls does to your p99 latency, your thread count and your error rate. The position in the pipeline is the whole point: chaos added outermost would be caught by nothing, while chaos added innermost sits where the dependency really is. Start with a low InjectionRate and a latency just past your attempt timeout, because that is the case most likely to be mishandled. Keep the environment check in code rather than configuration; a mistyped setting should not be able to turn this on in production.

using Microsoft.Extensions.Http.Resilience;
using Polly;
using Polly.Simmy;
using Polly.Simmy.Latency;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

// Decided once, from the environment, not from a config flag someone can
// copy into production by accident.
bool injectChaos = builder.Environment.IsStaging();

builder.Services.AddHttpClient("rates")
    .AddResilienceHandler("rates", pipeline =>
    {
        pipeline.AddRetry(new HttpRetryStrategyOptions
        {
            MaxRetryAttempts = 2,
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true
        });

        pipeline.AddTimeout(TimeSpan.FromSeconds(2));

        if (injectChaos)
        {
            // Innermost, so the delay happens on the real call and the
            // timeout and retry above it are the things being tested.
            // One call in twenty waits five seconds, well past the timeout.
            pipeline.AddChaosLatency(new ChaosLatencyStrategyOptions
            {
                InjectionRate = 0.05,
                Latency = TimeSpan.FromSeconds(5)
            });
        }
    });

WebApplication app = builder.Build();
app.Run();

A test that counts the attempts

The assertion on Attempts is what makes this a test rather than a demonstration. A retry strategy whose ShouldHandle predicate does not match your dependency's failures still returns a successful-looking configuration and retries nothing, and only the call count exposes that. HttpRetryStrategyOptions handles 5xx, 408 and 429 by default, which is why 503 is retried here with no predicate written. Setting Delay to TimeSpan.Zero with UseJitter off keeps the test fast and repeatable; a test that waits out a real exponential backoff is the kind of slow, occasionally-failing test a team eventually deletes.

using System.Net;
using Microsoft.Extensions.Http.Resilience;
using Polly;
using Xunit;

namespace Shop.Tests;

// Fails twice with 503, then succeeds, and records how often it was called.
internal sealed class CountingHandler : HttpMessageHandler
{
    public int Attempts { get; private set; }

    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        Attempts++;
        return Task.FromResult(new HttpResponseMessage(
            Attempts < 3 ? HttpStatusCode.ServiceUnavailable : HttpStatusCode.OK));
    }
}

public class RetryTests
{
    [Fact]
    public async Task Two_failures_then_success_costs_three_attempts()
    {
        CountingHandler handler = new();

        ResiliencePipeline<HttpResponseMessage> pipeline =
            new ResiliencePipelineBuilder<HttpResponseMessage>()
                .AddRetry(new HttpRetryStrategyOptions
                {
                    MaxRetryAttempts = 3,
                    BackoffType = DelayBackoffType.Constant,
                    Delay = TimeSpan.Zero,   // no real waiting in a unit test
                    UseJitter = false
                })
                .Build();

        using HttpClient client = new(handler)
        {
            BaseAddress = new Uri("https://rates.test")
        };

        HttpResponseMessage response = await pipeline.ExecuteAsync(
            async token => await client.GetAsync("/rates/GBP", token));

        Assert.Equal(HttpStatusCode.OK, response.StatusCode);
        Assert.Equal(3, handler.Attempts);
    }
}

Discussion

Be the first to comment on this lesson.

Test the timeout and the retry

Inject failure on purpose

The runbook

Example

When to use it

More examples

A test that proves the timeout fires

Injecting latency, in staging only

A test that counts the attempts

Related

Discussion