How Distributed Systems Fail

Why a distributed system fails in parts, why a slow dependency is worse than a dead one, and how retries turn a blip into an outage.

A distributed system is one where a single user action needs more than one machine. Your service, a Postgres database, a cache and a payment provider are four machines on a network. Each one can stop answering without the others noticing.

Partial failure

A single program either runs or crashes. A distributed system does neither. One dependency disappears and everything else carries on. This is partial failure, and it is the normal condition of a running system rather than an emergency. Code written as if every call succeeds has no behaviour for it, so it falls back to the worst behaviour available: a 500 for the user.

Slow is worse than down

A dependency that refuses connections fails in about a millisecond. A dependency that accepts the connection and then goes quiet holds your request open for as long as you allow. Every held request keeps a socket, some memory and a slot in whatever limits you have. Enough of them and your own service stops answering. Failure spreading outward like that is a cascading failure.

The retry storm

Retrying one failed call is sensible. Every caller retrying at the same instant is not. A dependency that is already struggling, called by clients configured with three retries, immediately sees four times its normal load. That is a retry storm, and it is how a brief blip becomes an outage. Two things prevent it: a hard cap on attempts, and jitter, a random variation added to each wait so retries spread out instead of arriving together.

Example

// How much extra load retries put on a dependency that is already failing.
// The numbers are fixed so the result is the same on every run.

int callsPerSecond = 100;
int retriesPerCall = 3;

// While the dependency is healthy it sees one request per call.
int healthyLoad = callsPerSecond;

// While it is failing it sees the first attempt plus every retry.
int failingLoad = callsPerSecond * (1 + retriesPerCall);

Console.WriteLine($"healthy: {healthyLoad} requests/second");
Console.WriteLine($"failing: {failingLoad} requests/second");

// Now put three services in a chain, each one retrying three times.
// The multiplier applies again at every hop.
int hops = 3;
int amplified = callsPerSecond;
for (int hop = 0; hop < hops; hop++)
{
    amplified *= 1 + retriesPerCall;
}

Console.WriteLine($"after {hops} hops: {amplified} requests/second");

// Output:
// healthy: 100 requests/second
// failing: 400 requests/second
// after 3 hops: 6400 requests/second

When to use it

Explaining to a team why last week's outage began as 200ms of extra latency in one service.
Reviewing a pull request that wraps an outbound POST in a three-attempt retry loop.
Choosing which of the six outbound calls in a checkout endpoint genuinely must succeed.
Writing the first incident review for a service that returned 500s for twenty minutes.

More examples

A slow dependency exhausting a connection pool

This is the mechanism behind "slow is worse than down". The pool stands in for anything finite: HTTP connections, database connections, or a concurrency limit you set yourself. Four slow calls occupy the whole pool, and the fifth caller gets nothing even though the dependency returned no errors at all. Reach for this shape when you want to show a team why a latency increase and an outage look identical from the outside.

// A pool of four connections. SemaphoreSlim hands out a fixed number of
// permits; a caller that cannot get one has to wait.
using SemaphoreSlim pool = new(initialCount: 4, maxCount: 4);

// Four callers take every permit and hold it while the dependency is slow.
List<Task> holders = [];
for (int i = 0; i < 4; i++)
{
    holders.Add(HoldAsync(pool));
}

// The fifth caller waits, but only for a bounded time, then gives up.
bool acquired = await pool.WaitAsync(TimeSpan.FromMilliseconds(200));
Console.WriteLine(acquired ? "got a connection" : "pool exhausted, failing fast");

await Task.WhenAll(holders);

static async Task HoldAsync(SemaphoreSlim pool)
{
    await pool.WaitAsync();
    try
    {
        await Task.Delay(TimeSpan.FromSeconds(2));  // the slow dependency
    }
    finally
    {
        pool.Release();
    }
}

// Output:
// pool exhausted, failing fast

Exponential backoff, with and without jitter

Backoff alone is not enough. Without jitter, a thousand callers that all failed at the same moment retry at the same moment, so the dependency gets one spike per round instead of steady load. The standard resilience handler in Microsoft.Extensions.Http.Resilience sets UseJitter to true by default for exactly this reason. Work the schedule out on paper like this when you are choosing retry numbers for a new dependency.

// Exponential backoff doubles the wait after every failure, so a caller
// that keeps failing backs off instead of hammering the dependency.
int baseDelayMs = 1000;

for (int attempt = 1; attempt <= 4; attempt++)
{
    // 1 << n is the shorthand for multiplying by 2 n times.
    // Longhand: (int)Math.Pow(2, attempt - 1)
    int plainMs = baseDelayMs * (1 << (attempt - 1));

    // Jitter multiplies that wait by a factor so callers do not line up on
    // the same instants. A real pipeline picks the factor at random; this
    // one is fixed at three quarters so the printed numbers never change.
    int jitteredMs = plainMs * 3 / 4;

    Console.WriteLine($"attempt {attempt}: plain {plainMs}ms, jittered {jitteredMs}ms");
}

// Output:
// attempt 1: plain 1000ms, jittered 750ms
// attempt 2: plain 2000ms, jittered 1500ms
// attempt 3: plain 4000ms, jittered 3000ms
// attempt 4: plain 8000ms, jittered 6000ms

Why requiring every dependency makes the request less reliable

Each dependency here looks acceptable on its own dashboard. The endpoint that needs all four is worse than any of them, because it fails whenever any one fails. This is the argument for the next three lessons: the way to make the request more reliable is not to make each dependency better, it is to stop requiring all of them.

// Rates are given as failures per 100000 requests so the arithmetic stays
// in whole numbers and prints the same in every region.
int[] failuresPer100k = [100, 100, 100, 500];
// Longhand for the line above: new int[] { 100, 100, 100, 500 }

int requestFailures = 0;
foreach (int rate in failuresPer100k)
{
    // A request that needs all four dependencies fails if any one fails,
    // so the rates add up (near enough, at these small rates).
    requestFailures += rate;
}

Console.WriteLine($"worst single dependency: 500 failures per 100000");
Console.WriteLine($"the request itself: about {requestFailures} failures per 100000");

// Output:
// worst single dependency: 500 failures per 100000
// the request itself: about 800 failures per 100000

Discussion

Be the first to comment on this lesson.

Partial failure

Slow is worse than down

The retry storm

Example

When to use it

More examples

A slow dependency exhausting a connection pool

Exponential backoff, with and without jitter

Why requiring every dependency makes the request less reliable

Related

Discussion