Graceful Degradation

Decide in advance what the service still does without each dependency, serve stale data on purpose, and label it when you do.

Syntaxstring value = await TryFreshAsync(key) ?? lastKnownGood[key];

Graceful degradation means deciding ahead of time what your service still does when a dependency is unavailable, and writing that decision into the code. There is no option to skip this. If you do not choose, the behaviour is chosen for you by whichever exception escapes first, and it is almost always a 500.

Sort the dependencies first

Walk through every outbound call an endpoint makes and label it. Required: no useful answer exists without it, so return an error. Stale allowed: an older value is better than nothing. Optional: leave that part out of the response and carry on. A product page needs the product. It does not need the recommendations panel, and it can live with an exchange rate from a minute ago.

Serving stale on purpose

Keep two cached copies of anything you might serve stale: a short-lived fresh copy, and a long-lived last-known-good copy that is only read when the live call fails. Stale here means a value you know is out of date and are returning anyway. That is sound for a rate on a dashboard. It is wrong for an account balance, a stock level you are about to sell against, or a permission check.

Say so

A degraded response must be visibly degraded. Put a flag in the response body, log a warning naming the dependency, and increment a counter so a dashboard shows it. Silent degradation looks exactly like a calculation bug to whoever investigates next.

Example

Example · csharp
// The last good value for each currency, kept so the service can still
// answer when the rates dependency is unavailable.
Dictionary<string, string> lastKnownRates = new()
{
    ["GBP"] = "1.27",
    ["EUR"] = "1.09"
};

string[] codes = ["GBP", "JPY"];
// Longhand for the line above: string[] codes = new string[] { "GBP", "JPY" };

foreach (string code in codes)
{
    string? fresh = await TryGetRateAsync(code);

    if (fresh is not null)
    {
        lastKnownRates[code] = fresh;
        Console.WriteLine($"{code}: {fresh} (live)");
    }
    else if (lastKnownRates.TryGetValue(code, out string? stale))
    {
        // A deliberate choice: on this screen an old rate beats an error.
        Console.WriteLine($"{code}: {stale} (stale)");
    }
    else
    {
        // Nothing cached, so this one value is genuinely unavailable.
        // Note that the rest of the loop still runs.
        Console.WriteLine($"{code}: unavailable");
    }
}

// The rates service is down for this run, so every call returns null.
static Task<string?> TryGetRateAsync(string currency)
{
    return Task.FromResult<string?>(null);
}

// Output:
// GBP: 1.27 (stale)
// JPY: unavailable

When to use it

  • Keeping a product page serving when the reviews service is down.
  • Deciding whether a cached exchange rate from ten minutes ago may be shown on an invoice.
  • Adding a degraded flag to an API response so the client can show a staleness warning.
  • Writing down, before launch, which of an endpoint's dependencies are genuinely required.

More examples

Two cache entries: one fresh, one last-known-good

One cache entry cannot do this job: a short expiry gives you no fallback, and a long expiry means normal traffic reads old data. Two entries separate the two questions, "is this current enough to serve normally" and "is there anything at all to serve in an emergency". The six-hour backup expiry is itself the policy decision; after six hours the service reports unavailable rather than quoting a rate from yesterday. The returned tuple carries IsStale so the caller cannot accidentally present stale data as live.

Example · csharp
using Microsoft.Extensions.Caching.Distributed;
using Microsoft.Extensions.Logging;

namespace Shop.Pricing;

// IRatesClient is the project's own HTTP client interface; it throws
// HttpRequestException when the rates service cannot be reached.
internal sealed class RateCache(
    IDistributedCache cache,
    IRatesClient client,
    ILogger<RateCache> logger)
{
    // Fresh values expire quickly so normal traffic stays up to date.
    private static readonly DistributedCacheEntryOptions Fresh = new()
    {
        AbsoluteExpirationRelativeToNow = TimeSpan.FromSeconds(30)
    };

    // The backup copy lives far longer and is only read on a failure.
    private static readonly DistributedCacheEntryOptions Backup = new()
    {
        AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(6)
    };

    public async Task<(string Rate, bool IsStale)?> GetRateAsync(
        string currency, CancellationToken token)
    {
        string? fresh = await cache.GetStringAsync($"rate:fresh:{currency}", token);
        if (fresh is not null)
        {
            return (fresh, false);
        }

        try
        {
            string rate = await client.GetRateAsync(currency, token);
            await cache.SetStringAsync($"rate:fresh:{currency}", rate, Fresh, token);
            await cache.SetStringAsync($"rate:backup:{currency}", rate, Backup, token);
            return (rate, false);
        }
        catch (HttpRequestException ex)
        {
            string? backup = await cache.GetStringAsync($"rate:backup:{currency}", token);
            if (backup is null)
            {
                return null;
            }

            logger.LogWarning(ex, "Serving stale rate for {Currency}", currency);
            return (backup, true);
        }
    }
}

An endpoint that reports its own degradation

Send a request with curl to http://localhost:5000/dashboard/GBP. While the rates service is healthy the response is 200 with a body of {"currency":"GBP","rate":"1.27","degraded":false}. While it is down and a backup exists, the status is still 200 but degraded is true, so the page can show the value with a "last updated" marker instead of a blank. With no backup at all, rate is null and degraded is true. The client gets one response shape to handle in all three cases, and nobody has to guess whether a number is current.

Example · csharp
app.MapGet("/dashboard/{currency}", async (
    string currency,
    RateCache rates,
    CancellationToken token) =>
{
    (string Rate, bool IsStale)? result = await rates.GetRateAsync(currency, token);

    // Null means nothing was available, not that the request was bad, so
    // the response is still 200 with the field left empty.
    return result is null
        ? Results.Ok(new { currency, rate = (string?)null, degraded = true })
        : Results.Ok(new { currency, rate = result.Value.Rate, degraded = result.Value.IsStale });
});

The degradation policy, written down in code

This table is the deliverable of the lesson. It turns an argument that otherwise happens during an incident into a code review, and it gives the resilience configuration a reason: Optional dependencies get a fallback, StaleAllowed ones get a last-known-good cache, Required ones get a tight timeout and an honest error. The default arm matters, because treating an unlisted dependency as Required fails loudly when somebody adds a call and forgets to classify it, instead of silently dropping data from responses.

Example · csharp
namespace Shop.Pricing;

// What each dependency is allowed to do to a response, decided once and
// kept next to the code rather than in somebody's head.
internal enum Criticality
{
    Required,      // no useful response exists without this dependency
    StaleAllowed,  // an out-of-date value is acceptable
    Optional       // leave this part out of the response entirely
}

internal static class DegradationPolicy
{
    // A switch expression. The longhand is a switch statement whose cases
    // each return; the two compile to the same thing.
    public static Criticality For(string dependency) => dependency switch
    {
        "catalog" => Criticality.Required,
        "inventory" => Criticality.Required,
        "exchange-rates" => Criticality.StaleAllowed,
        "recommendations" => Criticality.Optional,
        "reviews" => Criticality.Optional,
        _ => Criticality.Required  // anything unlisted is treated as required
    };
}

Related

dotnet memory cachedotnet hybrid cachedotnet distributed cachedotnet circuit breakerdotnet structured logging

Discussion

  • Be the first to comment on this lesson.