Graceful Shutdown

SIGTERM starts the host's stopping sequence; graceful shutdown means draining in-flight work and leaving nothing half-done.

Syntaxbuilder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(45));

Your process will be stopped. A deploy, a scale-down, a node drain. On Linux the stop begins as SIGTERM, a signal that means "please exit". Handling it well is called graceful shutdown: stop taking new work, finish or safely return what you are holding, then exit.

The host's stopping sequence

The host handles SIGTERM, SIGINT (Ctrl+C) and SIGQUIT for you. In order, it signals IHostApplicationLifetime.ApplicationStopping, then calls StopAsync on each hosted service, which is what cancels a BackgroundService stopping token, then signals ApplicationStopped. Kestrel, the web server, closes its listening ports so no new connection is accepted, and gives requests already in progress until a timeout to finish. That timeout is HostOptions.ShutdownTimeout and the default is 30 seconds. When it expires, the remaining requests are aborted.

app.Run() is the blocking form of await app.RunAsync(). Both start the host and then wait for a shutdown signal before returning, and they are the same thing.

Work in progress

Draining is not cancelling. Do not pass the stopping token into the item you are already handling; that aborts it halfway and is exactly what graceful shutdown is meant to avoid. Use the token to decide not to start the next item, and let the current one finish under its own timeout.

Anything you have taken off a queue but not completed has to go back, or be left uncommitted so the broker redelivers it. Two settings must also agree: Kubernetes kills the container with SIGKILL after terminationGracePeriodSeconds, which defaults to 30, so a ShutdownTimeout longer than that is never reached.

Example

Example · csharp
// Two items are in flight when the stop signal arrives, and one has not started.
using CancellationTokenSource stopping = new();
List<string> inFlight = ["order-1", "order-2"];
List<string> notStarted = ["order-3"];

stopping.Cancel();   // stands in for SIGTERM reaching the process

// First rule: stop taking new work.
if (stopping.IsCancellationRequested)
{
    Console.WriteLine($"refusing new work, {notStarted.Count} item requeued");
}

// Second rule: finish what is already running. The stopping token is
// deliberately NOT passed here. Cancelling in-flight work is the opposite of
// draining it.
foreach (string order in inFlight)
{
    await Task.Delay(1);
    Console.WriteLine($"finished {order}");
}

Console.WriteLine("exiting cleanly");

// Output:
// refusing new work, 1 item requeued
// finished order-1
// finished order-2
// exiting cleanly

When to use it

  • Deploy a new version without returning errors to requests that were already in progress.
  • Let a scaled-down replica finish the order it is fulfilling instead of leaving it half-applied.
  • Make a readiness probe fail the moment shutdown starts so the load balancer stops sending traffic.
  • Put a claimed queue item back when the pod is evicted mid-job.

More examples

Setting the timeout and watching the lifetime events

Press Ctrl+C and the log shows Draining, then Application is shutting down, then Stopped, and GET / stops being answered in between. Register on ApplicationStopping when something outside the app needs to know you are going away, most often to make a readiness probe start failing so the load balancer stops sending you traffic.

Example · csharp
WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

builder.Services.Configure<HostOptions>(options =>
{
    // How long in-flight requests and hosted services get to finish.
    // The default is 30 seconds. Keep it below terminationGracePeriodSeconds.
    options.ShutdownTimeout = TimeSpan.FromSeconds(20);
});

WebApplication app = builder.Build();

IHostApplicationLifetime lifetime = app.Lifetime;

// Fires as soon as the signal arrives, before requests are drained. This is
// where you flip a flag that the readiness probe reads.
lifetime.ApplicationStopping.Register(() => app.Logger.LogInformation("Draining"));

// Fires once every hosted service has stopped.
lifetime.ApplicationStopped.Register(() => app.Logger.LogInformation("Stopped"));

app.MapGet("/healthz/ready", () => Results.Ok("ready"));
app.MapGet("/", () => "ok");

app.Run();

A worker that gives back the item it was holding

The two cancellation tokens are the whole lesson here. The stopping token controls whether to take the next item; a separate per-item token bounds the current one. Use this shape for any worker whose unit of work must not be left half-applied, and note that the requeue call is given CancellationToken.None so shutdown cannot cancel the cleanup itself.

Example · csharp
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using System.Threading.Channels;

namespace Shop;

public sealed class OrderWorker(
    Channel<string> queue,
    IOrderStore store,
    ILogger<OrderWorker> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {
            while (await queue.Reader.WaitToReadAsync(stoppingToken))
            {
                while (queue.Reader.TryRead(out string? order))
                {
                    try
                    {
                        // Its own timeout, not the stopping token, so a running
                        // order is finished rather than cut in half.
                        using CancellationTokenSource perItem = new(TimeSpan.FromSeconds(10));
                        await store.FulfilAsync(order, perItem.Token);
                    }
                    catch (Exception ex)
                    {
                        logger.LogError(ex, "Order {Order} failed, requeueing", order);
                        await store.RequeueAsync(order, CancellationToken.None);
                    }

                    if (stoppingToken.IsCancellationRequested)
                    {
                        logger.LogInformation("Shutting down, taking no more orders");
                        return;
                    }
                }
            }
        }
        catch (OperationCanceledException)
        {
            // The wait was cancelled by shutdown. Nothing was in hand.
        }
    }
}

public interface IOrderStore
{
    Task FulfilAsync(string order, CancellationToken cancellationToken);
    Task RequeueAsync(string order, CancellationToken cancellationToken);
}

The Kubernetes side of the same shutdown

Graceful shutdown is a contract between the application and whatever stops it, and getting one side right is not enough. The preStop command runs before SIGTERM is sent, which buys the time Kubernetes does not otherwise give you. A terminationGracePeriodSeconds shorter than ShutdownTimeout means your drain is cut short by SIGKILL and the setting in code does nothing.

Example · yaml
spec:
  # SIGKILL arrives this long after SIGTERM, so it must exceed ShutdownTimeout.
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      image: registry.example.com/shop/api:1.4.0
      lifecycle:
        preStop:
          exec:
            # Endpoint removal and SIGTERM happen at the same moment, so wait
            # for the load balancers to notice before the app starts stopping.
            command: ["/bin/sleep", "10"]
      readinessProbe:
        httpGet:
          path: /healthz/ready
          port: 8080
        periodSeconds: 2

Related

dotnet background servicedotnet graceful cancellationdotnet k8s probesdotnet kestreldotnet queues jobs

Discussion

  • Be the first to comment on this lesson.