Graceful Shutdown

Stop an API without dropping requests: handle SIGTERM, drain in-flight work within HostOptions.ShutdownTimeout and end long-lived requests early.

Syntaxbuilder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(20)); app.Lifetime.ApplicationStopping.Register(() => logger.LogInformation("Draining"));

Every deploy, scale-in and node drain stops your process. A graceful stop finishes the requests already in progress, refuses new ones, flushes what must be flushed and exits before the platform loses patience. The host does most of this for you if you understand its sequence and its one timer.

The stop sequence

  1. ConsoleLifetime receives SIGTERM (Kubernetes, docker stop), SIGINT or Ctrl+C, or your code calls IHostApplicationLifetime.StopApplication().
  2. ApplicationStopping fires.
  3. Hosted services stop in reverse registration order. Kestrel was registered last, so it goes first: it closes its listeners, lets in-flight requests finish and closes idle keep-alive connections.
  4. Your hosted services' StopAsync run; BackgroundService cancels its stoppingToken and waits for ExecuteAsync.
  5. ApplicationStopped fires, the container disposes singletons, Main returns.

ShutdownTimeout

HostOptions.ShutdownTimeout is 30 seconds by default and covers the whole of step 3 and 4. When it expires the token passed to every StopAsync is cancelled, Kestrel aborts the remaining connections (handlers see RequestAborted fire and clients see a reset), and the host carries on stopping. Set it in code with Configure<HostOptions>, or through host configuration with the key shutdownTimeoutSeconds (for example DOTNET_SHUTDOWNTIMEOUTSECONDS=20).

The platform's clock

The host's timer runs inside the platform's: Kubernetes sends SIGKILL when terminationGracePeriodSeconds runs out, so any preStop delay plus ShutdownTimeout must stay below the grace period. The drain delay, the budget and the rollout settings are covered in Zero-Downtime Rollouts. The signal must also reach the .NET process, which needs an exec-form ENTRYPOINT (see A Dockerfile for an ASP.NET Core API).

Long-lived requests

Kestrel waits for in-flight requests, and a long poll, a Server-Sent Events stream or a WebSocket never finishes on its own. Each one holds the shutdown open until the timeout, then is cut off. Link ApplicationStopping into the request's token so these end cleanly at the start of shutdown and the client reconnects to another instance.

Work in flightWhat to do on stop
Short requestNothing; Kestrel drains it
Long poll, SSE, WebSocketEnd on ApplicationStopping
Queue message being processedFinish it, then stop pulling; ack only after success
Buffered telemetry or audit eventsFlush in StopAsync with a bounded wait

What goes wrong in production

  • Every deploy produces a burst of 502s: the listener closed while load balancers still routed to the pod, or SIGTERM never reached the process.
  • Pods take exactly the grace period to die: an open stream or a StopAsync that ignores its token.
  • Duplicate messages after deploys: a consumer killed mid-message because the timeout was shorter than one message's work. Make handlers idempotent anyway.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Flush the last batch of order audit events in StopAsync within a 5 second cap, logging how many were lost if the budget runs out.
  • End open /orders/changes long polls with 503 at the start of shutdown so clients reconnect to another pod instead of being cut off at the timeout.
  • Find why pods always take the full ShutdownTimeout to stop: an open order-status SSE stream that was never tied to ApplicationStopping.

More examples

Flushing buffered audit events within the stop budget

The flush gets its own 5 second cap and is also linked to the token the host passes to <code>StopAsync</code>, which is cancelled when <code>ShutdownTimeout</code> expires. Either way the method returns, the loss is logged, and the process exits on its own instead of being killed with nothing written. In a web app, register the flusher before other hosted services: services stop in reverse order, so it runs after the request drain and catches the events those last requests produced.

Example · csharp
var builder = Host.CreateApplicationBuilder(args);
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(20));
builder.Services.AddSingleton<AuditBuffer>();
builder.Services.AddHostedService<AuditFlusher>();
using var host = builder.Build();
await host.RunAsync();

record AuditEvent(string OrderId, string Action, DateTimeOffset At);

sealed class AuditBuffer
{
    private readonly List<AuditEvent> _events = new();
    private readonly Lock _gate = new();

    public void Add(AuditEvent auditEvent) { lock (_gate) _events.Add(auditEvent); }

    public List<AuditEvent> Drain()
    {
        lock (_gate)
        {
            var batch = new List<AuditEvent>(_events);
            _events.Clear();
            return batch;
        }
    }
}

sealed class AuditFlusher(AuditBuffer buffer, ILogger<AuditFlusher> logger) : IHostedService
{
    public Task StartAsync(CancellationToken cancellationToken) => Task.CompletedTask;

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        // Wrong: WriteAsync(batch, CancellationToken.None) can outlive the grace period and end in SIGKILL.
        // Right: a bounded wait that also honours the host's own token.
        using var budget = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
        budget.CancelAfter(TimeSpan.FromSeconds(5));
        var batch = buffer.Drain();
        try
        {
            await WriteAsync(batch, budget.Token);
            logger.LogInformation("Flushed {Count} audit events", batch.Count);
        }
        catch (OperationCanceledException)
        {
            logger.LogWarning("Stop budget spent; {Count} audit events were not flushed", batch.Count);
        }
    }

    private static Task WriteAsync(IReadOnlyList<AuditEvent> batch, CancellationToken ct) =>
        Task.Delay(TimeSpan.FromMilliseconds(20 * batch.Count), ct);
}

Server-Sent Events that end when shutdown starts

A stream bound only to <code>RequestAborted</code> keeps Kestrel's drain waiting until <code>ShutdownTimeout</code>, and then the connection is reset. Linking <code>ApplicationStopping</code> makes the generator finish normally as soon as the stop begins, so the response completes cleanly, the drain is not held open, and the browser's <code>EventSource</code> reconnects to an instance that is still running.

Example · csharp
using System.Net.ServerSentEvents;
using System.Runtime.CompilerServices;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/orders/{id:int}/status-stream", (int id, HttpContext context, IHostApplicationLifetime lifetime) =>
{
    // Ends the stream on client disconnect OR at the start of shutdown.
    var stop = CancellationTokenSource.CreateLinkedTokenSource(context.RequestAborted, lifetime.ApplicationStopping);
    context.Response.RegisterForDispose(stop);
    return TypedResults.ServerSentEvents(StatusUpdates(id, stop.Token));
});

app.Run();

static async IAsyncEnumerable<SseItem<OrderStatus>> StatusUpdates(int orderId, [EnumeratorCancellation] CancellationToken ct)
{
    string[] statuses = ["Paid", "Picking", "Packed", "Shipped"];
    foreach (var status in statuses)
    {
        yield return new SseItem<OrderStatus>(new OrderStatus(orderId, status), "status");
        if (!await WaitAsync(TimeSpan.FromSeconds(15), ct))
            yield break;
    }
}

static async Task<bool> WaitAsync(TimeSpan delay, CancellationToken ct)
{
    try { await Task.Delay(delay, ct); return true; }
    catch (OperationCanceledException) { return false; }
}

record OrderStatus(int OrderId, string Status);

A consumer that finishes the current message

Passing <code>stoppingToken</code> straight into the send would abandon a half-sent receipt the moment shutdown begins. Here the token only interrupts the wait for new work; the current message finishes within its own 10 second budget, which must fit inside <code>ShutdownTimeout</code>. The <code>Channel</code> stands in for a broker: an in-memory channel loses whatever is still buffered at shutdown, so real at-least-once delivery needs a broker that redelivers unacknowledged messages, or a durable outbox.

Example · csharp
using System.Threading.Channels;

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddSingleton(Channel.CreateBounded<PaymentCaptured>(500));
builder.Services.AddHostedService<ReceiptSender>();
using var host = builder.Build();
await host.RunAsync();

record PaymentCaptured(Guid PaymentId, string CustomerEmail, decimal Amount);

sealed class ReceiptSender(Channel<PaymentCaptured> queue, ILogger<ReceiptSender> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // stoppingToken only stops us waiting for the NEXT message.
        while (await queue.Reader.WaitToReadAsync(stoppingToken))
        {
            while (queue.Reader.TryRead(out var payment))
            {
                // The message in hand gets its own bounded budget instead of being abandoned.
                using var work = new CancellationTokenSource(TimeSpan.FromSeconds(10));
                await SendReceiptAsync(payment, work.Token);
                logger.LogInformation("Receipt sent for {PaymentId}", payment.PaymentId);
                if (stoppingToken.IsCancellationRequested) return;
            }
        }
    }

    private static Task SendReceiptAsync(PaymentCaptured payment, CancellationToken ct) =>
        Task.Delay(TimeSpan.FromMilliseconds(150), ct);
}

Related

net zero downtimenet key hostoptions shutdownnet hosted servicesnet server sent eventsnet dockerfile

Discussion

  • Be the first to comment on this lesson.