Zero-Downtime Rollouts

Roll out a new version of an ASP.NET Core API with no failed requests by lining up readiness, a drain delay, ShutdownTimeout and the grace period.

Syntax

terminationGracePeriodSeconds: 40
lifecycle:
  preStop:
    sleep: { seconds: 10 }
# Program.cs
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(20));

A rolling update starts new pods, waits for them to become ready, then terminates old ones. Requests fail during that process for one main reason: Kubernetes removes a terminating pod from load balancing at the same time as it signals the pod to stop, and the removal reaches kube-proxy, ingress controllers and cloud load balancers seconds later.

What happens when a pod is terminated

The pod is marked Terminating. In parallel, the EndpointSlice controller marks it not ready and the kubelet runs the preStop hook, then sends SIGTERM.
kube-proxy rules, the ingress controller's upstream list and any external load balancer catch up asynchronously, typically within a few seconds.
On SIGTERM the host begins StopAsync: ApplicationStopping fires, Kestrel stops accepting connections, closes idle keep-alive connections and waits for in-flight requests until HostOptions.ShutdownTimeout (default 30 seconds).
terminationGracePeriodSeconds (default 30) after termination began, counting the preStop hook, the kubelet sends SIGKILL.

Without a delay, step 3 closes the listener while step 2 still routes new connections to the pod: clients see connection refused, or 502 and 503 from the ingress.

The drain delay

Keep serving for a few seconds after termination starts, while reporting not ready. Two ways:

Option	Works with	Notes
`lifecycle.preStop.sleep.seconds`	Any image, current Kubernetes	No binary needed; the app does not see SIGTERM until it ends
`preStop.exec` running `sleep`	Images with a shell	Fails on chiseled images: there is no `sleep`
In-app delay in `IHostedLifecycleService.StoppingAsync`	Any image and platform	Runs before Kestrel stops; counts against `ShutdownTimeout`

The budget

Drain delay + time for the slowest normal request + hosted-service shutdown must fit inside the grace period with a margin. A typical API: delay 10 s, ShutdownTimeout 20 s, terminationGracePeriodSeconds: 40. Leave both at 30 with a preStop delay and the kubelet kills the process mid-drain.

The rollout itself

maxUnavailable: 0 and maxSurge: 1 (or 25%): capacity never drops below the replica count.
A real readiness probe gates each step: a new pod that cannot reach the database never receives traffic, and the rollout stalls instead of replacing healthy pods. Set progressDeadlineSeconds so the stall is reported.
minReadySeconds keeps a new pod under watch for a while before the next old pod goes.
A PodDisruptionBudget protects the same guarantees during node drains, which do not follow the Deployment strategy.

Long-lived connections

HTTP/1.1 keep-alive clients learn about the shutdown when Kestrel closes the idle connection; HTTP/2 and gRPC clients receive GOAWAY. SSE streams, WebSockets and gRPC streams never finish on their own, so end them on ApplicationStopping or they hold the pod for the full timeout and are then cut off.

Example

Loading editor…

Press Run to execute the code.

When to use it

Deploy the orders API ten times a day through an NGINX ingress without a single 502 in the access logs.
Drain a node for a Kubernetes upgrade while a PodDisruptionBudget keeps at least three of four API pods serving.
Run a 10-second drain delay on a chiseled image, which has no sleep binary for a preStop exec hook.

More examples

Rollout strategy, drain delay, grace period and PDB

The native <code>sleep</code> action needs no binary in the image, so it works on chiseled images where <code>exec: ["sleep", "10"]</code> fails. <code>DOTNET_SHUTDOWNTIMEOUTSECONDS</code> becomes the host key <code>shutdownTimeoutSeconds</code>, the only configuration key <code>HostOptions</code> reads; a <code>HostOptions__ShutdownTimeout</code> variable is silently ignored and leaves the 30 s default. Budget: 10 s delay plus 20 s shutdown timeout fits inside 40 s. The PodDisruptionBudget stops a node drain from evicting more than one of the four pods at a time.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 4
  minReadySeconds: 10
  progressDeadlineSeconds: 300
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      terminationGracePeriodSeconds: 40
      containers:
        - name: orders-api
          image: registry.example.com/shop/orders-api:1.5.0
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: DOTNET_SHUTDOWNTIMEOUTSECONDS
              value: "20"
          lifecycle:
            preStop:
              sleep:
                seconds: 10
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            periodSeconds: 5
            failureThreshold: 2
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: orders-api

The drain delay as reusable configuration

Use this when the platform cannot run a preStop hook, for example a non-Kubernetes container service, or when you want the delay to travel with the app. Do not combine it with a preStop sleep: the delays add up and eat the grace period. The delay counts against <code>ShutdownTimeout</code>, so keep it well below 20 seconds.

var builder = WebApplication.CreateBuilder(args);

builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(20));
builder.Services.AddSingleton<ShutdownGate>();
builder.Services.AddHostedService(sp => sp.GetRequiredService<ShutdownGate>());

var app = builder.Build();

app.MapGet("/health/ready", (ShutdownGate gate) =>
    gate.Draining ? Results.StatusCode(StatusCodes.Status503ServiceUnavailable) : Results.Ok());
app.MapGet("/orders/{id:int}", (int id) => Results.Ok(new { id }));
app.Run();

sealed class ShutdownGate(IConfiguration configuration, IHostEnvironment environment, ILogger<ShutdownGate> logger)
    : IHostedLifecycleService
{
    public volatile bool Draining;

    public async Task StoppingAsync(CancellationToken cancellationToken)
    {
        Draining = true;
        // Developers pressing Ctrl+C should not wait; clusters set Shutdown:DrainSeconds=10.
        var seconds = environment.IsDevelopment() ? 0 : configuration.GetValue("Shutdown:DrainSeconds", 10);
        logger.LogInformation("Draining for {Seconds} s before Kestrel stops", seconds);
        await Task.Delay(TimeSpan.FromSeconds(seconds), cancellationToken);
    }

    public Task StartingAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    public Task StartAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    public Task StartedAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    public Task StoppedAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}

Proving it: traffic during a rollout

A zero-downtime claim needs evidence: run steady traffic through the real ingress while the Deployment rolls, and treat any non-200 line as a bug. The rolling update deletes old pods, so their exit codes are gone; check in your log store that every old pod logged the stopped line. Do not read <code>lastState</code> for this: it records only restarts inside a running pod, and an exit 137 there with reason <code>OOMKilled</code> is a memory limit, not a grace period problem.

# Terminal 1: steady traffic, printing only failures
while true; do
  code=$(curl -s -o /dev/null -w '%{http_code}' https://orders.example.com/api/orders/7)
  [ "$code" = "200" ] || echo "$(date +%T) $code"
  sleep 0.05
done

# Terminal 2: roll every pod and wait for the result
kubectl rollout restart deployment/orders-api
kubectl rollout status deployment/orders-api --timeout=5m

# Terminal 3: follow an old pod to its end. The app logs "host stopped" from
# ApplicationStopped; a pod SIGKILLed at the deadline never prints it.
kubectl get pods -l app=orders-api -w
kubectl logs -f orders-api-7c9f6d8b5-xk2lp | grep -E 'shutting down|host stopped'

Discussion

Be the first to comment on this lesson.

What happens when a pod is terminated

The drain delay

The budget

The rollout itself

Long-lived connections

Example

When to use it

More examples

Rollout strategy, drain delay, grace period and PDB

The drain delay as reusable configuration

Proving it: traffic during a rollout

Related

Discussion