Rollouts and Rollbacks

How a rolling update replaces pods a few at a time, how to undo it, and why SIGTERM handling decides whether anyone notices.

Syntaxkubectl set image deployment/shop-api app=repo/shop-api:1.4.3 && kubectl rollout status deployment/shop-api

Changing the image in a Deployment starts a rolling update. Kubernetes does not replace every pod at once. It starts some new pods, waits for them to pass their readiness probe, then removes the same number of old ones, and repeats until every pod runs the new image. The service stays up while that happens.

Surge and unavailability

Two numbers control the pace. maxSurge is how many pods may exist above the requested count. maxUnavailable is how many of the requested count may be missing. Both default to 25%, and both may be written as a count or a percentage. They cannot both be zero. Setting maxUnavailable: 0 with maxSurge: 1 never drops below full capacity, at the cost of a slower rollout.

Each change keeps the old set of pods around, so kubectl rollout undo can bring it back. revisionHistoryLimit, 10 by default, decides how many old sets are kept. If the rollout makes no progress for progressDeadlineSeconds, 600 by default, it is marked failed. It is not undone for you.

Why shutdown matters here

A rollout deletes pods, and every delete is a chance to drop requests. The cluster sends your process SIGTERM, waits terminationGracePeriodSeconds (30 by default), then sends SIGKILL. A program that exits the moment it sees SIGTERM cuts off the requests it is still holding. The rollout then costs a few hundred failed requests, on every single deploy.

Example

package main

import (
	"context"
	"errors"
	"fmt"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	// SIGTERM is the first signal the cluster sends to replace a pod.
	// NotifyContext cancels ctx when it arrives.
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
	defer stop()

	// Port 0 picks any free port, so this cannot clash with a server
	// you already have running. In a pod the address is ":8080".
	srv := &http.Server{Addr: "127.0.0.1:0"}
	go func() {
		// ListenAndServe returns ErrServerClosed once Shutdown is called.
		if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
			fmt.Println("serve:", err)
		}
	}()

	// Stand in for the cluster deleting this pod.
	go func() {
		time.Sleep(50 * time.Millisecond)
		if err := syscall.Kill(os.Getpid(), syscall.SIGTERM); err != nil {
			fmt.Println("kill:", err)
		}
	}()

	<-ctx.Done()
	fmt.Println("SIGTERM received, draining")

	// Finish the handlers already running, but not past the grace period.
	drain, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	if err := srv.Shutdown(drain); err != nil {
		fmt.Println("shutdown:", err)
		return
	}
	fmt.Println("drained cleanly")
}

// Output:
// SIGTERM received, draining
// drained cleanly

When to use it

Deploy a new image version in the middle of the working day without dropping requests.
Get back to the last working version in one command after a bad release.
Replace every pod so a rotated Secret is read again, without changing the image.
Fail a CI job automatically when the new pods never become ready.

More examples

The rollout and shutdown settings in one Deployment

Everything that decides how a deploy feels to a user is in this one object. Note that maxSurge rounds up and maxUnavailable rounds down, so with four pods both come to one. minReadySeconds is the guard against a pod that passes readiness once and then falls over: without it, a new pod counts as available the instant it answers, and a broken version can roll all the way out. The alternative strategy type is Recreate, which stops every old pod before starting any new one; it causes an outage, and you want it only when two versions cannot run at the same time, such as during an incompatible schema change.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api
spec:
  replicas: 4
  # Shorthand: leave `strategy` out entirely and you get exactly this.
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%          # of 4 pods, rounded up:   1 extra pod
      maxUnavailable: 25%    # of 4 pods, rounded down: 1 missing pod
  minReadySeconds: 10        # a new pod must stay ready this long
  progressDeadlineSeconds: 600
  revisionHistoryLimit: 10   # how many old pod sets to keep
  selector:
    matchLabels:
      app: shop-api
  template:
    metadata:
      labels:
        app: shop-api
    spec:
      terminationGracePeriodSeconds: 45   # then SIGKILL
      containers:
        - name: app
          image: registry.example.com/shop-api:1.4.3
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 2
          lifecycle:
            preStop:
              exec:
                # Keep serving while the endpoint list catches up.
                # Needs a `sleep` binary, which a scratch or
                # distroless image does not have.
                command: ["sleep", "5"]

Driving and undoing a rollout

The five commands worth memorising. rollout status blocks until the new pods are available or the deadline passes, and its exit code is how a CI job learns that a deploy failed. undo does not delete anything: it creates a further revision whose contents are the old one, so the history keeps growing in one direction and you can undo an undo. rollout restart is the answer to "the Secret changed but the pods have the old value", because a container reads its environment only at start-up. If the service is installed with Helm, use helm rollback instead, so that Helm's own idea of the current revision stays true.

# Change the image, then watch. status exits non-zero if it fails,
# which is what makes it useful as the last step of a deploy job.
kubectl set image deployment/shop-api app=registry.example.com/shop-api:1.4.3
kubectl rollout status deployment/shop-api --timeout=5m

# Record why this revision exists, so history is readable later.
kubectl annotate deployment/shop-api \
  kubernetes.io/change-cause="image 1.4.3" --overwrite
kubectl rollout history deployment/shop-api

# Undo: one step back, or to a numbered revision from the history.
kubectl rollout undo deployment/shop-api
kubectl rollout undo deployment/shop-api --to-revision=7

# Replace every pod without changing the image, to reread a Secret.
kubectl rollout restart deployment/shop-api

# Hold a rollout part way, look around, then let it finish.
kubectl rollout pause deployment/shop-api
kubectl rollout resume deployment/shop-api

The order a shutdown has to happen in

Call this from the SIGTERM path of a real service. Shutting down immediately is the mistake: the five second pause is what turns a deploy from "a handful of 502s per pod" into "nothing in the logs". Two limits to know. Shutdown waits for ordinary requests but does not close hijacked connections, which is what a WebSocket becomes, so long-lived connections need their own goodbye and the client needs to reconnect. And if a handler takes longer than the grace period, SIGKILL still arrives, so a request that can run for minutes belongs in a background worker rather than in an HTTP handler.

// drain runs when SIGTERM arrives. The order of the three steps is
// the whole point.
func drain(srv *http.Server, h *health) error {
	// 1. Fail the readiness probe, so the cluster stops choosing this
	// pod for new requests.
	h.ready.Store(false)

	// 2. Keep serving for a few seconds while that removal reaches
	// every node. This is the preStop sleep, done in Go, which also
	// works in a scratch image with no shell in it.
	time.Sleep(5 * time.Second)

	// 3. Stop accepting connections and let the handlers still running
	// finish. Keep the deadline under terminationGracePeriodSeconds so
	// the process exits on its own rather than being SIGKILLed.
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	return srv.Shutdown(ctx)
}

Discussion

Be the first to comment on this lesson.