Graceful Shutdown

When a process is told to stop, it receives a signal. Kubernetes and Docker send SIGTERM. Ctrl+C sends SIGINT. The default reaction to either is to die immediately, cutting off every request that was being served. Graceful shutdown means stop accepting new requests, let the current ones finish, then exit.

How it works

signal.NotifyContext, added in Go 1.16, returns a context that is cancelled when one of the named signals arrives. Run srv.ListenAndServe in a goroutine, wait on <-ctx.Done(), then call srv.Shutdown. Shutdown closes the listening socket so no new connection is accepted, closes connections that are sitting idle, and waits for the handlers still running to return. Give Shutdown its own context with a timeout; if that deadline passes it returns the context's error and stops waiting, so one stuck handler cannot hold the process open forever.

What Kubernetes expects

Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds, then sends SIGKILL, which cannot be caught. Your shutdown timeout must be shorter than that grace period. Removing the pod from the load balancer happens in parallel with the signal, not before it, so requests can still arrive for a second or two afterwards. Fail the readiness probe first, pause briefly, and only then shut the server down.

What trips people up

ListenAndServe returns http.ErrServerClosed as soon as Shutdown begins. That is the normal path, not a failure. Compare with errors.Is and do not log it as an error.

package main

import (
	"context"
	"errors"
	"fmt"
	"log"
	"net/http"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("GET /slow", func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(3 * time.Second) // stands in for real work
		fmt.Fprintln(w, "done")
	})
	srv := &http.Server{Addr: ":8080", Handler: mux}

	// ctx is cancelled when the process is asked to stop.
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	go func() {
		// ErrServerClosed is the normal result of Shutdown.
		if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatal(err)
		}
	}()

	<-ctx.Done()
	fmt.Println("shutting down")

	// In-flight requests get 10 seconds to finish.
	timeout, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()
	if err := srv.Shutdown(timeout); err != nil {
		log.Fatalf("shutdown: %v", err)
	}
	fmt.Println("server stopped")
}

// Run with: go run .
// Start a slow request: curl http://localhost:8080/slow
// Press Ctrl+C while it is running. curl still prints "done",
// and the program prints:
// shutting down
// server stopped

var ready atomic.Bool // safe to read and write from many goroutines

func readyz(w http.ResponseWriter, r *http.Request) {
	if !ready.Load() {
		http.Error(w, "shutting down", http.StatusServiceUnavailable)
		return
	}
	fmt.Fprintln(w, "ok")
}

// At start-up: ready.Store(true)
// After the signal arrives:
//   ready.Store(false)
//   time.Sleep(5 * time.Second) // let the load balancer stop sending traffic
//   srv.Shutdown(timeout)

// Connections upgraded to WebSocket are hijacked, so Shutdown
// returns without waiting for them. Close them yourself.
srv.RegisterOnShutdown(func() {
	hub.CloseAll() // your own code: tell every live socket to finish
})

if err := srv.Shutdown(timeout); err != nil {
	fmt.Println("forced stop:", err)
	srv.Close() // give up on whatever is left open
}

if err := srv.Shutdown(timeout); err != nil {
	log.Printf("shutdown: %v", err)
}

// Only now is no handler running, so these are safe to close.
workers.Stop()  // background jobs
if err := db.Close(); err != nil {
	log.Printf("closing database: %v", err)
}