Graceful Shutdown

Handle SIGTERM with shutdown hooks so a service stops taking traffic, finishes in-flight requests and releases resources before it exits.

SyntaxRuntime.getRuntime().addShutdownHook(new Thread(() -> { server.stop(5); // stop intake, wait up to 5 s pool.shutdown(); // then drain workers; awaitTermination sets the deadline }, "shutdown"));

Services are stopped all the time: every deploy, scale-down and node drain. A graceful shutdown means none of that is visible to users: requests already in progress finish, no new ones arrive, and connections, files and message offsets are closed or committed properly.

What happens when a container is stopped

  1. Kubernetes marks the pod as terminating and starts removing it from Service endpoints. At the same moment it runs any preStop hook and then sends SIGTERM to process 1.
  2. The JVM handles SIGTERM by starting an orderly exit: it runs every registered shutdown hook, each in its own thread, in no guaranteed order.
  3. If the pod is still running after terminationGracePeriodSeconds (30 by default), it gets SIGKILL. No hooks run then.

Shutdown hooks also run on System.exit and when the last non-daemon thread ends. They do not run on SIGKILL, when the kernel kills the process for exceeding its memory limit, or on Runtime.halt.

The order that works

  1. Fail readiness, so load balancers stop choosing this instance.
  2. Stop accepting new connections.
  3. Wait, with a deadline, for in-flight requests and queued jobs.
  4. Close resources: message consumers, then connection pools.

Endpoint removal is not instant. Proxies and other nodes may route requests to the pod for a few seconds after SIGTERM arrives. A short preStop sleep keeps the service accepting during that window; without it, those requests fail with connection errors.

Mistakes people make

  • A shell-form ENTRYPOINT or a wrapper script without exec: SIGTERM reaches the shell, not the JVM, and the hooks never run.
  • Hooks that wait without a deadline. The JVM waits for every hook; one stuck hook holds the exit until SIGKILL.
  • A total shutdown time longer than the grace period. The preStop sleep plus the drain time must fit inside it.
  • Calling System.exit from inside a hook. It blocks forever, because shutdown is already running.

Spring Boot handles most of this. Graceful shutdown of the embedded web server is on by default since Spring Boot 3.4, the readiness state changes to refusing traffic during shutdown, and spring.lifecycle.timeout-per-shutdown-phase (30 seconds by default) bounds each phase. Your job is the preStop delay, a grace period that fits, and closing anything you started yourself.

Example

Example · java

import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

void main() throws Exception {
    ExecutorService workers = Executors.newFixedThreadPool(2);
    var server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
    server.setExecutor(workers);
    server.createContext("/orders", ex -> {
        byte[] body = "accepted".getBytes(StandardCharsets.UTF_8);
        ex.sendResponseHeaders(202, body.length);
        try (var out = ex.getResponseBody()) {
            out.write(body);
        }
    });
    server.start();

    // Runs on SIGTERM (docker stop, Kubernetes) and on System.exit
    Runtime.getRuntime().addShutdownHook(new Thread(() -> {
        IO.println("shutdown: stop accepting connections");
        server.stop(1); // waits up to 1 s for exchanges in progress
        workers.shutdown();
        try {
            boolean drained = workers.awaitTermination(5, TimeUnit.SECONDS);
            IO.println("shutdown: workers drained = " + drained);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        IO.println("shutdown: done");
    }, "graceful-shutdown"));

    try (var client = HttpClient.newHttpClient()) {
        var request = HttpRequest.newBuilder(
                URI.create("http://127.0.0.1:" + server.getAddress().getPort() + "/orders"))
                .POST(HttpRequest.BodyPublishers.ofString("{\"sku\":\"A-1\"}"))
                .build();
        var response = client.send(request, HttpResponse.BodyHandlers.ofString());
        IO.println("POST /orders -> " + response.statusCode() + " " + response.body());
    }

    IO.println("main: exit requested");
    System.exit(0); // starts the same shutdown sequence a SIGTERM would
}

// Output:
// POST /orders -> 202 accepted
// main: exit requested
// shutdown: stop accepting connections
// shutdown: workers drained = true
// shutdown: done

When to use it

  • Deploy a checkout API several times a day without clients seeing connection resets or half-finished payments.
  • Let a queue consumer finish its current message and commit the offset before a scale-down removes the pod.
  • Close a HikariCP pool cleanly on shutdown so the database does not keep hundreds of abandoned sessions after each deploy.

More examples

Kubernetes: a grace period that fits the drain

The timeline is 10 seconds of preStop, then SIGTERM and up to 30 seconds of draining, which fits inside 45 with room to spare. An exec sleep needs a sleep binary in the image; newer Kubernetes versions also offer a built-in sleep action for preStop.

Example · yaml
spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: orders
      image: registry.example.com/orders-service:1.4.0
      lifecycle:
        preStop:
          exec:
            # Keep serving while endpoint removal spreads through the cluster
            command: ["sleep", "10"]

Spring Boot shutdown settings

With graceful shutdown the embedded server stops accepting new requests and waits for active ones up to the phase timeout. Keep that timeout well below the pod's grace period minus the preStop delay.

Example · properties
# Graceful is already the default since Spring Boot 3.4; stated here for clarity
server.shutdown=graceful

# Upper bound for each shutdown phase, including waiting for active requests
spring.lifecycle.timeout-per-shutdown-phase=20s

A start script that passes signals on

Without exec the shell stays process 1, the JVM is its child, and SIGTERM goes to the shell, which ignores it. This is the most common reason shutdown hooks appear not to run in containers.

Example · bash
#!/bin/sh
set -eu

# Prepare anything the service needs at startup
mkdir -p /tmp/orders

# exec replaces the shell with java, so java becomes process 1 and gets SIGTERM
exec java ${JAVA_OPTS:-} -jar /app/orders-service.jar "$@"

Related

java health checksjava executorservicejava thread safety in web serversjava a docker image for javajava connection pooling with hikaricp

Discussion

  • Be the first to comment on this lesson.