Graceful Shutdown
Let in-flight requests and background work finish when the platform stops an instance, instead of cutting them off mid-way.
server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=20s
spring.task.execution.shutdown.await-termination=true
spring.task.execution.shutdown.await-termination-period=10sInstances stop all the time: every deployment, every scale-down, every node upgrade. The platform sends SIGTERM, waits a while, then sends SIGKILL. What your application does in between decides whether users see failed requests on every release.
What Spring Boot does
On SIGTERM the JVM runs shutdown hooks, and Spring Boot closes the application context. With server.shutdown=graceful, which is the default in current Spring Boot versions, the embedded web server stops accepting new requests and waits for the active ones to complete. Readiness switches to refusing traffic. Then beans are stopped in lifecycle phases and destroyed.
Each lifecycle phase gets at most spring.lifecycle.timeout-per-shutdown-phase, 30 seconds by default. Requests still running after that are cut off. Set it longer than your slowest legitimate request, and keep it shorter than the platform's grace period.
The Kubernetes race
Kubernetes sends SIGTERM and removes the pod from service endpoints at the same time, and the removal takes a few seconds to reach every load balancer and proxy. If the application stops accepting connections immediately, requests routed during those seconds fail. The usual fix is a short preStop sleep, so the pod keeps serving while the network catches up, then shuts down gracefully.
Do the arithmetic: terminationGracePeriodSeconds (30 by default) must be greater than the preStop sleep plus every wait in the shutdown added together: the web server's phase timeout, executor await periods and any SmartLifecycle bean that blocks in stop(). Otherwise Kubernetes kills the JVM part way through and the careful configuration achieves nothing.
Background work
Graceful shutdown of the web server does nothing for your own threads. Executors started by @Async and @Scheduled are not waited on unless you set spring.task.execution.shutdown.await-termination=true and a period, and the same for spring.task.scheduling. For your own long-running loops, such as a queue poller, implement SmartLifecycle: Spring calls stop() during shutdown, in reverse phase order, so you can finish the current item and exit cleanly.
Make work safe to interrupt anyway
Graceful shutdown is best effort. A node can lose power and kill -9 exists. Design jobs so that an item interrupted mid-way is retried rather than lost, with database transactions, idempotent handlers and message acknowledgement after processing. Graceful shutdown then reduces noise rather than being the only thing that protects your data.
Test it: start a slow request, send SIGTERM with kill, and check that the request completes and the logs show the graceful shutdown messages.
Example
import java.time.Duration;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.SmartLifecycle;
import org.springframework.stereotype.Component;
public class OutboxPoller implements SmartLifecycle {
private static final Logger log = LoggerFactory.getLogger(OutboxPoller.class);
private final ExecutorService worker = Executors.newSingleThreadExecutor();
private volatile boolean running;
public void start() {
running = true;
worker.submit(this::loop);
}
public void stop() {
// Called on shutdown: finish the current batch, then let the loop exit.
running = false;
worker.shutdown();
try {
if (!worker.awaitTermination(10, TimeUnit.SECONDS)) {
log.warn("Outbox poller did not finish in time");
worker.shutdownNow();
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
public boolean isRunning() {
return running;
}
private void loop() {
while (running) {
publishNextBatch();
try {
Thread.sleep(Duration.ofSeconds(1));
} catch (InterruptedException e) {
return;
}
}
}
private void publishNextBatch() {
// Read unsent rows, publish them, mark them sent in one transaction.
}
}
When to use it
- Deploy several times a day without a burst of 502 errors at every rollout.
- Let a report generation request that takes 15 seconds finish when the pod is scaled down.
- Stop a message poller cleanly so the batch it is working on is completed rather than processed twice.
More examples
Shutdown settings
Writing <code>server.shutdown=graceful</code> explicitly costs nothing and documents the intent for the next reader. The executor settings matter most for services with long-running background tasks, which otherwise are dropped the moment the context closes.
# Stop accepting new requests and let active ones finish.
server.shutdown=graceful
# Maximum wait per lifecycle phase; keep it under the platform's grace period.
spring.lifecycle.timeout-per-shutdown-phase=20s
# Wait for @Async and @Scheduled work instead of abandoning it.
spring.task.execution.shutdown.await-termination=true
spring.task.execution.shutdown.await-termination-period=10s
spring.task.scheduling.shutdown.await-termination=true
spring.task.scheduling.shutdown.await-termination-period=10s
Kubernetes timings
The 10 second sleep runs before <code>SIGTERM</code> is sent, giving load balancers time to stop routing new requests. Then add up the waits from this lesson: up to 20 seconds for in-flight requests, up to 10 for the outbox poller and up to 10 each for the task and scheduling executors, about 60 seconds at worst including the sleep. 70 leaves headroom. The <code>sleep</code> action needs a recent Kubernetes version; on older clusters use an <code>exec</code> hook running <code>sleep 10</code>, which needs a shell in the image.
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders
spec:
template:
spec:
terminationGracePeriodSeconds: 70
containers:
- name: orders
image: registry.example.com/orders:1.4.2
lifecycle:
preStop:
sleep:
seconds: 10
Checking it by hand
This is the simplest way to see graceful shutdown work before trusting it in production. If curl fails with a connection reset, the server is not waiting: check <code>server.shutdown</code> and whether something else, such as a wrapper script, is killing the JVM directly.
# Start the application, then hold a slow request open.
java -jar target/orders.jar &
APP_PID=$!
sleep 15
curl -s http://localhost:8080/reports/slow &
# Ask for a graceful stop while the request is running.
sleep 1
kill -TERM "$APP_PID"
# The curl call should still complete; the log should say
# "Commencing graceful shutdown. Waiting for active requests to complete".
wait
Discussion