Shutting Down Cleanly

SIGTERM is a request, not a kill: catch it, stop taking work, finish what is in flight, and exit before SIGKILL arrives.

Syntaxsignal.signal(signal.SIGTERM, handler) # handler(signum, frame)

When an orchestrator replaces your container it sends SIGTERM, waits, then sends SIGKILL. docker stop waits 10 seconds; Kubernetes waits terminationGracePeriodSeconds, 30 by default. SIGKILL cannot be caught, blocked or handled, so everything you intend to do on shutdown has to happen inside that window.

SIGTERM is not SIGINT

Python installs a handler for SIGINT that raises KeyboardInterrupt, which is why your except KeyboardInterrupt and your finally blocks run when you press Ctrl-C. SIGTERM keeps the system default action: the process dies at once — no finally, no atexit callback, no flushed buffer. except KeyboardInterrupt is the shorthand you get for free; for SIGTERM you write the long form yourself.

The handler does almost nothing

signal.signal(signal.SIGTERM, handler) installs it and returns the previous handler. The handler is called with (signum, frame), runs in the main thread of the main interpreter at the next bytecode boundary, and may only be installed from that thread — elsewhere it raises ValueError. Because it interrupts arbitrary code at an arbitrary point, it should only set a flag or a threading.Event. The main loop checks the flag, stops accepting new work, finishes what is in flight, and returns.

finally and atexit

finally is for a resource held by a scope. atexit.register(func, *args, **kwargs) is for process-wide teardown: callbacks run in reverse registration order and are skipped entirely on os._exit(), on a fatal interpreter error, and on any signal Python is not handling.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Finish the HTTP requests already in flight when a rolling deploy replaces the pod.
  • Acknowledge the queue messages you have processed and leave the rest for the next consumer.
  • Flush buffered metrics and log records before the process exits instead of losing the last minute.
  • Make Ctrl-C locally and SIGTERM in production take the same shutdown path.

More examples

A worker loop with a threading.Event

Everything the handler does is set an Event; all the decisions happen in the loop, where the code is not re-entrant. The timeout on receive is the part people forget - a worker blocked for 30 seconds in a receive call never checks the flag and is SIGKILLed with work in hand.

Example · python
import logging
import signal
import threading

log = logging.getLogger(__name__)
stop = threading.Event()

def handle_shutdown(signum, frame):
    # Event.set() is safe to call from a signal handler and is visible to
    # every thread, unlike a plain global read in a worker.
    log.info("shutdown requested", extra={"signal": signal.Signals(signum).name})
    stop.set()

for sig in (signal.SIGTERM, signal.SIGINT):
    signal.signal(sig, handle_shutdown)     # replaces the KeyboardInterrupt handler too

def main():
    try:
        while not stop.is_set():
            # Long poll with a timeout, not a bare blocking receive: the loop has
            # to come back and look at the flag for the flag to mean anything.
            messages = queue_client.receive(wait_seconds=5, max_messages=10)
            for message in messages:
                handle(message)             # finish the batch you already claimed
                message.ack()
                if stop.is_set():
                    break                   # ack what is done, leave the rest unacked
    finally:
        queue_client.close()
        logging.shutdown()                  # flush handlers before the process goes

Earning the grace period you need

A correct signal handler still drops requests if the load balancer has not stopped sending them yet; the preStop sleep covers that propagation gap. Note that the grace period is shared with the hook, so budget both together rather than setting the period to exactly your request timeout.

Example · yaml
spec:
  # SIGTERM, then this long, then SIGKILL. The default is 30.
  # Set it above your slowest acceptable request, not above your slowest ever.
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            # Endpoint removal is asynchronous: for a second or two after the
            # pod is marked Terminating, proxies are still routing to it.
            # Sleeping here keeps the process serving during that window, so
            # in-flight and just-arrived requests are not refused.
            command: ["sleep", "5"]
      readinessProbe:
        httpGet: { path: /readyz, port: 8080 }
        periodSeconds: 2
      # The clock starts at SIGTERM and includes the preStop hook, so the
      # process has roughly 40 seconds of real draining time here.

Why the handler must not do the work

The escalation pattern in the second version is worth copying: the first signal asks for a drain, the second restores SIG_DFL and re-raises so the process dies immediately. That gives an operator a way to stop a hung drain without reaching for kill -9.

Example · python
import signal
import sys

# WRONG. The handler interrupts the main thread at an arbitrary bytecode. If a
# second SIGTERM arrives while this is running, the handler re-enters itself and
# flush() runs twice, concurrently, on the same buffer. Anything that takes a
# lock can deadlock against a lock the interrupted code was already holding.
def bad(signum, frame):
    metrics.flush()          # network I/O, from inside a signal handler
    db.close()
    sys.exit(0)              # raises SystemExit from wherever the program was

# RIGHT. Flip a flag; optionally make the second signal mean "stop waiting".
_shutdown = False

def good(signum, frame):
    global _shutdown
    if _shutdown:
        # Second SIGTERM, or an impatient operator: give up the graceful path.
        signal.signal(signum, signal.SIG_DFL)   # restore the default action
        signal.raise_signal(signum)             # and let it kill us now
    _shutdown = True

signal.signal(signal.SIGTERM, good)

Related

py cloud containerpy cloud jobspy cloud healthpy adv context managerspy async pitfalls

Discussion

  • Be the first to comment on this lesson.