Scaling and Rollouts
Replicas, rolling updates with maxSurge and maxUnavailable, and the SIGTERM handling that keeps a deploy from dropping requests.
server.close([callback])Scaling a Node service in Kubernetes means changing one number. kubectl scale deployment/api --replicas=4 is the shorthand for editing spec.replicas in the manifest and applying it; they are the same edit, and the result is four identical copies with the Service spreading traffic over them.
Rolling updates
Changing the image starts a rolling update: new pods are added and old ones removed a few at a time. Two fields set the pace, both 25% of the replica count by default. maxUnavailable is how many of the wanted replicas may be missing, and maxSurge is how many extra may exist at once. A new pod counts only once its readiness probe passes, which is what keeps a broken image from replacing a working one.
Losing requests on the way out
When a pod is going away, the kubelet sends SIGTERM to your process while the control plane removes the pod's address from the Service. Those happen at the same time, so requests can still arrive for a moment after the signal. Node's own default for SIGTERM is to exit, which drops every open connection.
So: handle the signal, make readiness fail, then call server.close(callback). It stops new connections, closes idle ones, and runs the callback once the in-flight requests have finished. Finish before terminationGracePeriodSeconds (30 by default) runs out, or SIGKILL ends the process mid-request.
Example
When to use it
- Deploying a new image during business hours without returning errors to users.
- Handling a traffic peak by running more copies of the same container.
- Finding out why a deploy produces a short burst of 502s every time.
- Draining database connections and in-flight requests before a pod is removed.
More examples
A rollout that never drops below full capacity
maxUnavailable: 0 with maxSurge: 1 trades rollout speed for capacity: one new pod starts, becomes ready, then one old pod goes. The preStop hook buys the few seconds the endpoint change needs to reach every node, which is where most mysterious 502s during a deploy come from. sleep must exist in the image, which rules it out of a scratch-based image.
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0 # never fewer than 4 serving pods
maxSurge: 1 # at most 5 pods exist at once
minReadySeconds: 5 # ready for 5s before it counts as available
progressDeadlineSeconds: 300
template:
spec:
terminationGracePeriodSeconds: 45 # longer than the shutdown path needs
containers:
- name: api
image: registry.example.com/api:1.4.3
lifecycle:
preStop:
exec:
# Keep serving while the pod's address is being removed from
# the Service. SIGTERM is sent after this hook finishes.
command: ["sleep", "5"]Shutdown with a hard deadline
The deadline is the part people leave out. One long-polling client or a slow query holds server.close() open indefinitely, and SIGKILL then arrives before the connection pool is drained, leaving idle connections on the database until it times them out. process.once rather than process.on means a second SIGTERM does not start a second shutdown.
const SHUTDOWN_TIMEOUT_MS = 20_000; // under terminationGracePeriodSeconds
async function shutdown(signal) {
console.log(JSON.stringify({ level: 'info', msg: 'shutting down', signal }));
state.ready = false; // /readyz starts answering 503
const forced = setTimeout(() => {
// Still open after the deadline: cut them off rather than be SIGKILLed
// halfway through this function. Added in Node.js 18.2.0.
server.closeAllConnections();
}, SHUTDOWN_TIMEOUT_MS);
forced.unref();
await new Promise((resolve) => server.close(resolve));
clearTimeout(forced);
await pool.end(); // close the database connections
console.log(JSON.stringify({ level: 'info', msg: 'shutdown complete' }));
}
for (const signal of ['SIGTERM', 'SIGINT']) {
process.once(signal, () => {
shutdown(signal).then(
() => { process.exitCode = 0; },
(err) => { console.error(err); process.exitCode = 1; }
);
});
}Scaling on CPU instead of by hand
Reach for this when traffic has peaks worth paying for and troughs worth not paying for. Two things are required for it to work at all: the container must declare a CPU request, since Utilization is measured against it, and the metrics API must be installed in the cluster. Note also that spec.replicas in your Deployment manifest and an autoscaler both own that number, so applying the manifest can briefly undo the autoscaler's last decision.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
# A percentage of the container's CPU *request*, not of a whole CPU.
# With requests.cpu: 250m, 60% means 150m average per pod.
averageUtilization: 60
Discussion