Load Balancers
One address in front of many servers: what L4 and L7 can each route on, how draining works, and why stickiness is usually a smell.
upstream NAME { least_conn; server HOST:PORT weight=1 max_fails=1 fail_timeout=10s; }A load balancer is a single address in front of many interchangeable servers. It exists so capacity can be added by starting another process, and so a machine can die or be replaced without a client noticing.
Layer 4 against layer 7
An L4 balancer forwards TCP or UDP connections. It sees addresses, ports and, with TLS passthrough, the SNI name — nothing else — so it is cheap, protocol-agnostic, and can preserve the client's source IP. AWS Network Load Balancer works this way. An L7 balancer terminates HTTP, so it can route on host, path, method, header, query string and cookie, rewrite requests, and retry one on a different target. AWS Application Load Balancer is L7. Routing /api/v2/* somewhere else requires L7; anything that is not HTTP requires L4.
Health checks and draining
The balancer probes every target and pulls failures out of rotation after a threshold. An ALB target group defaults to a check every 30 seconds, 2 consecutive failures to go out of service and 5 consecutive successes to come back. Removal is not instant: during deregistration new requests stop but in-flight ones are allowed to finish, and ALB's deregistration delay defaults to 300 seconds.
Sticky sessions
Stickiness pins a client to one target with a cookie — AWSALB on an ALB, default duration one day. It makes a stateful app survive behind a balancer, and buys that with uneven load, sessions lost on every deploy, and a scale-out event that cannot relieve the hot target. Usually the honest fix is moving the state out.
Example
// Graceful shutdown that cooperates with the balancer instead of fighting it.
// The order matters: fail readiness FIRST, keep serving, close LAST.
const express = require("express");
const app = express();
let ready = true;
// Liveness: is the process alive? It must not depend on a database or a
// downstream, or one blip restarts every container in the fleet at once.
app.get("/healthz", (req, res) => res.sendStatus(200));
// Readiness: should this instance be sent traffic? This is the path the
// target group polls (HealthCheckPath).
app.get("/readyz", (req, res) =>
ready ? res.sendStatus(200) : res.sendStatus(503)
);
app.get("/", (req, res) => res.send("ok"));
const server = app.listen(8080);
process.on("SIGTERM", () => {
ready = false; // next health check fails -> the LB starts taking us out
// Keep accepting for longer than interval * unhealthyThreshold, so nothing
// is routed here after we stop listening. ALB defaults: 30s * 2 = 60s.
setTimeout(() => {
// close() stops accepting new connections and waits for in-flight ones.
server.close(() => process.exit(0));
}, 70_000).unref();
});When to use it
- Deciding whether a service needs an ALB or an NLB when the protocol is gRPC or raw TCP.
- Working out why a rolling deploy returns 502s for a few seconds on every release.
- Replacing sticky sessions with a shared session store so scale-out actually relieves load.
- Writing readiness and liveness endpoints that a target group and Kubernetes both behave sensibly against.
More examples
An nginx upstream, with the shorthand and the longhand side by side
Reach for this when the balancer is something you run yourself. Note what is missing: there is no health_check directive, because active checks are an NGINX Plus feature. Open-source nginx only learns a backend is bad by failing a real user's request, which is why max_fails and fail_timeout matter.
upstream api {
least_conn; # without this, weighted round-robin is used
# Shorthand: port 80 and every failure default implied.
server 10.0.1.11;
# The same server line written out in full. Identical behaviour.
server 10.0.1.12:80 weight=1 max_fails=1 fail_timeout=10s;
keepalive 32; # reuse upstream connections
}
server {
location / {
proxy_pass http://api;
# Retry the next server only on these conditions, and only once more,
# so one slow request cannot be amplified across the whole pool.
proxy_next_upstream error timeout http_502 http_503;
proxy_next_upstream_tries 2;
}
}An ALB target group tuned for fast, safe deploys
The defaults are tuned for safety, not for deploy speed. Shortening the interval and the healthy threshold gets a new task into service in roughly 20 seconds instead of 150, and a shorter deregistration delay must still exceed your slowest in-flight request.
resource "aws_lb_target_group" "api" {
name = "api"
port = 8080
protocol = "HTTP"
vpc_id = var.vpc_id
# Connection draining. AWS default is 300s, which makes every rolling
# deploy five minutes slower than it needs to be.
deregistration_delay = 30
health_check {
path = "/readyz"
interval = 10 # default 30
timeout = 5 # default 5
healthy_threshold = 2 # default 5: slow to return after a blip
unhealthy_threshold = 2 # default 2
matcher = "200"
}
stickiness {
enabled = false # the AWS default
type = "lb_cookie" # required even when disabled
}
}The routing decision only an L7 balancer can make
This is how a monolith gets carved up without changing a single client URL, and how a canary gets a slice of traffic by header or cookie. Whenever a design calls for path-based routing, header-based routing or request rewriting, it has chosen L7 and should say so.
# Send /api/v2/* to a different target group than the rest of the host.
# An L4 balancer cannot do this: it never sees the path.
aws elbv2 create-rule \
--listener-arn "$LISTENER_ARN" \
--priority 10 \
--conditions Field=path-pattern,Values='/api/v2/*' \
--actions Type=forward,TargetGroupArn="$TG_V2_ARN"
Discussion