Load Balancing Across Backends
Pick a backend per request with round robin or least connections, and stop sending work to one that is unhealthy.
i := next.Add(1) - 1; pr.SetURL(targets[i%uint64(len(targets))])Load balancing is choosing which backend gets each request when several backends run the same code. The proxy does the forwarding; the choice is made once per request, and the choice is the whole subject.
Round robin
Hand the backends out in turn. Keep a counter, take it modulo the number of backends, and use that as the index. The counter is read and written by many goroutines at the same time, so it must be an atomic.Uint64 (added in Go 1.19) and not a plain uint64, which would be a data race. Round robin shares requests out evenly. That shares work out evenly only when requests cost about the same.
Least connections
Count the requests each backend is handling right now. Add one before forwarding and subtract one in a defer, so the count drops even if the forward panics. Send each request to the smallest count. A backend that has gone slow holds its requests longer, so its count stays high and it stops attracting more.
Health
A health check is a small request the proxy sends on a timer, usually GET /healthz. Store the answer in an atomic.Bool and skip any backend marked down. Without that, a dead backend keeps its turn in the rotation and one request in every N fails.
What trips people up
Checking only that the port accepts a connection. A process can accept connections while being unable to answer anything. Make the check do something small and real, and give it a timeout shorter than the gap between checks.
Example
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"net/http/httputil"
"net/url"
"sync/atomic"
)
// next counts requests. atomic.Uint64 is safe when many goroutines add to
// it at once; a plain uint64 here would be a data race.
var next atomic.Uint64
func backend(name string) *httptest.Server {
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
fmt.Fprintln(w, "served by", name)
}))
}
func main() {
a, b := backend("a"), backend("b")
defer a.Close()
defer b.Close()
targets := []*url.URL{
{Scheme: "http", Host: a.Listener.Addr().String()},
{Scheme: "http", Host: b.Listener.Addr().String()},
}
proxy := &httputil.ReverseProxy{Rewrite: func(pr *httputil.ProxyRequest) {
// Add returns the value after adding, so take one off to start at 0.
i := next.Add(1) - 1
pr.SetURL(targets[i%uint64(len(targets))])
}}
for range 4 { // ranging over an integer needs Go 1.22 or later
rec := httptest.NewRecorder()
proxy.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/orders", nil))
fmt.Print(rec.Body.String())
}
}
// Output:
// served by a
// served by b
// served by a
// served by b
When to use it
- Spread traffic over three copies of the same service running in Kubernetes.
- Stop sending requests to a replica that has stopped answering /healthz.
- Keep a slow backend from being handed as much work as a fast one.
- Drain traffic from one instance before a deploy restarts it.
More examples
Least connections
Reach for this when your backends are not interchangeable in practice: different machine sizes, a cold cache after a restart, or one request that costs a hundred times another. The pattern to copy is making the choice in a handler and carrying it on the context, because Rewrite is given only the request and cannot be passed extra arguments.
type backend struct {
url *url.URL
inflight atomic.Int64 // requests this backend is handling right now
}
// The chosen backend rides on the request context, where Rewrite can read it.
type chosenKey struct{}
func pick(bs []*backend) *backend {
best := bs[0]
for _, b := range bs[1:] {
if b.inflight.Load() < best.inflight.Load() {
best = b
}
}
return best
}
func balance(bs []*backend, proxy *httputil.ReverseProxy) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
b := pick(bs)
b.inflight.Add(1)
defer b.inflight.Add(-1) // runs even if forwarding panics
ctx := context.WithValue(r.Context(), chosenKey{}, b)
proxy.ServeHTTP(w, r.WithContext(ctx))
})
}
var rewrite = func(pr *httputil.ProxyRequest) {
pr.SetURL(pr.In.Context().Value(chosenKey{}).(*backend).url)
}Health checks in the background
The backend struct here has one more field than the previous example: up atomic.Bool. Checking on a timer in the background, rather than when a request arrives, means a request never waits for a probe. Use it any time your backend list is longer than one, and read the up flag in pick so a failing backend is skipped before it can fail a user's request.
// watch flips each backend up or down every two seconds. Start it once,
// with go watch(ctx, bs, client). The picker skips backends marked down.
func watch(ctx context.Context, bs []*backend, client *http.Client) {
t := time.NewTicker(2 * time.Second)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
for _, b := range bs {
b.up.Store(probe(ctx, client, b.url.String()+"/healthz"))
}
}
}
}
func probe(ctx context.Context, client *http.Client, url string) bool {
// Shorter than the two-second gap, so probes cannot pile up.
ctx, cancel := context.WithTimeout(ctx, time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return false
}
resp, err := client.Do(req)
if err != nil {
return false
}
defer resp.Body.Close()
return resp.StatusCode == http.StatusOK
}One retry onto another backend
Assign this to proxy.ErrorHandler to turn a single unlucky connection into a served request instead of a 502. The attempt counter is what stops a ring of dead backends from retrying forever. It only helps if the picker chooses a different backend the second time, which round robin does on its own and a health-aware picker does better.
type attemptKey struct{}
// Called when the backend could not be reached at all. Nothing has been
// written to the client yet, so a second backend may still answer.
func retryOnce(proxy *httputil.ReverseProxy) func(http.ResponseWriter, *http.Request, error) {
return func(w http.ResponseWriter, r *http.Request, err error) {
attempt, _ := r.Context().Value(attemptKey{}).(int)
// Replay only requests that are safe to repeat, and only once.
if r.Method != http.MethodGet || attempt > 0 {
http.Error(w, "upstream unavailable", http.StatusBadGateway)
return
}
ctx := context.WithValue(r.Context(), attemptKey{}, attempt+1)
proxy.ServeHTTP(w, r.WithContext(ctx))
}
}
Discussion