Load Shedding and Backpressure

When more work arrives than you can do, reject some of it early and cheaply instead of accepting all of it and failing at everything.

Syntaxselect { case permits <- struct{}{}: default: http.Error(w, "busy", http.StatusServiceUnavailable) }

Every service has a ceiling. Past it, accepting more work makes things worse: queues grow, each request waits longer, memory climbs, and eventually every request times out. You serve nobody instead of most people.

Load shedding is deciding, on purpose, to refuse some requests so the rest succeed. Backpressure is the related idea of pushing that refusal back to whoever is producing the work, so they slow down rather than keep pushing.

Shed early and cheaply

A shed request should cost almost nothing. Check the limit in middleware, a wrapper that runs before a handler, before you parse a body, call the database or validate a token. Reply 503 Service Unavailable for overload, or 429 Too Many Requests when one client is the cause, and set the Retry-After header so clients wait instead of retrying immediately.

The non-blocking send is the whole mechanism. select with a default case tries the send and takes default if it would block. The longhand, a bare permits <- struct{}{}, waits instead, which is queueing.

Backpressure for workers

A worker reading from a queue gets backpressure for free if its channel is bounded and it acknowledges a message only after the work is queued. Shed differently here: at dequeue time, drop jobs whose deadline has already passed, because finishing them helps nobody.

What trips people up

Shedding without Retry-After invites an immediate retry, and the shed load comes straight back as a retry storm.

Example

Example · go
package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"
)

// shed refuses a request when every permit is taken, instead of queueing it.
func shed(limit int, next http.Handler) http.Handler {
	permits := make(chan struct{}, limit)
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		select {
		case permits <- struct{}{}:
			defer func() { <-permits }()
			next.ServeHTTP(w, r)
		default:
			w.Header().Set("Retry-After", "1") // seconds to wait before retrying
			http.Error(w, "server too busy", http.StatusServiceUnavailable)
		}
	})
}

func main() {
	started, release := make(chan struct{}), make(chan struct{})
	slow := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		close(started)
		<-release // a request that is still being handled
	})
	h := shed(1, slow)

	go h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
	<-started // the only permit is now held

	rec := httptest.NewRecorder()
	h.ServeHTTP(rec, httptest.NewRequest("GET", "/", nil))
	fmt.Println("status:", rec.Code)
	fmt.Println("retry-after:", rec.Header().Get("Retry-After"))
	close(release)
}

// Output:
// status: 503
// retry-after: 1

When to use it

  • Keep a traffic spike from pushing a service into a death spiral of timed-out requests.
  • Return 429 with Retry-After to one noisy API client without affecting the others.
  • Drain a backlog after an incident by dropping queued jobs whose callers have already given up.
  • Stop a fast message broker from filling your service's memory with undelivered work.

More examples

Shedding below HTTP, at the socket

Middleware still pays for accepting the connection, reading headers and routing. LimitListener from golang.org/x/net/netutil stops accepting new connections entirely once the limit is reached, which is the cheapest refusal available: the client simply waits in the kernel's backlog. Use it as an outer wall under the per-route limits, not instead of them.

Example · go
ln, err := net.Listen("tcp", ":8080")
if err != nil {
	log.Fatal(err)
}
// At most 1000 connections are accepted at once; the rest wait in the backlog.
ln = netutil.LimitListener(ln, 1000)

srv := &http.Server{Handler: mux, ReadHeaderTimeout: 2 * time.Second}
log.Fatal(srv.Serve(ln))

Dropping work that is already too late

A queue that filled up during an incident is full of jobs nobody is waiting for any more. Carrying the deadline on the job lets the worker discard them at dequeue time for almost no cost, which drains the backlog instead of grinding through it. Counting the drops matters: shed work is invisible otherwise, and a dashboard that only shows errors will look healthy.

Example · go
var droppedTotal atomic.Int64

type job struct {
	payload  []byte
	deadline time.Time
}

// worker abandons jobs whose answer is no longer wanted.
func worker(jobs <-chan job) {
	for j := range jobs {
		if time.Now().After(j.deadline) {
			droppedTotal.Add(1) // abandoned, not failed: count it separately
			continue
		}
		handle(j)
	}
}

Backpressure to a message broker

The bounded channel is what creates the backpressure. When the 64 slots are full the send blocks, the loop stops reading from the broker, and unprocessed messages stay the broker's problem rather than your heap. Acknowledging only after the job is safely queued is the other half: acknowledge first and a crash loses the message silently.

Example · go
// A bounded queue: when it is full the reader blocks, and that is the point.
jobs := make(chan job, 64)

for msg := range messages { // messages streamed from the broker
	jobs <- job{
		payload:  msg.Body,
		deadline: time.Now().Add(5 * time.Second),
	}
	// Acknowledge only once the job is queued, never before.
	if err := msg.Ack(); err != nil {
		log.Printf("ack: %v", err)
	}
}

Related

go rate limitinggo bulkheadsgo k8s probesgo job queuesgo status codes

Discussion

  • Be the first to comment on this lesson.