Stampedes and Singleflight
When a hot key expires every request misses at once; singleflight collapses those identical loads into one shared call.
v, err, shared := g.Do(key, func() (any, error) { ... })A cache stampede is what happens when a popular key leaves the cache and every request that wanted it goes to the database at the same moment. It is also called a thundering herd. The cache was absorbing a thousand reads a second. For the half second after that key expires, all thousand arrive at the database at once, and a database is not sized for that.
It is not a slow leak, it is a cliff. The service is healthy, one key expires, and the database queue fills.
Collapsing duplicate work
Those thousand loads are identical, so only one of them needs to run. The package golang.org/x/sync/singleflight does exactly that, and is added with go get golang.org/x/sync. A singleflight.Group is ready to use as its zero value. Do(key, fn) runs fn only if no call for that key is already in flight; otherwise it waits for the one that is, and returns that call's result. The third return value, shared, is true when the result went to more than one caller.
Do hands the value back as any, a value of unknown type, because the group holds no type of its own. Assert it back with name := v.(string). The blocking Do is the short form of the channel form: res := <-g.DoChan(key, fn), then read res.Val, res.Err and res.Shared. Same work, one explicit wait.
What trips people up
Singleflight is not a cache. It remembers nothing once the call has returned, so two requests a second apart both load. Keep the cache, and use singleflight only to cover the gap after a miss. Spreading TTLs slightly stops a batch of keys expiring together in the first place.
Example
package main
import (
"fmt"
"sync"
"sync/atomic"
"time"
"golang.org/x/sync/singleflight"
)
// loads counts how many times the expensive work really ran.
var loads atomic.Int64
// loadUser stands for one slow database round trip.
func loadUser() (any, error) {
loads.Add(1)
time.Sleep(100 * time.Millisecond) // the window others pile up in
return "ada", nil
}
func main() {
var g singleflight.Group // the zero value is ready to use
var wg sync.WaitGroup
results := make([]string, 5)
for i := range results {
// wg.Go (Go 1.25 and later) is shorthand for:
// wg.Add(1); go func() { defer wg.Done(); ... }()
wg.Go(func() {
// All five ask for the same key at the same moment.
v, err, _ := g.Do("user:7", loadUser)
if err != nil {
panic(err)
}
results[i] = v.(string) // Do returns any, so assert the type back
})
}
wg.Wait()
fmt.Println("results:", results)
fmt.Println("loads:", loads.Load())
}
// Output:
// results: [ada ada ada ada ada]
// loads: 1When to use it
- Protect the database when the home page's cached payload expires at peak traffic.
- Collapse hundreds of concurrent requests for one product page into a single query.
- Stop a just-restarted pod from issuing the same query once per in-flight request.
- Share one upstream API call between every request that arrives while it is running.
More examples
Cache aside with singleflight in front of the miss
This is the shape to copy into a real service: cache for the steady state, singleflight for the moment after a miss. The second lookup inside the closure matters, because the first caller fills the cache while the others are still waiting, and without it they would each load again as they wake up. The group lives on the server struct, so the same key collapses across every request this process is handling.
// get checks the cache first; on a miss only one goroutine per key loads.
func (s *server) get(ctx context.Context, id int) (string, error) {
key := fmt.Sprintf("user:v1:%d", id)
if v, ok := s.cache.get(key); ok {
return v, nil
}
v, err, _ := s.sf.Do(key, func() (any, error) {
// Look again. A caller that waited here may find the leader has
// already filled the cache.
if v, ok := s.cache.get(key); ok {
return v, nil
}
// WithoutCancel (Go 1.21) detaches the load from whichever request
// happened to start it, so one client leaving cannot kill it.
name, err := s.loadName(context.WithoutCancel(ctx), id)
if err != nil {
return nil, err
}
s.cache.set(key, name, time.Minute)
return name, nil
})
if err != nil {
return "", err
}
return v.(string), nil
}DoChan, so one caller can give up alone
Do blocks until the shared call finishes, however long that is, so a hung load hangs every waiter with it. DoChan gives you the result on a channel instead, which a select can race against the request's own deadline. Use it on any path where a caller must honour a timeout. There is also Forget(key), which drops the in-flight record so the next caller starts fresh; reach for it when a load has returned something you do not want other callers to receive.
// loadName lets this caller stop waiting without cancelling the shared load.
func loadName(ctx context.Context, g *singleflight.Group, key string) (string, error) {
ch := g.DoChan(key, func() (any, error) {
v, err := fetch(context.WithoutCancel(ctx), key)
return v, err
})
select {
case res := <-ch:
if res.Err != nil {
return "", res.Err
}
// res.Shared is true when other callers got this same result.
return res.Val.(string), nil
case <-ctx.Done():
// Our request is over. The load carries on for the others.
return "", ctx.Err()
}
}Jitter, so keys do not expire together
Singleflight collapses many callers asking for one key. It does nothing for a thousand different keys that all expire in the same second, which is what a cache warmed in a loop, or refilled after a deploy, sets up. Ten per cent of jitter turns that one cliff into a slope. Apply it everywhere a TTL is set, not only at warm-up, since every normal refill is also a batch.
// Keys written in one burst would otherwise expire in one burst. Spreading
// the TTL spreads the misses out as well.
func jittered(base time.Duration) time.Duration {
// rand.N (math/rand/v2, Go 1.22) returns a value in [0, n).
return base + rand.N(base/10)
}
// Warming a thousand keys at start-up, each with its own expiry:
//
// for id, name := range all {
// c.set(userKey(id), name, jittered(5*time.Minute))
// }
Discussion