Graceful Degradation
When a non-essential dependency fails, return a reduced but honest answer instead of turning the whole request into an error.
if v, err := optional(ctx); err == nil { resp.Extra = v } else { resp.Degraded = true }Most responses are not one thing. A product page has a price, a description, stock, reviews and recommendations. If the recommendation service is down, returning a 500 throws away the four parts that worked. Graceful degradation means deciding in advance which parts are essential and which are not, and serving the reduced answer when an optional part fails.
Essential and optional
Sort every dependency into one of the two. Essential means the response is wrong or meaningless without it: the price, the user's identity. Optional means the response is thinner without it: reviews, related items, a personalised banner. Only optional dependencies may be degraded.
In code the pattern is small. Give the optional call its own short timeout, then ignore its error and fall back. if v, err := optional(ctx); err == nil { ... } is the shorthand for declaring v and err on their own line and then testing; both forms are identical, except that the short form scopes the variables to the if.
Say that you degraded
A thin answer that looks exactly like a full one hides the outage. Set a field or a header, log it, and count it as a metric. Otherwise your dashboards stay green through a dependency failure and nobody fixes it.
What trips people up
Never degrade a write, a payment or an authorisation check. If you cannot confirm something, say no. Degradation applies to reads where less information is still useful.
Example
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
)
type product struct {
Name string `json:"name"`
Price int `json:"price_cents"`
Recommended []string `json:"recommended,omitempty"`
Degraded bool `json:"degraded,omitempty"`
}
// recommendations stands in for an optional service that is currently down.
func recommendations(ctx context.Context) ([]string, error) {
return nil, errors.New("recommender: connection refused")
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
// The essential part of the answer.
p := product{Name: "Kettle", Price: 2499}
// The optional part gets a small slice of the budget and may fail.
recCtx, recCancel := context.WithTimeout(ctx, 20*time.Millisecond)
defer recCancel()
if recs, err := recommendations(recCtx); err == nil {
p.Recommended = recs
} else {
p.Degraded = true // label it, so the thin answer is visible
}
out, err := json.Marshal(p)
if err != nil {
panic(err)
}
fmt.Println(string(out))
}
// Output:
// {"name":"Kettle","price_cents":2499,"degraded":true}When to use it
- Render a product page without its recommendation block when the recommender is down.
- Serve a cached price from two minutes ago rather than a 500 while the pricing service restarts.
- Skip a personalisation call for every request once its circuit breaker has opened.
- Return search results without facet counts when the aggregation query times out.
More examples
Serving a stale value instead of an error
The strongest form of degradation is an old answer rather than no answer. A price from two minutes ago is usually better than a 500. The boolean return tells the caller the value is stale so it can label the response and shorten the cache header, and the warning log makes the outage visible. Reach for this on any read where slightly old data is acceptable.
// price returns the live price, or the last known one when pricing is down.
// The bool reports whether the value is stale.
func (s *Server) price(ctx context.Context, sku string) (int, bool, error) {
live, err := s.pricing.Get(ctx, sku)
if err == nil {
s.cache.Set(sku, live)
return live, false, nil
}
stale, ok := s.cache.Get(sku)
if !ok {
return 0, false, fmt.Errorf("price %s: %w", sku, err) // nothing to serve
}
s.log.Warn("serving stale price", "sku", sku, "err", err)
return stale, true, nil
}One failing leg out of two
errgroup cancels the whole group when any goroutine returns an error, which is right for the essential leg and wrong for the optional one. So the optional goroutine always returns nil and keeps its failure to itself. The two goroutines write to different fields of p, which are different memory locations, so there is no data race. Use this whenever a page fans out to a mix of essential and optional services.
func buildPage(ctx context.Context, id string) (page, error) {
var p page
g, gctx := errgroup.WithContext(ctx)
g.Go(func() error {
var err error
p.Order, err = loadOrder(gctx, id) // essential: its error fails the request
return err
})
g.Go(func() error {
revCtx, cancel := context.WithTimeout(gctx, 100*time.Millisecond)
defer cancel()
reviews, err := loadReviews(revCtx, id)
if err != nil {
p.Degraded = true
return nil // optional: never fail the group
}
p.Reviews = reviews
return nil
})
if err := g.Wait(); err != nil {
return page{}, err
}
return p, nil
}Degrading before you even call
Combining a breaker with degradation removes the cost entirely: while the circuit is open the optional call is skipped, so the page renders at full speed during the outage rather than waiting out a timeout on every request. The nil slice is the signal that the block is missing, and the template renders the page without it.
// While the recommender's circuit is open, skip the call and render without it.
var recs []Item
err := recBreaker.Do(func() error {
got, err := recommend(ctx, userID)
recs = got
return err
})
if err != nil {
recs = nil // down, or the circuit is open: the block is simply absent
degradedTotal.Add(1)
}
render(w, page{Items: items, Recommended: recs, Degraded: recs == nil})
Discussion