Retries, Backoff and Idempotency

Retry transient failures with exponential backoff and jitter, and only where repeating the call is safe.

Syntaxselect { case <-ctx.Done(): return ctx.Err(); case <-time.After(backoff): }

Calls over a network fail in two ways. Some failures are transient: a dropped connection, a 503, a throttling error. The same call a moment later succeeds. Others are permanent: a malformed request, a missing record, a declined card. Repeating those only wastes time and money. A retry policy is the rule that separates the two.

Backoff and jitter

Retrying immediately makes an overloaded service worse. Exponential backoff waits longer each round: 100ms, then 200ms, 400ms, 800ms. Cap both the wait and the number of attempts.

If a thousand clients fail at the same instant and all back off by identical amounts, they retry in unison and arrive as waves. Jitter breaks up the pattern by randomising the wait. Full jitter picks a random duration between zero and the current cap, which spreads the load evenly.

Every wait must be cancellable. Use select over ctx.Done() and time.After rather than a bare time.Sleep, or a cancelled request keeps sleeping on a goroutine nobody wants. Keep the whole sequence inside the caller's deadline too: four attempts with growing waits easily outlast the timeout you were given.

Only retry what is safe

A retry is safe when doing the operation twice leaves the same result as doing it once. Reading is safe. Setting a value to X is safe. Adding 1, charging a card and sending an email are not, because a timeout never tells you whether the work happened on the other side. Make such a call safe first by sending an idempotency key the server remembers, then retry it.

The AWS SDK for Go v2 already does this for its own calls, with 3 attempts by default and a backoff capped at 20 seconds.

Example

Example · go
package main

import (
	"context"
	"errors"
	"fmt"
	"time"
)

// errBusy marks a failure worth retrying. Every other error is permanent.
var errBusy = errors.New("busy")

// do calls f until it succeeds, runs out of attempts, or the context ends.
// The wait doubles each round; the next example adds jitter to it.
func do(ctx context.Context, attempts int, f func() error) error {
	var err error
	for i := 1; i <= attempts; i++ {
		// Shorthand for: err = f(); if err == nil { return nil }
		if err = f(); err == nil {
			return nil
		}
		if !errors.Is(err, errBusy) {
			return err // permanent: hand it back untouched
		}
		if i == attempts {
			break
		}
		wait := 100 * time.Millisecond << (i - 1) // 100ms, 200ms, 400ms
		select {
		case <-ctx.Done():
			return ctx.Err() // the caller gave up while we were waiting
		case <-time.After(wait):
		}
	}
	return fmt.Errorf("after %d attempts: %w", attempts, err)
}

func main() {
	calls := 0
	err := do(context.Background(), 4, func() error {
		calls++
		if calls < 3 {
			return errBusy
		}
		return nil
	})
	fmt.Println("calls:", calls, "err:", err)

	calls = 0
	err = do(context.Background(), 4, func() error {
		calls++
		return errors.New("invalid card number")
	})
	fmt.Println("calls:", calls, "err:", err)
}

// Output:
// calls: 3 err: <nil>
// calls: 1 err: invalid card number

When to use it

  • Ride out a dependency's rolling restart instead of failing every request during it.
  • Back off when a third-party API answers 429 rather than hammering it harder.
  • Retry a payment capture safely by sending the same idempotency key each time.
  • Fail fast on a validation error that would be rejected identically on every attempt.

More examples

Full jitter

Swap this in wherever many clients can fail at once, which is any shared dependency. Without it, every instance of your service retries on the same schedule after an outage and the recovering dependency is knocked over again. Full jitter, shown here, sometimes waits almost nothing; the common variant called equal jitter returns half the window plus a random half, keeping a guaranteed minimum pause. The overflow check matters: a Duration is a 64-bit count of nanoseconds, and shifting left enough times turns it negative.

Example · go
// jitter returns a random wait in [0, window), where the window doubles per
// attempt and stops growing at maxWait. rand.N is from math/rand/v2, added
// in Go 1.22; it needs a value above zero, so base and maxWait must be too.
func jitter(attempt int, base, maxWait time.Duration) time.Duration {
	window := base << (attempt - 1)
	if window > maxWait || window <= 0 { // <= 0 means the shift overflowed
		window = maxWait
	}
	return rand.N(window)
}

// Attempt 1 waits somewhere in 0-100ms, attempt 2 in 0-200ms, and so on,
// never past 2s:
//
//	wait := jitter(attempt, 100*time.Millisecond, 2*time.Second)

Changing what the AWS SDK retries

Reach for this when the defaults fight your own deadlines. retry.DefaultMaxAttempts is 3 and retry.DefaultMaxBackoff is 20 seconds, so one unlucky S3 call can sit there for most of a minute, long past the point where your HTTP handler should have returned. Lowering MaxBackoff keeps the total inside your request budget. Note the Retryer field is a function: the SDK calls it to get a fresh retryer per operation, because retry state is per call.

Example · go
cfg, err := config.LoadDefaultConfig(ctx,
	// Five attempts rather than the default three.
	config.WithRetryMaxAttempts(5),
	// Adaptive mode also slows the client down when the service throttles it.
	config.WithRetryMode(aws.RetryModeAdaptive),
)
if err != nil {
	return err
}

// A custom retryer instead. It replaces both options above, which are then
// ignored, so set MaxAttempts here as well.
impatient := retry.NewStandard(func(o *retry.StandardOptions) {
	o.MaxAttempts = 5
	o.MaxBackoff = 2 * time.Second // the default cap is 20 seconds
})
cfg.Retryer = func() aws.Retryer { return impatient }

An idempotency key makes a POST retryable

This is the pattern for any outbound call that changes something. Two details decide whether it is correct. The key is generated once by the caller, for the business action, and reused by every attempt; generate it inside the closure and each retry becomes a new charge. And the request body is rebuilt per attempt, because a reader can only be read through once. Wrapping the transient cases with errBusy is what tells do to retry a 503 and give up on a 400.

Example · go
// charge sends the same Idempotency-Key on every attempt. The payment API
// remembers the key, so a retry after a timeout returns the first result
// instead of taking the money twice.
func charge(ctx context.Context, c *http.Client, body []byte, key string) error {
	return do(ctx, 3, func() error {
		// A fresh reader per attempt: the previous one is already consumed.
		req, err := http.NewRequestWithContext(ctx, http.MethodPost,
			"https://api.example/charges", bytes.NewReader(body))
		if err != nil {
			return err
		}
		req.Header.Set("Idempotency-Key", key)
		req.Header.Set("Content-Type", "application/json")

		res, err := c.Do(req)
		if err != nil {
			return fmt.Errorf("%w: %v", errBusy, err) // network failure: retry
		}
		defer res.Body.Close()

		switch {
		case res.StatusCode >= 500:
			return fmt.Errorf("%w: status %d", errBusy, res.StatusCode)
		case res.StatusCode >= 400:
			return fmt.Errorf("status %d", res.StatusCode) // our fault: stop
		}
		return nil
	})
}

Related

go timeout budgetsgo circuit breakergo context timeoutgo job idempotencygo aws sdk

Discussion

  • Be the first to comment on this lesson.