Retrying Without Making It Worse
Exponential backoff with jitter, a bounded attempt count, and a rule for which failures are worth retrying at all.
sleep = random(0, min(cap, base * 2 ** attempt)) // full jitter; 2 ** attempt is Math.pow(2, attempt)Most job failures are transient: a timeout, a 503, a deadlock, a dropped connection. Retrying is correct. Retrying badly is how a dependency that was briefly slow becomes a dependency that is down.
Backoff, and why jitter is not decoration
Fixed-interval retries stay in lockstep. Everything that failed at the same moment retries at the same moment, so load arrives in spikes exactly while the dependency is trying to recover. Exponential backoff spreads attempts apart in time; jitter spreads them apart from each other. The form AWS published is full jitter: sleep = random(0, min(cap, base * 2 ** attempt)). Drop the random and you have rebuilt the thundering herd with longer gaps between the stampedes.
Retryable or permanent
Classify before you retry. A timeout, a connection reset, a 429 or a 5xx deserves another attempt. A malformed payload, a failed validation or a reference to a deleted row will fail identically forever, and retrying it only spends the attempt budget that transient failures needed. Permanent failures should go to the dead-letter queue on the first attempt.
Bound the amplification
Retries multiply down a call chain: four attempts at each of three layers is sixty-four requests at the bottom for one user action. Google's SRE practice caps a request at three attempts and adds a per-client budget — retries may not exceed 10% of that client's requests — so a broad outage cannot turn into a load multiplier of your own making.
Example
const BASE_MS = 200;
const CAP_MS = 20_000;
const MAX_ATTEMPTS = 3; // SRE guidance: three attempts, then let the caller know
export class PermanentError extends Error {}
export function isRetryable(err) {
if (err instanceof PermanentError) return false;
if (err.name === "AbortError") return true; // our own timeout
if (["ECONNRESET", "ETIMEDOUT", "EAI_AGAIN"].includes(err.code)) return true;
// 408, 429 and 5xx are worth another attempt. Every other 4xx is a bug in the
// request and will fail the same way for ever.
return err.status === 408 || err.status === 429 || (err.status >= 500 && err.status <= 599);
}
// Full jitter, as published in the AWS architecture blog:
// sleep = random(0, min(cap, base * 2 ** attempt))
// 2 ** attempt is the exponentiation operator (ES2016); it is the same value as
// Math.pow(2, attempt).
export function fullJitterDelay(attempt) {
const ceiling = Math.min(CAP_MS, BASE_MS * 2 ** attempt);
return Math.random() * ceiling;
}
export async function withRetries(fn) {
for (let attempt = 0; ; attempt++) {
try {
return await fn(attempt);
} catch (err) {
if (!isRetryable(err) || attempt >= MAX_ATTEMPTS - 1) throw err;
log.warn({ err, attempt }, "retrying");
await new Promise((resolve) => setTimeout(resolve, fullJitterDelay(attempt)));
}
}
}When to use it
- Replace a fixed one-second retry loop that spikes a recovering service every second.
- Decide which exceptions should go straight to the dead-letter queue.
- Pick an attempt count for a payment job where every attempt is visible to the customer.
- Stop a nested service chain turning one failed request into dozens.
More examples
Let the broker hold the backoff
The idiomatic retry for queue work, as opposed to the in-process retry used for a single HTTP call. It matters most when the dependency is slow rather than broken: a hundred workers sleeping for thirty seconds is a hundred idle workers and a growing backlog, whereas a hundred released messages cost nothing. Combine the two — in-process retries for the fast transient blip, broker backoff for anything longer than a second or two.
// Sleeping inside a worker holds the lease, the thread and the database connection
// for the whole backoff. Cheaper: hand the message back with a delay and let the
// broker redeliver it later. The receive count is the attempt number.
const attempt = Number(message.Attributes.ApproximateReceiveCount);
try {
await processJob(JSON.parse(message.Body));
await sqs.send(new DeleteMessageCommand({
QueueUrl,
ReceiptHandle: message.ReceiptHandle,
}));
} catch (err) {
if (!isRetryable(err)) {
await deadLetter(message, err); // deliberate, with the reason attached
return;
}
const ceiling = Math.min(900, 2 * 2 ** attempt); // seconds; our cap, not the API's
await sqs.send(new ChangeMessageVisibilityCommand({
QueueUrl,
ReceiptHandle: message.ReceiptHandle,
VisibilityTimeout: Math.ceil(Math.random() * ceiling), // full jitter, in seconds
}));
}
// The worker is free immediately, and maxReceiveCount on the redrive policy still
// bounds the total number of attempts without any counter of your own.BullMQ: the policy is declared on the job
The declarative equivalent, and the shape most job libraries use: attempts and backoff live on the job rather than in the handler. Worth contrasting with the hand-rolled loop because the defaults differ from what you would choose — BullMQ's default is no retries at all, and failures are kept for ever unless removeOnFail is set. Reach for this when the queue is Redis-backed rather than a managed broker and you have no visibility timeout to manipulate.
import { Queue } from "bullmq";
const queue = new Queue("receipts", { connection: { host: "127.0.0.1", port: 6379 } });
// BullMQ applies the backoff itself between attempts, so the worker does not sleep.
// jitter is a factor from 0 to 1: with delay 3000 and jitter 0.5, each wait is a
// random value between half the computed delay and the whole of it. It is
// documented for BullMQ 5 - check the version in your lockfile before relying on it.
await queue.add(
"send-receipt",
{ orderId },
{
attempts: 5,
backoff: { type: "exponential", delay: 3000, jitter: 0.5 },
jobId: `receipt:${orderId}`, // stable id: adding it twice while it exists is a no-op
removeOnComplete: 1000, // keep the last 1000, or Redis grows for ever
removeOnFail: false, // keep failures: they are your dead-letter list
}
);
// { type: "fixed", delay: 1000 } is the other built-in strategy - the same delay
// every attempt, which is what you want for a dependency with a fixed rate limit.Bounding the amplification across layers
The control that distinguishes a retry policy from a load amplifier. Per-request limits alone still let a general outage triple your traffic, because every request is independently entitled to its three attempts. Add a budget to any client that calls a shared dependency — a database, an internal service, a payment provider — and make the retry decision consult it. The rolling window is the part people leave out and then find their budget permanently exhausted after one bad day.
// Google's SRE practice uses two controls: at most three attempts per request, and
// retries capped at 10% of a client's outgoing requests. Without the second,
// four attempts at each of three layers is 4^3 = 64 requests at the bottom for one
// user action - and the bottom is usually the database that is already overloaded.
class RetryBudget {
constructor(ratio = 0.1, windowMs = 10_000) {
this.ratio = ratio;
this.windowMs = windowMs;
this.requests = [];
this.retries = [];
}
#prune(now) {
const cutoff = now - this.windowMs;
// A rolling window, so a quiet hour does not pay for the current outage.
this.requests = this.requests.filter((t) => t > cutoff);
this.retries = this.retries.filter((t) => t > cutoff);
}
record() { const now = Date.now(); this.#prune(now); this.requests.push(now); }
tryRetry() {
const now = Date.now();
this.#prune(now);
if (this.retries.length >= this.ratio * this.requests.length) return false;
this.retries.push(now);
return true;
}
}
// The important property: when nearly everything is failing, the budget is already
// spent, so the system degrades to one attempt per request instead of four.
Discussion