In-Flight, Redelivered, Duplicated
A queue leases a message rather than handing it over, and a lease that expires mid-job creates a second worker on the same work.
ChangeMessageVisibility(QueueUrl, ReceiptHandle, VisibilityTimeout) // 0-43200 seconds, counted from the callA queue does not give a message to a worker; it leases it. On receive, the message stays in the queue but becomes invisible to other consumers for the visibility timeout — in Amazon SQS 30 seconds by default, 0 at minimum, 12 hours at maximum. Delete it inside that window and it is gone. Let the window close and the message becomes visible again, and the next receive hands the same work to a second worker while the first is still running.
Size the lease from the slow case
Timeouts get set from the median duration, which guarantees the slow tail is processed twice. Measure the p99 of the job. If the spread is wide, heartbeat instead: call ChangeMessageVisibility every few seconds while the work runs. The new timeout counts from the moment of the call, not from the receive, so renewing at half the lease leaves room for one failed renewal.
The duplicate that is not a bug
The classic case contains no exception at all. The handler does its write, the commit takes 40 seconds under load, the 30-second lease expires, a second worker picks the message up, and the work happens twice. Nothing threw. ApproximateReceiveCount on the received message tells a handler how many times this message has been delivered, which is useful for logging and for deciding to give up — but recognising a redelivery is not the same as being safe. Only idempotence is.
Example
import {
SQSClient,
ReceiveMessageCommand,
DeleteMessageCommand,
ChangeMessageVisibilityCommand,
} from "@aws-sdk/client-sqs";
const sqs = new SQSClient({});
const QueueUrl = process.env.JOBS_QUEUE_URL;
const LEASE_SECONDS = 60;
async function receiveOne() {
const out = await sqs.send(new ReceiveMessageCommand({
QueueUrl,
MaxNumberOfMessages: 1,
WaitTimeSeconds: 20, // long polling; 20 is the maximum SQS allows
VisibilityTimeout: LEASE_SECONDS, // per-receive override of the queue default
// AttributeNames is deprecated in favour of MessageSystemAttributeNames.
MessageSystemAttributeNames: ["ApproximateReceiveCount"],
}));
// out.Messages?.[0] is shorthand for: out.Messages == null ? undefined : out.Messages[0]
return out.Messages?.[0];
}
export async function runOnce() {
const message = await receiveOne();
if (!message) return;
const attempt = Number(message.Attributes.ApproximateReceiveCount);
// Renew at half the lease, so one failed renewal does not lose it.
const heartbeat = setInterval(() => {
sqs.send(new ChangeMessageVisibilityCommand({
QueueUrl,
ReceiptHandle: message.ReceiptHandle,
VisibilityTimeout: LEASE_SECONDS, // counted from now, not from the receive
})).catch((err) => log.warn({ err }, "lease renewal failed"));
}, (LEASE_SECONDS / 2) * 1000);
try {
await processJob(JSON.parse(message.Body), { attempt });
// Delete only after the work has committed. Deleting first turns any crash
// into lost work.
await sqs.send(new DeleteMessageCommand({
QueueUrl,
ReceiptHandle: message.ReceiptHandle,
}));
} finally {
clearInterval(heartbeat);
}
}When to use it
- Choose a visibility timeout for a job whose p99 is twelve times its median.
- Add a heartbeat to a video transcode so it is not handed to a second worker.
- Explain a duplicate charge that happened with no error in the logs.
- Release leased messages immediately when a worker receives SIGTERM.
More examples
Giving a message back instead of holding it
Shows that the lease is a two-way control, not just something that expires. Returning a message deliberately shortens recovery from minutes to nothing during a deploy, and it is the correct response to "this job is not ready" — far better than failing it towards the dead-letter queue. The backoff variant matters because every release costs a receive, and receives are what maxReceiveCount counts.
// Setting VisibilityTimeout to 0 terminates the lease immediately, so the message
// is available to another consumer at once rather than after the timeout.
//
// Two good uses: a worker that is shutting down and will not finish, and a worker
// that discovers it cannot do this job yet (a dependency row has not replicated).
async function release(message) {
await sqs.send(new ChangeMessageVisibilityCommand({
QueueUrl,
ReceiptHandle: message.ReceiptHandle,
VisibilityTimeout: 0,
}));
}
// Use with care in the "not ready yet" case: releasing at 0 produces a hot loop
// between your workers and the queue, each receive counting towards
// maxReceiveCount. Release with a backoff instead.
async function releaseLater(message, attempt) {
const ceiling = Math.min(300, 2 * 2 ** attempt); // seconds
await sqs.send(new ChangeMessageVisibilityCommand({
QueueUrl,
ReceiptHandle: message.ReceiptHandle,
VisibilityTimeout: Math.ceil(Math.random() * ceiling),
}));
}The lease you do not control: Lambda
Managed consumers hide the lease, which means the duplicate work shows up as an unexplained double-send rather than as an obvious timeout. These two settings are the ones people discover after the incident. The partial batch response is also the only way to use batches of ten without making every failure cost nine redeliveries, so reach for it on any Lambda-plus-SQS consumer that does not process one message at a time.
// With an SQS event source mapping, Lambda receives, leases and deletes for you.
// Two configuration facts decide whether it behaves:
//
// 1. The queue's visibility timeout should be at least six times the combined
// total of the function timeout and MaximumBatchingWindowInSeconds. Lower, and
// Lambda's own retries hand the batch to a second invocation while the first
// still runs.
// 2. The mapping needs FunctionResponseTypes: ["ReportBatchItemFailures"], or one
// bad message in a batch of ten redelivers the nine that succeeded.
export const handler = async (event) => {
const batchItemFailures = [];
for (const record of event.Records) {
try {
await processJob(JSON.parse(record.body));
} catch (err) {
log.error({ err, messageId: record.messageId }, "job failed");
// Returning the identifier keeps only this message on the queue; the others
// are deleted by Lambda as successes.
batchItemFailures.push({ itemIdentifier: record.messageId });
}
}
return { batchItemFailures };
};Aggregated client and command client: the same request
Worth knowing because both forms appear constantly in blog posts and in the same repository, and a reader who has only seen one assumes the other is a different SDK version. Choose one per service and stay with it. The command form is the one to standardise on if you bundle for Lambda, since tree-shaking can drop the operations you never call.
// LONGHAND: the command form, which every AWS SDK v3 example uses.
import { SQSClient, ReceiveMessageCommand } from "@aws-sdk/client-sqs";
const client = new SQSClient({});
const a = await client.send(new ReceiveMessageCommand({ QueueUrl, MaxNumberOfMessages: 10 }));
// SHORTHAND: the aggregated client exposes one method per operation. It builds the
// same command and sends it through the same client, so the request on the wire is
// identical and so is every parameter name.
import { SQS } from "@aws-sdk/client-sqs";
const sqs = new SQS({});
const b = await sqs.receiveMessage({ QueueUrl, MaxNumberOfMessages: 10 });
// The cost of the shorthand: importing SQS pulls in every operation, so code that
// cares about bundle size keeps the command form. Mixing both in one codebase is
// the real problem - a grep for "ReceiveMessageCommand" then misses half the call
// sites.
Discussion