Where Poison Messages Go

A dead-letter queue takes messages that fail every attempt, so they stop consuming workers and start being investigated.

SyntaxRedrivePolicy = {"deadLetterTargetArn": "arn:aws:sqs:...:jobs-dlq", "maxReceiveCount": 5}

A poison message is one that will fail every time: a payload the handler cannot parse, a reference to a row somebody deleted, a bug that throws on one tenant's data. Left on the main queue it is redelivered until the retention period expires — in SQS that is four days by default and fourteen at most — taking a worker slot on every cycle. A dead-letter queue is where it goes after a bounded number of receives.

Bounding the receives

In SQS the bound is the redrive policy's maxReceiveCount, valid from 1 to 1000. It counts deliveries, not exception types: after that many receives without a delete, the message is moved to deadLetterTargetArn. Five or ten is usual. One is tempting and almost always wrong, because then a single deploy restart dead-letters work that was merely interrupted.

Make the message understandable later

Automatic redrive moves the original message unchanged, so the DLQ contains no explanation of why anything failed. The DeadLetterQueueSourceArn system attribute tells you which queue it came from; everything else has to be recovered from logs correlated by message id. If you want the error, the stack, the attempt count and the build that failed attached to the message, catch the permanent failure yourself and publish an enriched copy.

Alarm and replay

Put the alarm on the DLQ's depth, not on the main queue's error rate, with a threshold of one. Then replay with a redrive task once the fix is deployed.

Example

Example · javascript
import {
  SQSClient,
  StartMessageMoveTaskCommand,
  ListMessageMoveTasksCommand,
} from "@aws-sdk/client-sqs";

const sqs = new SQSClient({});
const SourceArn = process.env.DLQ_ARN;

// Replay a dead-letter queue after the fix is deployed. Omit DestinationArn and
// each message goes back to the queue it originally failed on, which is what you
// want when one DLQ serves several source queues.
const { TaskHandle } = await sqs.send(new StartMessageMoveTaskCommand({
  SourceArn,
  // Throttle it. The consumers and their database absorb this on top of live
  // traffic, and a DLQ with 200k messages will otherwise arrive all at once.
  // 500 messages per second is the maximum SQS accepts here.
  MaxNumberOfMessagesPerSecond: 50,
}));

log.info({ TaskHandle }, "redrive started");

// Only one move task may run per queue at a time, so check before starting another.
// Cancelling a task stops further movement; it does not put back what already moved.
const { Results } = await sqs.send(new ListMessageMoveTasksCommand({
  SourceArn,
  MaxResults: 1,
}));
const task = Results[0];
log.info({
  status: task.Status,                                  // RUNNING | COMPLETED | CANCELLED | FAILED
  moved: task.ApproximateNumberOfMessagesMoved,
  toMove: task.ApproximateNumberOfMessagesToMove,
});

// Before replaying: confirm the fix handles these messages. A replay into the same
// bug refills the DLQ and spends another maxReceiveCount of worker time doing it.

When to use it

  • Stop one unparseable message from consuming a worker every thirty seconds for four days.
  • Decide maxReceiveCount for a queue whose consumers restart on every deploy.
  • Capture the stack trace and build sha alongside a failed job for later investigation.
  • Replay 40,000 dead-lettered messages at a rate the database can absorb.

More examples

Configuring the queue pair

The configuration that is easy to get subtly wrong: a stringified policy, a maxReceiveCount of 1 that dead-letters perfectly good work, and a DLQ with the default four-day retention that empties itself before Monday. Set this up with the queue, not after the first incident, because a queue without a redrive policy has no bound on redelivery at all.

Example · javascript
import { SetQueueAttributesCommand } from "@aws-sdk/client-sqs";

await sqs.send(new SetQueueAttributesCommand({
  QueueUrl: JOBS_QUEUE_URL,
  Attributes: {
    // RedrivePolicy is a JSON *string*, not a nested object. Passing an object
    // here is a validation error, and the message names the attribute rather than
    // the shape, so it reads like a permissions problem.
    RedrivePolicy: JSON.stringify({
      deadLetterTargetArn: DLQ_ARN,
      maxReceiveCount: 5,   // 1-1000; 5 survives a deploy restart, 1 does not
    }),
    VisibilityTimeout: "60",         // every attribute value is a string
    ReceiveMessageWaitTimeSeconds: "20",
  },
}));

// Give the DLQ a longer retention period than the source queue (up to 14 days), or
// the evidence expires over a long weekend before anyone investigates.
// The DLQ must be the same type as its source: FIFO sources need a FIFO DLQ.

Dead-lettering deliberately, with the reason attached

The difference between a DLQ you can act on and a pile of JSON nobody can interpret three weeks later. Use it wherever the handler can tell that retrying is pointless: a schema validation failure, a 404 from a dependency, an unsupported file type. The two rules that make replay possible are keeping the original body untouched and recording the build that failed, so you can tell which messages a fix actually covers.

Example · javascript
// Automatic redrive moves the original bytes, so the DLQ holds no explanation. For
// failures you can recognise as permanent, publish an enriched record yourself and
// delete the original, so it stops costing receive cycles immediately.
async function deadLetter(message, err) {
  await sqs.send(new SendMessageCommand({
    QueueUrl: process.env.FAILED_JOBS_QUEUE_URL,
    MessageBody: JSON.stringify({
      originalBody: message.Body,      // verbatim, so a replay is still possible
      messageId: message.MessageId,
      receiveCount: Number(message.Attributes.ApproximateReceiveCount),
      error: { name: err.name, message: err.message, stack: err.stack },
      traceId: currentTraceId(),       // the link back to the logs
      handlerVersion: process.env.GIT_SHA,   // which build failed
      failedAt: new Date().toISOString(),
    }),
  }));

  await sqs.send(new DeleteMessageCommand({
    QueueUrl,
    ReceiptHandle: message.ReceiptHandle,
  }));
}

// Keep originalBody as a string and never reshape it. A replay has to be able to
// put exactly the original message back on the source queue.

The alarm that belongs on the DLQ

The one alarm every queue should have. It is cheap, it has no false positives by construction, and it catches the class of bug that error-rate alarms miss entirely — a handler that fails on 0.01% of messages, which is invisible in a rate and obvious as a count. TreatMissingData is the detail that decides whether the alarm works at all, because SQS stops publishing metrics for a queue that has had no API activity for six hours, so a quiet DLQ reports no data rather than a zero.

Example · javascript
import { CloudWatchClient, PutMetricAlarmCommand } from "@aws-sdk/client-cloudwatch";

const cw = new CloudWatchClient({});

// Alarm on the DLQ's depth. Any message here means a message that no number of
// retries could process, which is worth waking someone for even at a count of one.
await cw.send(new PutMetricAlarmCommand({
  AlarmName: "jobs-dlq-not-empty",
  AlarmDescription: "A job failed every attempt. Investigate before replaying.",
  Namespace: "AWS/SQS",
  MetricName: "ApproximateNumberOfMessagesVisible",
  Dimensions: [{ Name: "QueueName", Value: "jobs-dlq" }],
  Statistic: "Maximum",
  Period: 300,
  EvaluationPeriods: 1,
  Threshold: 0,
  ComparisonOperator: "GreaterThanThreshold",
  // SQS stops publishing metrics for a queue with no API activity for six hours,
  // so an idle DLQ reports no data rather than zero and the default treatment
  // would leave this alarm permanently in INSUFFICIENT_DATA.
  TreatMissingData: "notBreaching",
  AlarmActions: [process.env.ONCALL_TOPIC_ARN],
}));

Related

sd queue retriessd queue visibilitysd queue failuresd logs metrics tracessd queue delivery

Discussion

  • Be the first to comment on this lesson.