At Least Once, and What It Costs You

Brokers cannot offer exactly-once delivery, so every consumer must be safe to run twice with the same message.

Syntaxreceive -> process -> acknowledge // a crash anywhere before the ack means redelivery

Every broker you are likely to use offers at-least-once delivery, and the reason is not laziness. A consumer and a broker exchange two things: the delivery and the acknowledgement. If the ack is lost, the broker cannot distinguish "the consumer never processed it" from "the consumer processed it and then died". It has two choices — assume success and risk losing the work (at-most-once), or assume failure and risk doing it twice (at-least-once). Nothing in the network lets it be right every time.

The consumer carries the cost

At-least-once obliges every handler to be safe to run twice with the same input. That is the contract, not an optimisation to add after the first duplicate charge. The usual implementation is an idempotency key derived from the message — never a fresh random value, because a redelivery must produce the same key — recorded under a unique constraint in the same transaction as the side effect. Insert the key; if the insert conflicts, the work is already done, so acknowledge and move on.

What "exactly-once" means when a product claims it

Kafka's exactly-once semantics cover a read-process-write loop inside Kafka: the idempotent producer discards duplicate retries per partition, and transactions commit the output and the consumer offset together. Since Kafka 3.0, enable.idempotence defaults to true. None of it extends to the charge you make at Stripe or the row you write in MySQL. Those you make idempotent yourself.

Example

import { pool } from "./db.js";

// The key must be derivable from the message. A uuid generated per attempt would
// differ on a redelivery, which defeats the entire mechanism.
function idempotencyKey(body) {
  return `${body.type}:${body.orderId}`;
}

export async function handleOrderPlaced(message) {
  const body = JSON.parse(message.Body);
  const key = idempotencyKey(body);
  const client = await pool.connect();

  try {
    await client.query("BEGIN");

    // processed_messages has key as its primary key. ON CONFLICT DO NOTHING makes
    // the second insert a no-op, and rowCount tells us which case we are in.
    const claim = await client.query(
      `INSERT INTO processed_messages (key, processed_at)
       VALUES ($1, now())
       ON CONFLICT (key) DO NOTHING`,
      [key]
    );

    if (claim.rowCount === 0) {
      await client.query("ROLLBACK");
      return { duplicate: true };   // already done: acknowledge, do not redo
    }

    // The side effect and the claim commit together. There is no window in which
    // the key is recorded but the work was not done, or the reverse.
    await client.query(
      "UPDATE orders SET fulfilment_state = 'queued' WHERE id = $1",
      [body.orderId]
    );

    await client.query("COMMIT");
    return { duplicate: false };
  } catch (err) {
    await client.query("ROLLBACK");
    // No acknowledgement: the message becomes visible again and is redelivered.
    throw err;
  } finally {
    client.release();
  }
}

When to use it

Stop a redelivered order event from charging the customer a second time.
Make a webhook consumer safe when the sender retries after a timeout.
Decide whether a feature needs FIFO deduplication or just an idempotent handler.
Explain to a reviewer why a random uuid is not an idempotency key.

More examples

Producer-side deduplication in SQS FIFO: longhand and shorthand

Deduplication at the producer protects against a different failure from the consumer's idempotency key: a SendMessage that times out after the broker accepted it. Use the explicit form whenever the natural key is not the whole body — an order id, a payout id — because content hashing makes any incidental difference, such as an embedded timestamp, defeat the dedup entirely. Neither form removes the need for an idempotent consumer: the window is five minutes, and redelivery can happen hours later.

import { createHash } from "node:crypto";
import { SendMessageCommand } from "@aws-sdk/client-sqs";

const body = JSON.stringify({ type: "payout.requested", payoutId });

// LONGHAND: you compute the deduplication id. Two sends with the same id inside
// the five-minute deduplication interval put exactly one message on the queue, so
// a producer that retries after a timeout cannot create a second payout.
await sqs.send(new SendMessageCommand({
  QueueUrl: FIFO_QUEUE_URL,
  MessageBody: body,
  MessageGroupId: payoutId,
  MessageDeduplicationId: createHash("sha256").update(body).digest("hex"),
}));

// SHORTHAND: set ContentBasedDeduplication = true on the queue and omit the field.
// SQS then derives the id by hashing the message body for you. Same mechanism,
// same five-minute window - with one difference: it hashes the body only, so two
// messages that differ solely in their attributes are treated as duplicates.
await sqs.send(new SendMessageCommand({
  QueueUrl: FIFO_QUEUE_URL,
  MessageBody: body,
  MessageGroupId: payoutId,
}));

An idempotent call to a third party

The general shape for any external side effect: find the provider's idempotency mechanism and feed it a key derived from the message. Payment APIs, transactional email and most webhook senders have one. Where a provider has none — an SMTP server, for instance — the fallback is to record the attempt in your own database before the call and check it after, accepting that a crash in the middle may send one duplicate email.

// The handler may run twice; the charge must not. Stripe deduplicates on the
// Idempotency-Key header, so the key must come from the message - never from
// Date.now() or a uuid generated in the handler.
const key = `charge:${body.orderId}`;

const intent = await stripe.paymentIntents.create(
  {
    amount: body.amountCents,
    currency: "usd",
    customer: body.stripeCustomerId,
  },
  { idempotencyKey: key }   // the node library sends this as the Idempotency-Key header
);

// On the second run Stripe replays the original response instead of charging
// again, so the handler can treat both runs identically. Record intent.id against
// the order in the same transaction as your own idempotency row, or you will call
// Stripe a third time to find out what happened.

The table the consumer deduplicates against

The deduplication store is a real operational object: it needs a retention policy, and its retention has to outlast the broker's or the guarantee quietly lapses. Use this when the side effect has no natural unique key of its own. When it does — an orders row keyed by request id, a payment keyed by intent id — prefer that constraint and skip the extra table, since one unique index is cheaper than two things that must agree.

CREATE TABLE processed_messages (
  key          text        PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);

-- Needed only for pruning; the primary key serves the lookup.
CREATE INDEX processed_messages_processed_at_idx ON processed_messages (processed_at);

-- The table grows forever unless you prune it. Keep the window comfortably longer
-- than the queue's retention period (SQS: 4 days by default, 14 at most), so a
-- message that sat in the queue for days cannot find its key already deleted.
DELETE FROM processed_messages
 WHERE processed_at < now() - interval '30 days';

Discussion

Be the first to comment on this lesson.

The consumer carries the cost

What "exactly-once" means when a product claims it

Example

When to use it

More examples

Producer-side deduplication in SQS FIFO: longhand and shorthand

An idempotent call to a third party

The table the consumer deduplicates against

Related

Discussion