When Order Actually Matters

Global ordering is rarely the requirement; per-entity ordering is, and the ordering key is also your parallelism limit.

SyntaxSendMessage(QueueUrl, MessageBody, MessageGroupId) // FIFO: ordered within a group, parallel between groups

Global ordering across a queue is expensive and almost never the actual requirement. "Status updates must not arrive out of order" means per order, not across every order in the system. The useful unit is a key: all events for one entity ordered relative to each other, with no promise between keys.

How brokers express it

Kafka orders within a partition and nowhere else. A record with a key goes to (murmur2(keyBytes) & 0x7fffffff) % numPartitions, so the same key always lands in the same partition and keeps its order there. SQS FIFO queues use MessageGroupId: messages in one group are delivered one at a time in order, while different groups proceed in parallel. In both, the ordering key is also the unit of parallelism — that is the trade, stated in one sentence.

The hot key

Choose the tenant as the key and you get correct ordering plus a tenant whose work is single-threaded. One large customer then serialises behind itself while small customers fly through. Worse, in a FIFO group a failing message blocks everything behind it in that group until it is deleted or its visibility timeout expires, so one poison message stops one tenant completely. Keys should be as fine-grained as the invariant allows: the order, the document, the conversation; rarely the account.

The cheaper answer

Design events so arrival order does not matter. Carry the entity's version, write with WHERE version < $new, and let a stale event update zero rows instead of corrupting state.

Example

// The ordering key is the entity whose history must stay consistent: the order,
// not the tenant. A tenant-wide group serialises every order that customer will
// ever place, behind whichever one is slowest.
await sqs.send(new SendMessageCommand({
  QueueUrl: ORDER_EVENTS_FIFO_URL,
  MessageGroupId: `order:${order.id}`,                  // one group per order
  MessageDeduplicationId: `${order.id}:${event.version}`,  // stable across producer retries
  MessageBody: JSON.stringify({
    type: "order.status_changed",
    orderId: order.id,
    status: event.status,
    version: event.version,        // monotonic per order, assigned by the writer
  }),
}));

// The consumer still guards on version. FIFO gives ordered delivery, not ordered
// processing: a redelivery can arrive after a later event has already been applied,
// and two consumers can be mid-flight on two groups that touch the same row.
const { rowCount } = await pool.query(
  `UPDATE orders
      SET status = $1, status_version = $2
    WHERE id = $3
      AND status_version < $2`,    // a stale event updates zero rows
  [event.status, event.version, event.orderId]
);

if (rowCount === 0) {
  // Not an error. The event was superseded; acknowledge it and move on.
  metrics.increment("order_events.stale");
}

When to use it

Keep a single order's status transitions in sequence without serialising all orders.
Diagnose why one large tenant's jobs run one at a time while small tenants are fast.
Choose a Kafka message key for a chat application so a conversation reads correctly.
Replace a FIFO queue with version-guarded writes to lift a throughput ceiling.

More examples

Kafka: the key is the ordering decision

The partition count is a durable decision, not a tuning knob, and this example is mainly here for the last comment. Use a key wherever an entity has a history: a conversation, an account balance, a device's telemetry. Leave the key null for genuinely independent samples, where you want the broker to spread load however it likes.

import { Kafka } from "kafkajs";

const producer = new Kafka({ brokers: [process.env.KAFKA_BROKER] }).producer();
await producer.connect();

// With a key, the partition is (murmur2(keyBytes) & 0x7fffffff) % numPartitions, so
// every event for this conversation lands in one partition and keeps its order.
await producer.send({
  topic: "chat.messages",
  messages: [{ key: conversationId, value: JSON.stringify(message) }],
});

// With no key there is no ordering guarantee at all. Since Kafka 2.4 the producer
// also sticks consecutive keyless records to one partition per batch rather than
// round-robining record by record, which changes how they spread but not the
// guarantee: there still is none.
await producer.send({
  topic: "metrics.raw",
  messages: [{ value: JSON.stringify(sample) }],
});

// Adding partitions later rehashes keys. A key that lived in partition 3 may move
// to partition 7, so records for one key then exist in two partitions and their
// relative order is gone. Size partitions for future throughput, or key by
// something you can repartition deliberately.

Making out-of-order arrival harmless

The design move that makes most ordering requirements disappear, and therefore the first thing to try before reaching for a FIFO queue and its throughput ceiling. It costs one monotonic field on the entity — a version, a sequence number, or a server-assigned updated_at — and it removes the need for the broker to order anything. Reach for it for projections, caches, search indexes and any read model fed by events.

-- State-based events survive reordering. Delta-based events do not.
--
-- Delta: "add 3 to stock". Applied twice, the number is wrong. Applied out of
-- order, the intermediate values are wrong and nothing detects it.
UPDATE inventory SET quantity = quantity + 3 WHERE sku = 'ABC-1';

-- State: "quantity is 17 as of version 42". Idempotent and order-insensitive.
UPDATE inventory
   SET quantity = 17, version = 42
 WHERE sku = 'ABC-1'
   AND version < 42;
-- 0 rows updated means the event was stale. That is success, not an error, and the
-- consumer should acknowledge rather than retry.

-- The same guard with an upsert, for events that may arrive before the row exists:
INSERT INTO inventory (sku, quantity, version)
VALUES ('ABC-1', 17, 42)
ON CONFLICT (sku) DO UPDATE
   SET quantity = EXCLUDED.quantity, version = EXCLUDED.version
 WHERE inventory.version < EXCLUDED.version;

Splitting a hot group

The fix for the most common ordering incident: throughput that is fine in aggregate and unacceptable for one customer. The metric in the last comment is the diagnostic, because queue depth alone cannot distinguish "not enough consumers" from "not enough groups" and only the second is helped by scaling out. Apply the same reasoning to Kafka, where the equivalent symptom is one partition with consumer lag while the rest are idle.

// BEFORE: one group per tenant. A tenant importing 50,000 rows holds its single
// group for hours, and that tenant's interactive work queues behind the import.
const groupId = `tenant:${tenantId}`;

// AFTER: the group is the finest unit the invariant needs. Per-document ordering
// still holds, and the tenant's work spreads across as many consumers as it has
// documents.
const groupId2 = `tenant:${tenantId}:doc:${documentId}`;

// If the invariant really is tenant-wide, split by traffic class instead of by
// entity: bulk work and interactive work on separate queues, so a backlog in one
// cannot delay the other even though each stays ordered within itself.
const queueUrl = job.kind === "import" ? BULK_FIFO_URL : INTERACTIVE_FIFO_URL;

// Watch ApproximateNumberOfGroupsWithInflightMessages on a FIFO queue: if the
// backlog is large while that number stays small, the work is concentrated in a
// few groups and adding consumers will do nothing.

Discussion

Be the first to comment on this lesson.