How Many Workers
Arrival rate times service time gives the concurrency you need; queue depth and age tell you whether you have it.
concurrency = ceil(arrivalRate * serviceTime) // Little's Law: L = lambda * WTwo numbers decide whether a queue is healthy: how deep it is and how old its oldest message is. Depth alone says nothing — ten thousand messages draining in twenty seconds is fine, two hundred that have been waiting an hour is an incident. Age is the number that maps to a promise you made to a user, so age is what to alarm on.
How many workers
Little's Law gives the concurrency: arrival rate multiplied by service time. 500 messages per second at 200 ms each means 100 messages in flight, so 100 concurrent handlers merely to break even — and breaking even leaves nothing spare for a backlog. Size from the peak arrival rate and the p99 service time, then add headroom, because a fleet sized from averages is a fleet that falls behind exactly when anyone is watching.
How long a backlog takes to drain
Drain time is the backlog divided by the difference between service rate and arrival rate. At 500 arriving and 600 served, a backlog of a million messages takes around 10,000 seconds — nearly three hours. If that difference is zero or negative, the backlog never drains and the only answers are more capacity or less work.
The real ceiling
Consumers often cannot be scaled to the number Little's Law asks for, because each one holds a database connection. The pool, the primary's write throughput or a third-party rate limit caps the fleet long before the queue does. Measure where the service time goes before adding workers.
Example
// Little's Law: the concurrency you need is arrival rate x service time. Use the
// peak arrival rate and the p99 service time; sizing from averages guarantees a
// backlog at the only moments anyone cares.
export function sizeFleet({ arrivalPerSecond, serviceSeconds, utilisation = 0.7 }) {
const inFlight = arrivalPerSecond * serviceSeconds; // L = lambda * W
return Math.ceil(inFlight / utilisation); // headroom for the tail
}
sizeFleet({ arrivalPerSecond: 500, serviceSeconds: 0.2 });
// 143 concurrent handlers. At 100% utilisation it would be 100, which is the
// number at which the queue never empties and every latency spike becomes a
// backlog.
// How long an existing backlog takes to clear with the fleet you have.
export function drainSeconds({ backlog, servedPerSecond, arrivalPerSecond }) {
const net = servedPerSecond - arrivalPerSecond;
if (net <= 0) return Infinity; // it never drains: shed load or add capacity
return backlog / net;
}
drainSeconds({ backlog: 1_000_000, servedPerSecond: 600, arrivalPerSecond: 500 });
// 10,000 seconds - just under three hours. If the recovery target is 15 minutes,
// 600/s is not a fleet that can recover from this backlog, whatever it does in
// steady state. Capacity for recovery is a separate number from capacity for
// steady state, and it is the larger one.When to use it
- Decide how many consumers a new 500-messages-per-second queue needs.
- Answer "how long until this backlog clears" during an incident, with a number.
- Replace CPU-based autoscaling on an IO-bound consumer that never scales out.
- Show that adding workers cannot help because the database connection pool is the limit.
More examples
Reading the two numbers
The diagnostic pass before touching any scaling setting. The combination of depth and in-flight distinguishes the three cases that look identical on a depth graph: not enough workers, workers that are stuck, and workers that are not consuming at all. Each has a different fix and only the first one is solved by scaling out.
import { GetQueueAttributesCommand } from "@aws-sdk/client-sqs";
// The queue attribute names and the CloudWatch metric names differ for the same
// quantity: GetQueueAttributes calls it ApproximateNumberOfMessages, CloudWatch
// publishes it as ApproximateNumberOfMessagesVisible. Scripts and dashboards
// disagree about this constantly.
const { Attributes } = await sqs.send(new GetQueueAttributesCommand({
QueueUrl,
AttributeNames: [
"ApproximateNumberOfMessages", // waiting to be received
"ApproximateNumberOfMessagesNotVisible", // in flight, leased by a worker
"ApproximateNumberOfMessagesDelayed", // not yet eligible
],
}));
const depth = Number(Attributes.ApproximateNumberOfMessages);
const inFlight = Number(Attributes.ApproximateNumberOfMessagesNotVisible);
// inFlight close to your worker count means the fleet is saturated; inFlight near
// zero with a large depth means the consumers are not receiving at all - a
// permissions error or a crash loop, not a capacity problem.
//
// Age has no queue attribute. It exists only as the CloudWatch metric
// ApproximateAgeOfOldestMessage, and it is the one to alarm on, because it is the
// only number expressed in the same units as a user-facing promise.Scaling on backlog per worker
The formula that turns a latency objective into an autoscaler target, which is the only honest way to pick one. Both halves come from measurements you should already have: the acceptable latency from the objective you promised, the average service time from your own metrics. Recompute the target whenever the handler's service time changes materially, because a faster handler means the same latency tolerates a larger backlog per worker.
// The documented way to autoscale on a queue: do not target raw depth, because the
// number of messages does not move proportionally with the size of the fleet.
// Target backlog per worker instead.
//
// backlogPerWorker = ApproximateNumberOfMessages / runningWorkers
// acceptableBacklogPerWorker = acceptableLatencySeconds / averageServiceSeconds
//
// AWS's own example: 1,500 messages, 10 instances, 0.1s per message, 10s of
// acceptable latency. The target is 10 / 0.1 = 100 messages per instance. The
// current value is 1500 / 10 = 150, so the group scales out - by five instances -
// until the ratio reaches the target.
const target = acceptableLatencySeconds / averageServiceSeconds;
const current = depth / runningWorkers;
// Publish `current` as a custom metric and let a target-tracking policy chase
// `target`. Scaling on CPU instead is the classic mistake: an IO-bound consumer
// waiting on a database sits at 10% CPU with an hour of backlog behind it, and the
// autoscaler sees a fleet that is comfortably over-provisioned.Where the ceiling actually is
The reason a scaling change sometimes makes throughput worse, and the argument to have before approving one. A consumer fleet is rarely limited by the queue; it is limited by the most contended thing it touches. Put the four-term decomposition in your traces so this is a measurement rather than an argument, and set a connection pool size per process that multiplies out to something the database can actually accept.
import { Pool } from "pg";
// 143 workers each holding a connection, against a pool of 100, does not process
// faster: requests queue inside the driver, which inflates measured service time,
// which raises the concurrency Little's Law asks for. The loop feeds back on itself.
const pool = new Pool({ max: 20 }); // per process
// 8 processes x 20 = 160 connections from this service alone. Compare that with
// max_connections on the primary (PostgreSQL's default is 100) and with every
// other service sharing it, before scaling the deployment.
// Decompose the service time before adding workers:
// service time = receive wait + handler CPU + database time + third-party time
// If database time dominates, the fix is batching, an index or a read replica.
// If third-party time dominates, the fix is their rate limit, and more workers
// just convert a backlog into 429s.
// If CPU dominates, more workers genuinely help - until the host runs out.
Discussion