Vertical and Horizontal Scaling

Scaling up buys capacity without touching your code; scaling out also buys availability, and breaks every assumption about shared state.

Syntaxscale up: same topology, bigger instance | scale out: more instances + a router, and no state in the process

Vertical scaling (scaling up) means running the same system on a bigger machine. Horizontal scaling (scaling out) means running more copies of it and putting something in front to spread the work. Both buy capacity; only one of them changes your code.

Why the first answer is often a bigger box

One machine goes further than most designs assume. AWS's u7in-32tb.224xlarge is a single instance with 896 vCPUs, 32 TiB of memory and up to 200 Gbps of network. A service doing tens of thousands of queries per second against a few terabytes usually needs a correct index and more RAM, not a distributed database. Vertical scaling also needs no application change at all, which is the real argument for it: it buys you months in an afternoon.

What the crossing costs

Scaling up ends at a ceiling and a steep price curve, and it never buys availability — one instance is one failure domain. Crossing to many machines, you pay once in design. Anything living inside the process is now wrong: in-process caches, session state, counters, rate-limit buckets, scheduled timers, files on local disk. Partial failure becomes the normal state, where one node of ten is slow or gone while the rest serve.

What trips people up

Scale-out is sublinear. Each node you add contends for whatever they share — the database, a lock, the leader — so throughput flattens and can decline, the shape Gunther's Universal Scalability Law describes. Doubling the nodes rarely doubles the throughput. Measure the second node before you buy the tenth.

Example

Example · javascript
// Rate limiting: the same feature written for one box, and for many.
import { createClient } from 'redis';

// --- Correct on one process, wrong on ten ---------------------------------
// Each replica counts only the requests it happened to receive, so behind a
// load balancer with 10 replicas the effective limit is 10x what you set.
const counts = new Map();

function allowLocal(userId, limit = 100) {
  const n = (counts.get(userId) ?? 0) + 1; // ?? not ||: a stored 0 must stay 0
  counts.set(userId, n);
  return n <= limit;
}

// --- Scales out ----------------------------------------------------------
// The counter moves out of the process into one store every replica shares.
const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

async function allowShared(userId, limit = 100, windowSeconds = 60) {
  // Fixed window: the bucket is part of the key, so old buckets just expire.
  const bucket = Math.floor(Date.now() / 1000 / windowSeconds);
  const key = `rl:${userId}:${bucket}`;

  // INCR returns the new value, so increment and read are one atomic round
  // trip. Two replicas cannot both read 99 and both decide to allow.
  const n = await redis.incr(key);
  if (n === 1) await redis.expire(key, windowSeconds); // only the creator sets TTL
  return n <= limit;
}

When to use it

  • A single Postgres primary is at 80% CPU and the next instance size doubles the cores for one restart.
  • Deciding whether a service can run two replicas, before enabling autoscaling on it.
  • Explaining to an interviewer why your first diagram has one database and not six.
  • A load test shows throughput flattening between 8 and 16 app nodes and you need to name the shared bottleneck.

More examples

The state that blocks a second replica

The audit to run before putting a second replica behind the load balancer. Each of these three reports as an intermittent bug that only happens sometimes, because whether it happens depends on which replica the request landed on, which makes them expensive to find after the fact rather than before.

Example · javascript
import fs from 'node:fs/promises';

// Each of these is correct on one machine and a bug on ten.

// 1. In-process cache. Ten replicas means ten copies that expire
//    independently, and an invalidation delivered to one does nothing
//    to the other nine.
const cache = new Map();

// 2. A timer. Every replica runs it, so the nightly digest goes out ten times.
setInterval(sendDigests, 24 * 60 * 60 * 1000);

// 3. Local disk. The upload lands on replica 3; the download request is
//    routed to replica 7 and 404s.
await fs.writeFile(`/var/uploads/${id}.jpg`, buf);

// The scale-out versions:
//   1. a shared cache (Redis, Memcached), optionally with a short local TTL
//      in front of it to absorb the hottest keys
//   2. a leader-elected scheduler, or a queue with one consumer group
//   3. object storage, addressed by key, served through a CDN

Up and out, spelled out in the same manifest

Worth seeing side by side because the two changes behave differently: editing replicas adds or removes Pods and leaves the existing ones alone, while editing anything under template changes the Pod template and therefore replaces every Pod in a rolling update. A vertical change is a deployment; a horizontal change is not.

Example · yaml
# Scaling out and scaling up are two different fields of one object.
#
# Shorthand, and the patch it is exactly equivalent to:
#   kubectl scale deployment/api --replicas=10
#   kubectl patch deployment/api --type=merge -p '{"spec":{"replicas":10}}'
# Both write spec.replicas and neither touches the Pods already running.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 10               # scale out: this number
  template:
    spec:
      containers:
        - name: api
          resources:
            requests:
              cpu: "2"       # scale up: these numbers
              memory: 8Gi
            limits:
              memory: 8Gi

Deciding whether the next node helps

The number to put on a whiteboard when an interviewer asks what adding capacity buys. Efficiency falling as nodes rise means the bottleneck has moved off the application tier onto something shared, and the honest next step is to find it rather than to scale the tier that is no longer the constraint.

Example · javascript
// Scale-out efficiency, from a load test rather than from hope.
const runs = [
  { nodes: 1,  rps: 4200 },
  { nodes: 2,  rps: 8100 },
  { nodes: 4,  rps: 15200 },
  { nodes: 8,  rps: 24800 },
  { nodes: 16, rps: 27500 }, // nearly flat: something shared is the limit now
];

for (const r of runs) {
  const speedup = r.rps / runs[0].rps;
  const efficiency = speedup / r.nodes; // 1.00 would be perfectly linear
  console.log(
    `${r.nodes}\t${(r.rps / r.nodes).toFixed(0)} rps/node\t` +
    `efficiency ${efficiency.toFixed(2)}`,
  );
}

// 16 nodes returns 1.11x the throughput of 8 for twice the cost. The next
// thing to fix is the contended resource, not the node count.

Related

sd load balancerssd app tiersd replicationsd shardingsd databases

Discussion

  • Be the first to comment on this lesson.