Resharding Without Downtime

Changing the key-to-shard mapping while the system serves traffic: backfill, verify, switch reads, then writes, and keep rollback cheap.

Syntaxbackfill -> tail change stream -> verify -> switch reads -> freeze and drain -> switch writes -> keep old readable

Resharding changes the mapping from keys to shards on a system that is already serving traffic — splitting a shard that outgrew its machine, adding capacity, or changing the shard key entirely. Everybody eventually needs it, because the first shard count and the first shard key were both guesses made before the traffic existed.

The shape of a safe cutover

  1. Backfill the new shard from a snapshot of the source, then keep it current by tailing the source's change stream — binlog, WAL or change feed.
  2. Verify row by row while both sides are live. Vitess makes this an explicit step: you run a VDiff and wait for it to report clean before cutting over.
  3. Switch reads first, and by traffic class — read-only and replica traffic well before anything that reads from a primary.
  4. Switch writes: briefly freeze writes to the source, let the stream drain, then move the routing.
  5. Keep the old shard readable afterwards, so rollback stays a routing change rather than a restore.

Double-reading through the transition

While the mapping is in flight the application must tolerate a key existing in either place. The usual pattern is to write to both sides idempotently, read the new side and fall back to the old on a miss, and only drop the fallback once the backfill and the verification pass both say the new side is complete. Figma did the equivalent with logical shards — views over a single physical host — so the routing change was proven correct before any data moved between machines.

What trips people up

Pre-splitting is far cheaper than resharding. Map keys once to a large fixed number of logical shards, then place logical shards on physical nodes: growing moves whole logical shards and never rehashes a key.

Example

Example · javascript
// Moving keys from an old shard to a new one while both serve traffic. The
// phase is a runtime flag, advanced only after the previous step has been
// verified, and every phase is reversible by moving it back.
const PHASE = {
  OFF: 0,      // old shard only
  BACKFILL: 1, // copier running, writes go to both, reads still old
  DUAL: 2,     // writes to both, read new and fall back to old
  NEW: 3,      // new only, old kept readable for rollback
};

let phase = Number(process.env.RESHARD_PHASE ?? PHASE.OFF);

// null for keys this migration does not touch, so the blast radius is bounded.
function sides(key) {
  if (!isMoving(key)) return null;
  return { old: oldShard(key), new: newShard(key) };
}

async function put(key, row) {
  const s = sides(key);
  if (!s || phase === PHASE.OFF) return oldShard(key).upsert(key, row);
  if (phase === PHASE.NEW) return s.new.upsert(key, row);
  // Old side first: if the new write fails, nothing has yet been told to
  // trust the new side. upsert must be idempotent, because the backfill
  // copier and this write can touch the same row in either order.
  await s.old.upsert(key, row);
  await s.new.upsert(key, row);
}

async function del(key) {
  const s = sides(key);
  if (!s || phase === PHASE.OFF) return oldShard(key).delete(key);
  // Both sides, always, for as long as the fallback read below is live --
  // otherwise the old copy resurrects the row on the next miss.
  await Promise.all([s.old.delete(key), s.new.delete(key)]);
}

async function get(key) {
  const s = sides(key);
  if (!s || phase <= PHASE.BACKFILL) return oldShard(key).get(key);
  if (phase === PHASE.NEW) return s.new.get(key);
  const row = await s.new.get(key);
  if (row) return row;
  // This counter is the gate on advancing to PHASE.NEW: it has to reach zero
  // and stay there, not merely look small.
  metrics.increment('reshard.fallback_hit');
  return s.old.get(key);
}

When to use it

  • One shard of four holds 60% of the data and needs splitting while the service stays up.
  • The shard key turned out to be wrong and the data has to move to a new one without a maintenance window.
  • Adding four nodes to a 1024-logical-shard cluster and deciding which logical shards move.
  • Writing the rollback plan a reviewer will ask for before approving the migration.

More examples

The same cutover, driven by Vitess

The same five steps as the hand-rolled version, with the copier, the change-stream tail and the routing table supplied. Worth reading even if you never run Vitess, because the command set is a checklist: a tool built for this names verification and read-before-write switching as separate steps, and leaves a reverse path in place until you explicitly complete.

Example · bash
# Splitting shard 0 of keyspace "customer" into shards -80 and 80-.

# 1. Create the workflow: snapshot the source shard, then tail its binlog.
vtctldclient --server localhost:15999 Reshard \
  --workflow cust2cust --target-keyspace customer create \
  --source-shards '0' --target-shards '-80,80-'

# 2. Watch the copy phase and the replication lag behind it.
vtctldclient --server localhost:15999 Reshard \
  --workflow cust2cust --target-keyspace customer status

# 3. Compare the two sides row by row. Do not skip this step.
vtctldclient --server localhost:15999 VDiff \
  --workflow cust2cust --target-keyspace customer create
vtctldclient --server localhost:15999 VDiff \
  --workflow cust2cust --target-keyspace customer show last

# 4. Reads first, writes second. SwitchTraffic replaced the older separate
#    SwitchReads and SwitchWrites commands with one that takes tablet types.
vtctldclient ... Reshard --workflow cust2cust --target-keyspace customer \
  switchtraffic --tablet-types "rdonly,replica"
vtctldclient ... Reshard --workflow cust2cust --target-keyspace customer \
  switchtraffic --tablet-types "primary"

# Rollback, while the reverse workflow still exists:
#   ... Reshard --workflow cust2cust --target-keyspace customer reversetraffic
# Only once you are certain, because it drops the reverse workflow with it:
#   ... Reshard --workflow cust2cust --target-keyspace customer complete

Pre-splitting, so resharding is only a placement change

The decision that makes every future reshard cheap, and it has to be made before launch. The cost is committing to a logical count with no traffic data, and raising it later is exactly the expensive key-rehashing migration you were trying to avoid, so err high: unused logical shards cost almost nothing.

Example · javascript
import { createHash } from 'node:crypto';

// Hash each key once into a large fixed number of LOGICAL shards, then map
// logical shards onto physical nodes in a table you can edit. Growing the
// cluster moves whole logical shards and never rehashes a key.
const LOGICAL = 1024; // chosen once, before launch, and never changed

const range = (a, b) => Array.from({ length: b - a }, (_, i) => a + i);

const logicalShard = (key) =>
  createHash('sha256').update(String(key)).digest().readUInt32BE(0) % LOGICAL;

// The only thing that changes when you add a node.
const placement = {
  'db-0': range(0, 256),
  'db-1': range(256, 512),
  'db-2': range(512, 768),
  'db-3': range(768, 1024),
};

function nodeFor(key) {
  const ls = logicalShard(key);
  for (const [node, owned] of Object.entries(placement)) {
    if (owned.includes(ls)) return node; // use a Set or a flat array in real code
  }
  throw new Error(`logical shard ${ls} is not placed on any node`);
}

// Adding db-4 means taking about 51 logical shards from each existing node and
// copying only those. 1024 is a common choice: it divides cleanly for a long
// time, and 1024 schemas or table prefixes spread over a few hosts is still
// something a human can operate.

A backfill that will not overwrite live writes

The copier is where silent data loss comes from, because it runs for hours against rows the application is also changing. Two properties make it safe: it is resumable, so a crash costs one batch, and its write is conditional on being newer, so racing the application's dual write cannot lose the application's version.

Example · sql
-- Copy in key order, resumable from a cursor, and safe to run while the
-- application is already dual-writing. Across separate hosts this is two
-- statements in the copier (or one through postgres_fdw); the ON CONFLICT
-- clause is the part that matters either way.
INSERT INTO orders AS o (id, user_id, total_cents, updated_at)
SELECT id, user_id, total_cents, updated_at
  FROM old_orders
 WHERE id > :cursor        -- last id copied, so a crashed batch just resumes
 ORDER BY id
 LIMIT 5000                -- small batches: long transactions block vacuum
ON CONFLICT (id) DO UPDATE
   SET total_cents = EXCLUDED.total_cents,
       updated_at  = EXCLUDED.updated_at
 -- The guard that makes the copier safe next to live traffic: a row the
 -- application has already written with a newer timestamp is left alone,
 -- so the backfill can never undo a user's edit.
 WHERE o.updated_at < EXCLUDED.updated_at;

Related

sd shardingsd consistent hashingsd idempotencysd schema evolutionsd deployment

Discussion

  • Be the first to comment on this lesson.