Hot Keys and Skew
One key taking a disproportionate share of traffic, and the splitting, salting and caching answers to it.
physicalKey = `${logicalKey}#${Math.floor(Math.random() * SHARD_COUNT)}`Sharding assumes traffic spreads across keys. Real traffic follows a power law: a handful of keys — a celebrity account, a flash-sale product, a global counter — take a large share of it. Because a key lives on exactly one shard, the hot key's shard saturates while the rest of the cluster idles, and adding capacity to the cluster does nothing.
The limits are per partition
DynamoDB guarantees up to 3,000 read capacity units and 1,000 write capacity units per partition; table-level provisioning cannot raise that ceiling for one key. Adaptive capacity absorbs short bursts and isolates frequently accessed items, but it is a safety net, not a design. Redis Cluster hashes keys into 16,384 slots with CRC16 of the key modulo 16384, and a slot belongs to one node — so a brace-delimited hash tag, which forces related keys into the same slot, is also a way to concentrate load on purpose. In Kafka, all records with the same key go to one partition and therefore one consumer.
The three answers
For a hot read: cache it close to the reader and collapse concurrent misses into one origin request, so an expiry does not become a stampede. For a hot write: salt the key with a suffix spanning N shards, so one logical key becomes N physical ones and the read merges them. For a known offender: split it out — give that tenant or product its own partition, cluster or queue, and stop pretending it is average.
What trips people up
You cannot find hot keys in aggregate metrics. Per-shard and per-key counters are what reveal them.
Example
// Hot read path: one cache lookup, and at most one origin fetch in flight per
// key no matter how many concurrent requests miss it.
const inFlight = new Map(); // cacheKey -> Promise, the single-flight gate
async function getProfile(redis, db, userId) {
// Braces are a Redis Cluster hash tag: only the text inside them is hashed,
// so profile:{42} and sessions:{42} land in the same slot. Useful for MGET,
// and a way to accidentally concentrate a celebrity's keys on one node.
const cacheKey = `profile:{${userId}}`;
const hit = await redis.get(cacheKey);
if (hit) return JSON.parse(hit);
// Without this gate, one expiry on a celebrity key sends every concurrent
// request to the database at the same instant — a stampede on a single row.
let pending = inFlight.get(cacheKey);
if (!pending) {
pending = db
.one('SELECT id, display_name, avatar_url FROM profiles WHERE id = $1', [userId])
.then(async (row) => {
// Jittered TTL: a fixed TTL makes keys written together expire together.
const ttl = 60 + Math.floor(Math.random() * 30);
await redis.set(cacheKey, JSON.stringify(row), 'EX', ttl);
return row;
})
.finally(() => inFlight.delete(cacheKey));
inFlight.set(cacheKey, pending);
}
return pending;
}
// Hot write path: salt the key across a fixed number of shards.
const COUNTER_SHARDS = 16; // fixed at design time — see the note on lowering it
function counterWriteKey(counterId) {
// Random suffix spreads writes evenly. A hash of the writer's id instead
// would give each writer a stable shard, which some read paths prefer.
return `${counterId}#${Math.floor(Math.random() * COUNTER_SHARDS)}`;
}
async function incrementViews(redis, counterId) {
await redis.incr(`views:${counterWriteKey(counterId)}`);
}
// The cost of salting: the read now touches every shard and merges.
async function readViews(redis, counterId) {
const keys = Array.from(
{ length: COUNTER_SHARDS },
(_, i) => `views:${counterId}#${i}`,
);
// Keys are spread across slots on purpose, so this cannot be one MGET on a
// cluster; issue it as a pipeline of GETs and sum what comes back.
const values = await Promise.all(keys.map((k) => redis.get(k)));
return values.reduce((sum, v) => sum + Number(v ?? 0), 0);
}When to use it
- Diagnosing one saturated shard or Redis node while the rest of the cluster sits near idle.
- Serving a celebrity account's profile or post without a cache expiry stampeding the database.
- Raising the write ceiling on a single global counter or flash-sale inventory row.
- Isolating the largest tenant in a multi-tenant system so its traffic cannot affect everyone else's latency.
More examples
Finding the hot key before you design around it
The measurement step that has to come first: per-shard dashboards tell you a shard is hot, and this tells you which key made it hot. Reach for it when one shard, partition or cache node is saturated and the aggregate request rate looks unremarkable.
// Hot keys are invisible in averages. Sample the key on every request and keep
// a bounded count: a fixed-size map with decay, so memory stays flat.
class HeavyHitters {
constructor({ capacity = 1000, decayMs = 60_000 } = {}) {
this.capacity = capacity;
this.decayMs = decayMs;
this.counts = new Map();
this.lastDecay = Date.now();
}
observe(key, cost = 1) {
this.maybeDecay();
this.counts.set(key, (this.counts.get(key) ?? 0) + cost);
if (this.counts.size > this.capacity) {
// Evict the current minimum. This is approximate by construction — it is
// meant to surface the top few keys, not to count them exactly.
let min = null;
for (const [k, v] of this.counts) if (min === null || v < this.counts.get(min)) min = k;
this.counts.delete(min);
}
}
maybeDecay() {
const now = Date.now();
if (now - this.lastDecay < this.decayMs) return;
// Halve everything, so yesterday's hot key does not stay hot forever.
for (const [k, v] of this.counts) {
if (v <= 1) this.counts.delete(k);
else this.counts.set(k, Math.floor(v / 2));
}
this.lastDecay = now;
}
top(n = 10) {
return [...this.counts.entries()].sort((a, b) => b[1] - a[1]).slice(0, n);
}
}
const hitters = new HeavyHitters();
// Cost-weighted, not request-weighted: a 400 KB item read is not one unit.
app.use((req, res, next) => {
if (req.params.userId) hitters.observe(`user:${req.params.userId}`, 1);
next();
});
// Export hitters.top() as a gauge, and alert when the leader's share of total
// traffic crosses the per-partition ceiling you are designed for.Splitting a known offender out of the shared pool
Shows the answer that needs no clever hashing: recognise the outlier and give it its own capacity, driven by runtime config so it can be applied mid-incident. Reach for it in multi-tenant systems where one customer is an order of magnitude larger than the median, and on queues where one publisher's backlog would otherwise delay everyone.
// Some keys are permanently abnormal: the largest tenant, the official account,
// the one product in a flash sale. Routing them to dedicated capacity is
// cheaper and far more predictable than making the shared path handle them.
const DEDICATED_TENANTS = new Set(['tenant_acme', 'tenant_globex']);
function routeFor(tenantId, pools) {
// An explicit allow-list, reloaded from config, not a code deploy — you want
// to move a tenant during an incident, not after a release.
return DEDICATED_TENANTS.has(tenantId) ? pools.dedicated[tenantId] : pools.shared;
}
// The same idea on a queue: a noisy publisher gets its own topic, so its
// backlog cannot delay everyone else's. Shared consumers keep their SLO.
function topicFor(tenantId) {
return DEDICATED_TENANTS.has(tenantId) ? `events.${tenantId}` : 'events.shared';
}
// Kafka's hashing puts every record with the same key on one partition, which
// is what gives per-key ordering. If one key exceeds what a single consumer can
// process, the fix is a different key — not more partitions.
function kafkaKey(event) {
// Ordering is required per conversation, not per tenant, so key on the
// narrower thing and the load spreads while ordering still holds.
return event.conversationId;
}
Discussion