Why You Watch p99, Not the Mean
The mean describes the users who are fine; p99 describes the ones who are not, and fan-out turns a 1% tail into a majority.
histogram_quantile(φ scalar, b instant-vector)A latency distribution has a hard floor and a long right tail, so its mean sits near the common case and says nothing about the slow one. The SRE book's example: a service averaging 100 ms at 1,000 requests per second can still have 1% of requests taking 5 seconds — ten badly served users every second, invisible in the average.
Amazon states SLAs at the 99.9th percentile instead. The Dynamo paper gives the shape: "a response within 300ms for 99.9% of its requests for a peak client load of 500 requests per second", chosen because the customers with the most data do the most work and therefore live permanently in the tail — exactly the users a mean excludes.
Fan-out amplifies the tail
Dean and Barroso's "The Tail at Scale" describes a leaf server that usually answers in 10 ms but has a one-second 99th percentile. Scatter one user request across 100 of them and wait for all, and 63% of user requests take over a second: 1 - 0.99^100. A rare event has become the normal case.
What a p99 target implies
Invert it. To keep 99% of user requests inside budget at a fan-out of 100, each leaf may miss at most 1 - 0.99^(1/100), about 0.01% — p99.99, not p99. Sequential calls add rather than compound: a 200 ms p99 target over four hops leaves roughly 50 ms each, including network, queueing and any retry.
What trips people up
Percentiles do not average. The mean of per-instance p99s, or a one-minute p99 series averaged into an hour, is not a percentile of anything. Aggregate the histogram buckets first — sum by (le) — then compute the quantile once.
Example
// What a latency target actually demands of each dependency. There are two
// different sums, and using the wrong one is how budgets get missed.
/**
* Parallel fan-out: slowness compounds.
* P(request is slow) = 1 - (1 - p)^n for n independent calls you WAIT ON.
* @param {number} p fraction of calls that land in the slow tail, e.g. 0.01
* @param {number} fanOut how many of them one user request waits for
*/
const chanceRequestIsSlow = (p, fanOut) => 1 - Math.pow(1 - p, fanOut);
/** The inverse: solve 1 - (1 - p)^n = miss for p. */
const perLeafMiss = (miss, fanOut) => 1 - Math.pow(1 - miss, 1 / fanOut);
// Dean & Barroso, "The Tail at Scale": leaf servers that usually answer in
// 10ms but have a one-second 99th percentile, scattered over 100 shards.
chanceRequestIsSlow(0.01, 100); // 0.634 -> 63% of users wait over a second
// Flip it. To keep 99% of USER requests under the target at that fan-out:
perLeafMiss(0.01, 100); // 0.0001 -> each shard needs p99.99, not p99
/**
* Serial calls: latency ADDS, so the budget is divided rather than compounded.
* overheadMsPerHop is the network + queueing cost paid before any work starts.
*/
function serialBudget(targetMs, hops, overheadMsPerHop = 2) {
return (targetMs - hops * overheadMsPerHop) / hops;
}
serialBudget(200, 4); // 48 -> 48ms of real work per hop, at p99
// Both sums are why "just add a service" is never free: a fifth serial hop
// takes ~10ms off every other hop's budget, and a 101st shard raises the tail.
// Set each dependency's timeout from these numbers, not from a round guess.When to use it
- Setting a latency SLO for a team that currently reports average response time.
- Explaining why adding a sixth dependency call moved the p99 but left the mean unchanged.
- Sizing per-dependency timeouts so their sum fits a 200 ms page-load target.
- A p99 panel that jumps between two values and never sits between them.
More examples
Computing p99 correctly in PromQL
The two forms are the same function, `histogram_quantile(φ scalar, b instant-vector)`, over the two histogram representations; recognising both matters because dashboards and exporters are mid-migration. Use the aggregate-then-quantile ordering every time you break a latency SLI down by route, region or tenant — and treat any dashboard panel that averages a pre-computed percentile as broken.
# LONGHAND - classic histogram. One counter series per bucket, suffix
# _bucket, label `le` holding the inclusive upper bound. Samples without an
# `le` label are silently ignored. Aggregate buckets FIRST, quantile once.
histogram_quantile(
0.99,
sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)
# SHORTHAND - native histogram: buckets live inside a single sample, so there
# is no _bucket suffix and no `le` to sum over. Same signature, same meaning.
# Experimental since Prometheus 2.40 (--enable-feature=native-histograms),
# stable from 3.8.
histogram_quantile(
0.99,
sum by (route) (rate(http_request_duration_seconds[5m]))
)
# WRONG, and it will not error - it just returns a plausible lie:
# avg by (route) (instance:latency_p99)
# The mean of nine machines' p99s is not the p99 of the nine together.Hedged requests to cut the tail
Shows the cheapest structural fix for tail latency: the extra load is bounded by the hedge delay, because a request that is already fast never triggers a second copy. Reach for it on fan-out reads where one slow replica dominates the user-visible p99 and you cannot make every replica uniformly fast.
// Hedging: send a second copy once the first call exceeds the p95, and take
// whichever answers first. "The Tail at Scale": reading 1,000 keys from a
// BigTable spread over 100 servers, a hedge after 10ms cut the 99.9th
// percentile from 1,800ms to 74ms while sending 2% more requests.
async function hedged(call, { hedgeAfterMs, replicas }) {
const controllers = [];
const attempt = (i) => {
const ac = new AbortController();
controllers.push(ac);
return call(replicas[i], ac.signal);
};
const first = attempt(0);
const hedgeTimer = new Promise((r) => setTimeout(r, hedgeAfterMs, 'hedge'));
// The common case: the first call beats the delay and nothing extra is sent.
// A rejection propagates here - hedging is not a retry policy.
const raced = await Promise.race([first, hedgeTimer]);
if (raced !== 'hedge') return raced;
try {
return await Promise.any([first, attempt(1)]);
} finally {
controllers.forEach((ac) => ac.abort()); // cancel the loser; do not pay twice
}
}
// hedgeAfterMs is the p95 of THIS call, read from the same histogram that
// feeds the dashboard - hardcode it and it rots within a quarter. Safe for
// reads and for idempotent writes only (see sd-idempotency).Bucket boundaries chosen around the SLO
Bucket edges are a measurement decision, not a detail: they set the resolution of every percentile you will ever compute from that instrument, and changing them later invalidates comparisons with history. Do this when you first write a latency SLO, placing boundaries on both sides of the target so you can see the SLI approach it.
// The OpenTelemetry default explicit boundaries are
// [0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000]
// (milliseconds). A p99 of 180ms is interpolated inside the 100-250 bucket,
// so it reports in steps and a 20ms regression is invisible.
import { MeterProvider, InstrumentType, AggregationType } from '@opentelemetry/sdk-metrics';
// @opentelemetry/sdk-metrics 2.x: views are plain ViewOptions objects.
const provider = new MeterProvider({
views: [
{
instrumentName: 'http.server.request.duration',
instrumentType: InstrumentType.HISTOGRAM,
aggregation: {
type: AggregationType.EXPLICIT_BUCKET_HISTOGRAM,
// Edges clustered around the 120ms SLO so the quantile lands between
// two nearby boundaries instead of inside one 150ms-wide bucket.
options: { boundaries: [10, 25, 50, 80, 100, 120, 150, 200, 400, 800, 2000] },
},
},
],
});
// The same view in 1.x, where aggregations were classes. The Aggregation
// class hierarchy, including ExplicitBucketHistogramAggregation, was removed
// in 2.0 - recognise both forms when reading an existing codebase:
// new View({
// instrumentName: 'http.server.request.duration',
// instrumentType: InstrumentType.HISTOGRAM,
// aggregation: new ExplicitBucketHistogramAggregation(
// [10, 25, 50, 80, 100, 120, 150, 200, 400, 800, 2000]),
// })
Discussion