How Distributed Systems Fail

Partial failure, the fallacies of distributed computing, and grey failure: the node that is slow rather than down.

A single-process program fails in one way: it stops. A distributed system fails in a way that has no local analogue, partial failure, where some components are working, some are not, and no component can see which is which. That is the whole difficulty of the field, and nearly every pattern in this category exists to contain it.

You cannot tell a dead node from a slow one

The only evidence you ever get about a remote process is a message that arrived or a message that did not. A request that times out may have been lost on the way out, may have run and had its reply lost, or may be sitting in a queue on a machine that will run it ten seconds from now. Those three are indistinguishable from the caller, so the caller has to be correct under all three. This is why a retry needs an idempotency key, and why did my write land? cannot be answered without reading it back.

The fallacies of distributed computing

L. Peter Deutsch listed seven assumptions that quietly stop holding at scale, and James Gosling added an eighth: the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology does not change; there is one administrator; transport cost is zero; the network is homogeneous.

Grey failure

The dangerous case is not down, it is sick. Gray Failure: The Achilles' Heel of Cloud-Scale Systems (Huang et al., HotOS '17) names the property that makes it dangerous: differential observability, where the system's own failure detectors see a healthy node while applications using it see eight-second responses. A node that is down leaves the pool in seconds. A node that is merely slow stays in it and poisons every request routed to it.

Example

Example · javascript
// Every remote call has THREE outcomes, not two. A caller that models only
// success and failure is the caller that double-charges somebody.
const Outcome = {
  SUCCEEDED: 'succeeded', // a reply came back
  FAILED: 'failed',       // the server saw it and refused: nothing happened
  UNKNOWN: 'unknown',     // timeout, reset, 5xx: it may have happened anyway
};

async function createCharge(http, idempotencyKey, body) {
  try {
    const res = await http.post('/v1/charges', body, {
      // Stripe-style key: the server dedupes, which is what makes UNKNOWN
      // safe to retry. Generate it once per logical action, not per attempt.
      headers: { 'Idempotency-Key': idempotencyKey },
      signal: AbortSignal.timeout(2000), // Node >= 17.3.0 (16.14.0 backport)
    });
    return { outcome: Outcome.SUCCEEDED, charge: res.data };
  } catch (err) {
    // A 4xx that is not 429 is the ONLY branch where we know the work did
    // not run: the server parsed the request and rejected it on purpose.
    if (err.status >= 400 && err.status < 500 && err.status !== 429) {
      return { outcome: Outcome.FAILED, err };
    }
    // Timeout, ECONNRESET, 5xx, 429: the charge may exist. Two valid moves,
    // and guessing is not one of them.
    return { outcome: Outcome.UNKNOWN, err };
  }
}

// Resolving UNKNOWN: retry with the SAME key, or read the charge back by
// the key you chose. Never create a second charge with a fresh key.
async function settle(http, key, body) {
  const first = await createCharge(http, key, body);
  if (first.outcome !== Outcome.UNKNOWN) return first;
  return createCharge(http, key, body); // same key: the server returns the
                                        // original charge if it was created
}

When to use it

  • Explaining why a 2-second timeout on a payment call is incomplete without an idempotency key behind it.
  • Diagnosing a drop in throughput after a deploy that removed no instances: one node is slow, not down.
  • Deciding which dependencies a request is allowed to lose before you commit to an availability number.

More examples

A health check that can actually fail

Most outages that take a long time to diagnose are grey failures: the instance is in the load balancer pool because /healthz is hardcoded to 200. Checking real work, and checking it against a latency bound rather than a boolean, is what makes the balancer able to route around a sick instance. The trap in the second handler is correlated failure: if db.query fails because the shared database is down, every instance returns 503 at once and the balancer has nothing left to route to. Readiness should reflect 'this instance is worse than its peers', so keep shared-dependency failures out of it or make the balancer fail open when all backends are unhealthy.

Example · javascript
// A liveness probe that cannot fail is a probe that hides grey failure.
// Shallow: answers 200 while every query in the process is timing out.
app.get('/healthz', (req, res) => res.status(200).send('ok'));

// Deep enough to be useful, cheap enough to run every second.
app.get('/readyz', async (req, res) => {
  const checks = {};

  // Measure, do not just test for truthiness. "Up but slow" is exactly the
  // case a boolean check misses.
  const t0 = performance.now();
  try {
    await db.query('SELECT 1');
    checks.dbMs = Math.round(performance.now() - t0);
  } catch {
    return res.status(503).json({ db: 'unreachable' });
  }

  // Local symptoms: the process can be sick while its dependencies are fine.
  checks.queueDepth = admissionQueue.length;
  checks.loopLagMs = Math.round(loopDelayHistogram.percentile(99) / 1e6);

  const sick = checks.dbMs > 200 || checks.queueDepth > 500 || checks.loopLagMs > 500;
  res.status(sick ? 503 : 200).json(checks);
});

Ejecting a host that is bad but not down

This is the infrastructure answer to grey failure, and worth knowing because it shows what the mechanism can and cannot see. A host that returns 200 slowly trips none of these, which is why the upstream timeout matters: a timeout turns slowness into a 504 and makes it countable. max_ejection_percent defaulting to 10 is the guard against the obvious disaster, a correlated failure in which every host looks bad and outlier detection ejects the entire cluster.

Example · yaml
# Envoy cluster config. Two separate mechanisms, often confused:
# circuit_breakers caps CONCURRENCY; outlier_detection removes a bad host.
clusters:
  - name: payments
    connect_timeout: 0.25s
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          max_connections: 1024      # defaults: all three are 1024
          max_pending_requests: 1024 # queue toward this cluster
          max_requests: 1024         # concurrent requests in flight
          max_retries: 3             # default 3
    outlier_detection:
      consecutive_5xx: 5          # default 5
      interval: 10s               # default 10s: the analysis sweep
      base_ejection_time: 30s     # default 30s, multiplied per ejection
      max_ejection_percent: 10    # default 10: never eject the whole pool
      # Statistical ejection catches the host that is merely worse than
      # its peers, which consecutive_5xx never sees.
      success_rate_stdev_factor: 1900  # 1900 / 1000 = 1.9 std deviations
      success_rate_minimum_hosts: 5    # needs 5 hosts to have a mean
      success_rate_request_volume: 100 # and 100 requests on the host

Why four reliable dependencies are not a reliable service

Reach for this when someone proposes a target like 'four nines' without counting the call graph. It also reframes the work: the cheapest reliability improvement is usually not making a dependency better, it is deciding that a response is still valid without it. The last line is the same multiplication applied to latency, and it is why a p99 that looks fine per service can mean a majority of slow user requests once a request fans out.

Example · javascript
// Serial dependencies multiply. This is the arithmetic to do BEFORE you
// write an availability target into a design doc.
const deps = [
  { name: 'auth',    availability: 0.9995, critical: true  },
  { name: 'profile', availability: 0.999,  critical: true  },
  { name: 'feed',    availability: 0.999,  critical: true  },
  { name: 'ads',     availability: 0.99,   critical: false },
];

const product = (ds) => ds.reduce((p, d) => p * d.availability, 1);
const hoursDownPerYear = (a) => ((1 - a) * 8760).toFixed(1);

// If the request fails when ANY dependency fails:
console.log(product(deps).toFixed(5));                  // 0.98753
console.log(hoursDownPerYear(product(deps)));           // 109.3 hours

// Make 'ads' optional (render the page without it) and its failures stop
// counting. One line of degradation buys more than any amount of retrying.
const required = deps.filter((d) => d.critical);
console.log(product(required).toFixed(5));              // 0.99750
console.log(hoursDownPerYear(product(required)));       // 21.9 hours

// Fan-out does the same thing to latency. Dean and Barroso, "The Tail at
// Scale": scatter to 100 replicas whose p99 is 1s and most requests are slow.
console.log((1 - Math.pow(0.99, 100)).toFixed(3));      // 0.634

Related

sd timeouts retriessd circuit breakerssd idempotencysd common failuressd slos

Discussion

  • Be the first to comment on this lesson.