Health and Readiness

Liveness says the process is alive, readiness says it should get traffic, and the two must check different things.

SyntaxGET /livez -> always 200 | GET /readyz -> 200 when servable, 503 when not

Services usually run as containers: your application and its dependencies packaged together and started as one process. Several identical copies, called replicas, run behind a load balancer that spreads requests across them. The platform needs to know whether each copy is working, without asking you. It finds out with a probe: an HTTP request to a route on your service, repeated every few seconds.

Two questions, two routes

Liveness asks whether this process is still running usefully. After it fails a set number of times in a row, the platform kills the container and starts a fresh one. Readiness asks whether this replica should be sent traffic right now. When it fails, the replica is taken out of the load balancer and left running.

Because the consequences differ, the checks differ. Liveness is cheap and local: answer 200 and nothing more. Readiness may check what you cannot serve a request without, such as a database connection, with a short timeout.

What a probe must not do

Liveness must never touch the database. A thirty second database stall would fail liveness on every replica at once and restart all of them, turning a recovery into an outage. Readiness must not call your other services either, or one slow dependency marks the whole system unready.

A third probe, startup, covers slow boots: the platform waits on it before liveness begins, so a long migration is not mistaken for a hang.

When asked to shut down, fail readiness first so traffic drains, then stop accepting connections.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Keep a replica out of the load balancer until its database pool has connected.
  • Stop a deployment from sending traffic to a new version that cannot reach its cache.
  • Let a container that has deadlocked be restarted without anyone being paged.
  • Finish in-flight requests during a rolling deploy instead of returning 502s.

More examples

Readiness with a real dependency check and a timeout

SELECT 1 proves a connection is obtainable from the pool and the server answers. The 500 millisecond cap is the important part, and it must be shorter than the probe's own timeout so your answer arrives before the platform gives up waiting.

Example · javascript
const pool = require('./db'); // a configured connection pool

function withTimeout(promise, ms) {
  let timer;
  const timeout = new Promise((_, reject) => {
    timer = setTimeout(() => reject(new Error('timeout')), ms);
  });
  // unref() keeps a pending timer from holding the process open.
  timer.unref();
  return Promise.race([promise, timeout]).finally(() => clearTimeout(timer));
}

async function readiness(req, res) {
  try {
    // The cheapest query that proves a connection can be had.
    await withTimeout(pool.query('SELECT 1'), 500);
    res.writeHead(200, { 'content-type': 'application/json' });
    res.end(JSON.stringify({ status: 'ready' }));
  } catch (err) {
    // 503 means "not me, not now", which is what readiness failure is.
    res.writeHead(503, { 'content-type': 'application/json' });
    res.end(JSON.stringify({ status: 'not-ready', reason: err.message }));
  }
}

How the platform is told about the routes

Readiness is checked often with a tight timeout because leaving the load balancer is cheap and reversible. Liveness is checked with more patience because the penalty is a restart. The startup probe exists so that generosity at boot does not mean generosity forever.

Example · yaml
# The two probes point at different paths and are tuned differently.
livenessProbe:
  httpGet:
    path: /livez
    port: 3000
  periodSeconds: 10      # how often to ask
  timeoutSeconds: 2      # how long to wait for an answer
  failureThreshold: 3    # consecutive failures before a restart

readinessProbe:
  httpGet:
    path: /readyz
    port: 3000
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 2    # leave the load balancer quickly
  successThreshold: 1    # rejoin as soon as one check passes

# Slow boots get their own probe; liveness does not start until it passes.
startupProbe:
  httpGet:
    path: /livez
    port: 3000
  periodSeconds: 5
  failureThreshold: 30   # up to 150 seconds to finish starting

Draining before exit

Without the pause, the process stops listening while the load balancer still believes in it, and the requests sent in that window fail. The sleep has to exceed the readiness period, so the two numbers are chosen together.

Example · javascript
let ready = true;

async function shutdown() {
  // 1. Fail readiness so the load balancer stops sending new requests.
  ready = false;

  // 2. Wait longer than one readiness period, so it notices before we close.
  await new Promise((resolve) => setTimeout(resolve, 6000));

  // 3. Stop accepting connections; in-flight requests still finish.
  server.close(async () => {
    await pool.end(); // close database connections
    process.exitCode = 0;
  });

  // On Node 18 and earlier, sockets sitting idle between keep-alive requests
  // held the close open and this call was needed. Since Node 19, server.close()
  // closes idle connections itself, so on Node 24 this line is harmless but
  // no longer required. Added in Node 18.2.
  server.closeIdleConnections();
}

Related

node metricsnode k8s probesnode signalsnode loggingnode db pool tuning

Discussion

  • Be the first to comment on this lesson.