Health Checks

Expose separate liveness and readiness endpoints, decide what each should check, and wire them to Kubernetes probes without causing restart storms.

SyntaxGET /health/live -> 200 while the process can serve at all GET /health/ready -> 200 when it should get traffic, 503 when not

An orchestrator cannot read your logs to decide whether a service is well. It calls an HTTP endpoint and looks at the status code. Kubernetes asks three different questions, and mixing them up is the most common way health checks cause outages instead of preventing them.

Three probes, three questions

ProbeQuestionOn failure
LivenessIs this process stuck beyond repair?The container is restarted.
ReadinessShould traffic be sent here right now?The pod is removed from the Service's endpoints until it passes again.
StartupHas the application finished starting?Liveness and readiness wait; after too many failures the container is restarted.

A 2xx or 3xx response passes; anything else, or a timeout, fails.

What each should check

Liveness checks only the process itself. If it pings the database and the database has a bad minute, every pod fails liveness, every pod restarts, and the restarts add load just as the database recovers. A restart only helps when restarting would fix it, such as a deadlock or a broken internal state. For many services, "the HTTP thread answered" is enough.

Readiness can include dependencies the instance cannot work without, such as its own database. When readiness fails, the pod keeps running and simply stops getting new requests. Report not ready while caches are warming at startup and as the first step of shutdown.

Startup covers slow starts. Without it, you raise the liveness delay to fit the slowest start ever seen, and real hangs are then detected late.

Keep them cheap and honest

  • Probes run every few seconds on every pod. Do not run heavy queries; a SELECT 1 or the pool's own validation is enough.
  • Give dependency checks their own short timeout, shorter than the probe's timeoutSeconds.
  • Return 503 Service Unavailable for not ready, with a small JSON body that says which check failed. Humans read it during incidents.
  • Do not put health endpoints behind authentication the kubelet cannot pass, and do not expose detailed dependency information publicly.

In Spring Boot, Actuator provides /actuator/health/liveness and /actuator/health/readiness. They are switched on automatically when the application detects Kubernetes, and you choose which indicators each group includes.

Example

Example · java

import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.atomic.AtomicBoolean;

final AtomicBoolean ready = new AtomicBoolean(false);

void main() throws Exception {
    var server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
    // Liveness: only "can this process answer?"
    server.createContext("/health/live", ex -> send(ex, 200, "{\"status\":\"UP\"}"));
    // Readiness: "should traffic come here now?"
    server.createContext("/health/ready", ex -> {
        if (ready.get()) {
            send(ex, 200, "{\"status\":\"UP\"}");
        } else {
            send(ex, 503, "{\"status\":\"OUT_OF_SERVICE\",\"reason\":\"warming caches\"}");
        }
    });
    server.start();

    String base = "http://127.0.0.1:" + server.getAddress().getPort();
    try (var client = HttpClient.newHttpClient()) {
        probe(client, base + "/health/live");
        probe(client, base + "/health/ready");
        ready.set(true); // startup work finished
        probe(client, base + "/health/ready");
    } finally {
        server.stop(0);
    }
}

void send(HttpExchange ex, int status, String json) throws IOException {
    byte[] body = json.getBytes(StandardCharsets.UTF_8);
    ex.getResponseHeaders().set("Content-Type", "application/json");
    ex.sendResponseHeaders(status, body.length);
    try (var out = ex.getResponseBody()) {
        out.write(body);
    }
}

void probe(HttpClient client, String url) throws Exception {
    var request = HttpRequest.newBuilder(URI.create(url)).build();
    var response = client.send(request, HttpResponse.BodyHandlers.ofString());
    IO.println(URI.create(url).getPath() + " " + response.statusCode() + " " + response.body());
}

// Output:
// /health/live 200 {"status":"UP"}
// /health/ready 503 {"status":"OUT_OF_SERVICE","reason":"warming caches"}
// /health/ready 200 {"status":"UP"}

When to use it

  • Keep a search API pod out of the load balancer until its in-memory index has loaded, then mark it ready.
  • Stop a database blip from restarting every pod of an order service by checking the database in readiness only.
  • Give a large legacy service a startup probe so its slow boot no longer forces a long liveness delay on every restart.

More examples

Kubernetes probes for a Spring Boot service

The startup probe gives a slow start two minutes, after which liveness takes over with a tight setting that catches real hangs quickly. Readiness fails faster than liveness because taking a pod out of rotation is cheap and a restart is not.

Example · yaml
containers:
  - name: orders
    image: registry.example.com/orders-service:1.4.0
    ports:
      - containerPort: 8080
    startupProbe:
      httpGet: { path: /actuator/health/liveness, port: 8080 }
      periodSeconds: 5
      failureThreshold: 24      # up to 2 minutes to start
    livenessProbe:
      httpGet: { path: /actuator/health/liveness, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet: { path: /actuator/health/readiness, port: 8080 }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2

Spring Boot Actuator health groups

Groups decide which health indicators feed each endpoint, so the db check affects readiness and never liveness. show-details keeps dependency names and errors away from anonymous callers.

Example · properties
# Probe endpoints even outside Kubernetes (for local testing)
management.endpoint.health.probes.enabled=true

# Liveness: the application's own state only
management.endpoint.health.group.liveness.include=livenessState

# Readiness: also the database this instance cannot work without
management.endpoint.health.group.readiness.include=readinessState,db

# Show details to authorised users only
management.endpoint.health.show-details=when-authorized

A readiness check with a bounded database ping

Connection.isValid runs the driver's own lightweight check with a timeout, but getConnection() can still wait for the pool's connection timeout (30 s by default in HikariCP) when the database is down or the pool is busy. Keep that timeout short, or run this check on a schedule and let the probe read the cached result.

Example · java
import java.sql.Connection;
import java.sql.SQLException;
import javax.sql.DataSource;

final class DatabaseReadiness {
    private final DataSource dataSource;

    DatabaseReadiness(DataSource dataSource) {
        this.dataSource = dataSource;
    }

    /** True when a pooled connection is available and answers within one second. */
    boolean isReady() {
        try (Connection connection = dataSource.getConnection()) {
            return connection.isValid(1); // seconds; bounds the check, not getConnection()
        } catch (SQLException e) {
            return false;
        }
    }
}

Related

java graceful shutdownjava a server with com sun net httpserverjava status codes and headersjava connection pooling with hikaricpjava what a framework adds

Discussion

  • Be the first to comment on this lesson.