Health Checks and Kubernetes Probes

Use the health endpoint, liveness and readiness groups correctly so the platform restarts broken instances and stops routing to busy ones.

SyntaxGET /actuator/health -> {"status":"UP"} GET /actuator/health/liveness GET /actuator/health/readiness class XHealthIndicator extends AbstractHealthIndicator { protected void doHealthCheck(Health.Builder b) { b.up().withDetail(...); } }

/actuator/health aggregates health indicators into one status: UP, DOWN, OUT_OF_SERVICE or UNKNOWN. Spring Boot registers indicators for what it finds: a db indicator for a DataSource, redis, rabbit, diskSpace and others. A DOWN status returns HTTP 503, which is what load balancers act on.

Liveness and readiness are different questions

Kubernetes asks two things. Liveness: is this process broken beyond repair, so it should be restarted? Readiness: should traffic be sent here right now? Spring Boot answers each with a health group, at /actuator/health/liveness and /actuator/health/readiness. They are backed by the application's own availability state, which Spring Boot moves through starting, ready and refusing traffic during startup and shutdown.

The classic mistake is putting the database in the liveness check. When the database has a short outage, every pod fails liveness, Kubernetes restarts all of them at once, and you have turned a database blip into a full outage with cold starts. A restart cannot fix a remote dependency. Dependencies belong in readiness, if anywhere:

management.endpoint.health.group.readiness.include=readinessState,db

Even readiness deserves thought. If every instance depends on the same database, taking all of them out of rotation turns slow responses into connection refused. Many teams include only dependencies that are specific to the instance, such as a local cache warm-up, and let failing requests return proper errors instead.

Details and who sees them

management.endpoint.health.show-details defaults to never, so anonymous callers see only the overall status. Set it to when-authorized so operators can see which component failed without publishing your topology. Probes do not need details.

Your own indicators

Extend AbstractHealthIndicator (in org.springframework.boot.health.contributor in Spring Boot 4) and the bean name minus the HealthIndicator suffix becomes the component name. Keep checks cheap and bounded with a timeout: probes run every few seconds on every pod, and a health check that waits 30 seconds on a dead service makes the probe itself time out. management.endpoint.health.cache.time-to-live caches only the plain /actuator/health response: requests for a group such as /actuator/health/readiness carry a path and are never cached, so an expensive check that sits in a probe group must cache its own result.

Changing readiness yourself

Publish an AvailabilityChangeEvent with ReadinessState.REFUSING_TRAFFIC to take an instance out of rotation without stopping it, for example while it rebuilds a large in-memory index. Publish ACCEPTING_TRAFFIC when it is done.

With management.endpoint.health.probes.add-additional-paths=true the two groups are also served on the main port as /livez and /readyz, which matters when Actuator runs on a separate management port: a probe on the main port proves the application connector itself is answering.

Example

Example · java
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import org.springframework.boot.health.contributor.AbstractHealthIndicator;
import org.springframework.boot.health.contributor.Health;
import org.springframework.stereotype.Component;

// Shows up as the "paymentProvider" component of /actuator/health.
@Component
public class PaymentProviderHealthIndicator extends AbstractHealthIndicator {

    private final HttpClient client = HttpClient.newBuilder()
            .connectTimeout(Duration.ofSeconds(2))
            .build();

    @Override
    protected void doHealthCheck(Health.Builder builder) throws Exception {
        HttpRequest request = HttpRequest.newBuilder(URI.create("https://payments.example.com/status"))
                .timeout(Duration.ofSeconds(2))
                .GET()
                .build();
        HttpResponse<Void> response = client.send(request, HttpResponse.BodyHandlers.discarding());
        if (response.statusCode() == 200) {
            builder.up().withDetail("latencyBudgetMs", 2000);
        } else {
            builder.down().withDetail("status", response.statusCode());
        }
    }
}

When to use it

  • Stop Kubernetes from restarting every pod when the shared PostgreSQL database fails over for twenty seconds.
  • Report a third-party payment provider as down in the health details without taking the whole service out of rotation.
  • Hold an instance out of the load balancer while it warms a large cache after deployment.

More examples

Health groups for probes

Groups are defined by listing component names. The custom indicator from the main example appears as <code>paymentProvider</code> in <code>/actuator/health</code>, but it is kept out of both probe groups: the provider is shared by every instance, so putting it in readiness would take the whole fleet out of rotation when the provider has a bad minute. The cache stops dashboards and monitoring from calling the provider on every request to <code>/actuator/health</code>; it does not apply to <code>/actuator/health/readiness</code> or <code>/readyz</code>, which is one more reason to keep slow checks out of probe groups.

Example · properties
# Probes are served at /actuator/health/liveness and /actuator/health/readiness.
management.endpoint.health.probes.enabled=true

# Liveness and readiness: only the application's own state.
# paymentProvider still shows in /actuator/health, but no probe calls it.
management.endpoint.health.group.liveness.include=livenessState
management.endpoint.health.group.readiness.include=readinessState

# Operators can see component details; anonymous callers cannot.
management.endpoint.health.show-details=when-authorized

# Cache the full /actuator/health response (group paths are never cached).
management.endpoint.health.cache.time-to-live=10s

# Also serve /livez and /readyz on the main application port.
management.endpoint.health.probes.add-additional-paths=true

Taking an instance out of rotation

While the readiness state is <code>REFUSING_TRAFFIC</code>, <code>/actuator/health/readiness</code> answers 503 and Kubernetes removes the pod from the service endpoints. The liveness probe stays <code>UP</code>, so the pod is not restarted half way through. The <code>finally</code> block makes sure the instance comes back even if the rebuild fails.

Example · java
import org.springframework.boot.availability.AvailabilityChangeEvent;
import org.springframework.boot.availability.ReadinessState;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.stereotype.Service;

@Service
public class SearchIndexService {

    private final ApplicationEventPublisher events;

    SearchIndexService(ApplicationEventPublisher events) {
        this.events = events;
    }

    public void rebuild() {
        AvailabilityChangeEvent.publish(events, this, ReadinessState.REFUSING_TRAFFIC);
        try {
            loadEverything();
        } finally {
            AvailabilityChangeEvent.publish(events, this, ReadinessState.ACCEPTING_TRAFFIC);
        }
    }

    private void loadEverything() {
        // Rebuild the in-memory index from the database.
    }
}

Kubernetes probe configuration

The startup probe gives the application up to 150 seconds to start before liveness begins, so slow starts are not mistaken for hangs. Liveness is deliberately tolerant (three failures), readiness reacts quickly. The paths are the additional ones on the main port, enabled by <code>probes.add-additional-paths</code>.

Example · yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
spec:
  template:
    spec:
      containers:
        - name: orders
          image: registry.example.com/orders:1.4.2
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /livez
              port: 8080
            periodSeconds: 5
            failureThreshold: 30
          livenessProbe:
            httpGet:
              path: /livez
              port: 8080
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 5
            failureThreshold: 2

Discussion

  • Be the first to comment on this lesson.