Probes and Health Checks

How liveness, readiness and startup probes map onto ASP.NET Core health endpoints, and which mistakes restart healthy pods.

Syntaxapp.MapHealthChecks("/readyz", new HealthCheckOptions { Predicate = r => r.Tags.Contains("ready") });

A probe is a request Kubernetes makes to your container on a schedule to decide what to do with it. ASP.NET Core answers probes with health checks: AddHealthChecks() registers checks and MapHealthChecks() publishes an endpoint that runs them.

Three probes, three jobs

  • liveness: if it fails, Kubernetes restarts the container.
  • readiness: if it fails, the pod is removed from every Service's endpoint list. The process keeps running, it just receives no traffic.
  • startup: runs first. While it is failing the other two are disabled, and once it succeeds it never runs again.

Mapping them to endpoints

Liveness should run no checks at all, so set Predicate = _ => false. A 200 then means only that the process still accepts and answers an HTTP request. Readiness runs the checks that prove the pod can do work, selected by tag. The middleware returns 200 for Healthy and Degraded, and 503 for Unhealthy. MapHealthChecks("/readyz") with no options is the same as passing a HealthCheckOptions whose Predicate is left null: both run every registered check.

What trips people up

The default period is 10 seconds, the default timeout is 1 second, and the default failureThreshold is 3, so a container is killed about 30 seconds after it starts failing. A slow start is handled with a startupProbe, not a large initialDelaySeconds, because a startup probe stops waiting as soon as the app is up.

Example

Example · csharp
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;

var builder = WebApplication.CreateBuilder(args);

// One registered check, tagged "ready". The tag is how it is selected below.
builder.Services.AddHealthChecks()
    .AddCheck<DatabaseCheck>("database", tags: ["ready"]);

var app = builder.Build();

app.MapGet("/orders/1", () => new { id = 1, total = 42 });

// Liveness: run no checks. A 200 means the process answers HTTP.
app.MapHealthChecks("/healthz", new HealthCheckOptions
{
    Predicate = _ => false
});

// Readiness: run only the checks tagged "ready".
app.MapHealthChecks("/readyz", new HealthCheckOptions
{
    Predicate = registration => registration.Tags.Contains("ready")
});

app.Run();

public sealed class DatabaseCheck : IHealthCheck
{
    // A real check opens a connection and runs SELECT 1.
    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        return Task.FromResult(HealthCheckResult.Healthy("connected"));
    }
}

When to use it

  • Stop a slow-starting service from being killed before it finishes loading.
  • Keep a pod out of the load balancer while it reconnects to a dependency.
  • Explain a pod that restarts every few minutes while the application logs look fine.

More examples

The probes in the pod spec

This is the shape to copy for any ASP.NET Core service. The startup probe is generous and the liveness probe is not, which is exactly the split you want: slow boots are normal, a process that stops answering is not.

Example · yaml
containers:
  - name: api
    image: registry.example.com/orders:v1
    ports:
      - containerPort: 8080
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 2
      failureThreshold: 30     # up to 60s to start, then the container is killed
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 10        # default
      failureThreshold: 3      # default: dead after about 30s of failures
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      periodSeconds: 5
      failureThreshold: 2

Failing readiness before shutdown

Without this, a terminating pod keeps receiving new requests for several seconds after it has begun shutting down, and some of them are refused. Volatile.Read and Volatile.Write are used because the flag is written on the shutdown thread and read on request threads.

Example · csharp
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;

var builder = WebApplication.CreateBuilder(args);

// Flipped to false when Kubernetes sends SIGTERM.
bool accepting = true;

builder.Services.AddHealthChecks()
    .AddCheck("accepting", () => Volatile.Read(ref accepting)
        ? HealthCheckResult.Healthy()
        : HealthCheckResult.Unhealthy("shutting down"), tags: ["ready"]);

var app = builder.Build();

app.MapHealthChecks("/readyz", new HealthCheckOptions
{
    Predicate = registration => registration.Tags.Contains("ready")
});

// Fail readiness first. Kubernetes needs a probe period or two to stop
// sending new requests, and in-flight requests still have to finish.
app.Lifetime.ApplicationStopping.Register(
    () => Volatile.Write(ref accepting, false));

app.Run();

Diagnosing a restart loop

A RESTARTS count that keeps climbing on a pod that reports Running is almost always a liveness probe that is too strict, or one pointed at an endpoint that runs dependency checks. --previous reads the logs of the container that was killed, which the plain logs command no longer shows.

Example · bash
kubectl get pods -l app=orders
# NAME                      READY   STATUS    RESTARTS   AGE
# orders-7d9f5c8d4b-x4r2q   1/1     Running   4          6m

# Why was it killed? The events at the bottom name the probe.
kubectl describe pod orders-7d9f5c8d4b-x4r2q

# What did the app say before it died?
kubectl logs orders-7d9f5c8d4b-x4r2q --previous

Related

dotnet health checksdotnet k8s objectsdotnet k8s deploydotnet shutdowndotnet minimal api

Discussion

  • Be the first to comment on this lesson.