Health Checks

Expose separate liveness and readiness endpoints with tagged health checks, check the database safely and keep details private.

Syntaxbuilder.Services.AddHealthChecks() .AddCheck<DbCheck>("db", tags: ["ready"]); app.MapHealthChecks("/health/live", new() { Predicate = _ => false }); app.MapHealthChecks("/health/ready", new() { Predicate = c => c.Tags.Contains("ready") });

builder.Services.AddHealthChecks() registers HealthCheckService, and .AddCheck<T>("name", tags: [...]) adds checks. app.MapHealthChecks("/health/ready", options) maps an endpoint that runs every matching check concurrently, takes the worst status, and writes it. Defaults: Healthy and Degraded return 200, Unhealthy returns 503, and the body is the plain-text status and nothing else. A check that throws is reported with its failureStatus (Unhealthy unless you set it). Each run gets its own DI scope, so a check can depend on a scoped DbContext.

Liveness, readiness, startup

ProbeQuestionChecksOn failure
LivenessIs the process stuck?Nothing externalKubelet restarts the container
ReadinessCan it serve traffic now?Database, cache, required configPod removed from the Service endpoints; no restart
StartupHas it finished starting?In-process warm-up only, never a dependencyLiveness and readiness wait; past failureThreshold the kubelet restarts the container

The rule that prevents outages: liveness never checks a dependency. If the database is down and liveness includes it, Kubernetes restarts every pod, they come back still unable to reach the database, and a dependency incident becomes a crash loop with cold caches and a connection storm on recovery. The same goes for the startup probe, which restarts the container once it exceeds its failure threshold. Split them with tags and a Predicate: liveness runs a self check or none (Predicate = _ => false returns 200 whenever the app can answer), readiness runs checks tagged ready.

Database and dependency checks

AspNetCore.HealthChecks.NpgSql's AddNpgSql opens a connection and runs SELECT 1;. Give every dependency check a timeout of a few seconds, below the probe's timeoutSeconds, so a hung dependency is reported instead of making the probe itself time out. Probes multiply: 30 pods probed every 10 seconds is 3 queries a second, which is fine for SELECT 1 and not for a real query or a paid API. Register optional dependencies (a recommendations service) with failureStatus: HealthStatus.Degraded: visible on dashboards, still 200, pod stays in rotation.

Never expose details publicly

A JSON writer listing every check with exception messages is useful to operators and to attackers: host names, database names, versions, internal error text. Keep public endpoints on the default plain-text writer and always put the details endpoint behind RequireAuthorization, ideally also on a management port with no ingress route. RequireHost("*:8081") does not do that: it matches the client-supplied Host header, so a request to the public port with Host: x:8081 passes. Check the port the connection arrived on, HttpContext.Connection.LocalPort, in an endpoint filter or middleware. If the app has a fallback authorization policy, probe endpoints need AllowAnonymous() because the kubelet sends no token, and rate limiting and output caching must not apply to them.

Failure modes

  • Requests refused during shutdown: a terminating pod leaves the Service endpoints because it is terminating, not because readiness fails, and kube-proxy and the ingress take seconds to notice while Kestrel stops accepting connections on SIGTERM. Add a preStop sleep of 5-10 s, and set HostOptions.ShutdownTimeout and terminationGracePeriodSeconds so in-flight requests finish (see net-zero-downtime).
  • A shared dependency in readiness: when the one database every replica uses is down, all pods leave rotation together and clients get the ingress's 503 instead of your ProblemDetails. Put per-instance dependencies in readiness; for a shared one, accept that trade-off deliberately or report it as Degraded and let requests fail with a proper error.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Take one pod out of the load balancer while its local cache sidecar is unreachable, without Kubernetes restarting it or touching the other replicas.
  • Report a slow recommendations service as Degraded on dashboards while orders keep being accepted.
  • Give operators a JSON report of every check on a management port that the public ingress never routes to.

More examples

PostgreSQL and an optional dependency

<code>AddNpgSql</code> comes from the AspNetCore.HealthChecks.NpgSql package and runs <code>SELECT 1;</code>. With the connection-string overload it uses its own data source and pool, so it proves the database answers but not that the app's pool has free connections; resolve the app's <code>NpgsqlDataSource</code> from DI if you want the check to share that pool. The recommendations check is registered as Degraded: when that service fails, readiness still returns 200 and the pod keeps taking orders, while dashboards show the degradation. Returning <code>context.Registration.FailureStatus</code> keeps the decision in the registration, not hard-coded in the check.

Example · csharp
using Microsoft.Extensions.Diagnostics.HealthChecks;

var builder = WebApplication.CreateBuilder(args);
var ordersDb = builder.Configuration.GetConnectionString("Orders")
    ?? throw new InvalidOperationException("ConnectionStrings:Orders is not configured");

builder.Services.AddHttpClient("recommendations", c => c.BaseAddress = new Uri("http://recommendations:8080"));
builder.Services.AddHealthChecks()
    .AddNpgSql(ordersDb, name: "orders-db", tags: ["ready"], timeout: TimeSpan.FromSeconds(3))
    .AddCheck<RecommendationsCheck>("recommendations",
        failureStatus: HealthStatus.Degraded, tags: ["ready"], timeout: TimeSpan.FromSeconds(2));

var app = builder.Build();
app.MapHealthChecks("/health/live", new() { Predicate = _ => false });
app.MapHealthChecks("/health/ready", new() { Predicate = c => c.Tags.Contains("ready") });
app.Run();

public sealed class RecommendationsCheck(IHttpClientFactory clients) : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct)
    {
        using var response = await clients.CreateClient("recommendations").GetAsync("/health/live", ct);
        return response.IsSuccessStatusCode
            ? HealthCheckResult.Healthy()
            : new HealthCheckResult(context.Registration.FailureStatus, $"status {(int)response.StatusCode}");
    }
}

Kubernetes probes wired to the endpoints

The startup probe allows up to 60 seconds (30 x 2 s) before liveness starts counting, so a slow start is not mistaken for a hang. It hits <code>/health/live</code>, not readiness: past its threshold the kubelet restarts the container, so a database outage during a rollout would otherwise crash-loop every new pod. Readiness <code>timeoutSeconds: 5</code> is longer than the 3-second check timeout in code, so a slow database is reported as Unhealthy rather than as a probe timeout. Liveness only hits <code>/health/live</code>, which checks nothing external.

Example · yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: orders-api
          image: registry.example.com/shop/orders-api:1.4.2
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /health/live
              port: 8080
            periodSeconds: 2
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3

Health endpoints next to a fallback policy

With a fallback policy every endpoint without metadata requires a user, so without <code>AllowAnonymous()</code> the probes get 401, Kubernetes marks every pod unready and the service disappears right after deploy. The details endpoint checks <code>Connection.LocalPort</code>, the port the connection really arrived on; <code>RequireHost</code> would match only the <code>Host</code> header, which any client can spoof. The operator role requirement is the real protection, in case the port is reachable from inside the cluster.

Example · csharp
using Microsoft.AspNetCore.Authorization;

var builder = WebApplication.CreateBuilder(args);
builder.WebHost.UseUrls("http://+:8080", "http://+:8081"); // 8081: management, no ingress route
builder.Services.AddAuthentication().AddJwtBearer(); // the only scheme, so it is the default challenge
builder.Services.AddAuthorization(options =>
{
    // Every endpoint requires an authenticated user unless it opts out.
    options.FallbackPolicy = new AuthorizationPolicyBuilder().RequireAuthenticatedUser().Build();
});
builder.Services.AddHealthChecks();

var app = builder.Build();
app.UseAuthentication();
app.UseAuthorization();

// The kubelet sends no token, so probes must be anonymous.
app.MapHealthChecks("/health/live", new() { Predicate = _ => false }).AllowAnonymous();
app.MapHealthChecks("/health/ready").AllowAnonymous();

// Full report only on the management port and only for operators.
app.MapHealthChecks("/health/details")
    .AddEndpointFilter(async (ctx, next) =>
        ctx.HttpContext.Connection.LocalPort == 8081 ? await next(ctx) : Results.NotFound())
    .RequireAuthorization(policy => policy.RequireRole("operator"));

app.MapGet("/orders/{id:int}", (int id) => Results.Ok(new { id, status = "shipped" }));
app.Run();

Discussion

  • Be the first to comment on this lesson.