Running on Kubernetes

Liveness restarts the container, readiness only removes it from the Service; here are the probes, resources and env config a Nest pod needs.

SyntaxreadinessProbe: { httpGet: { path: /health/ready, port: 3000 }, periodSeconds: 5, timeoutSeconds: 2 }

Kubernetes runs your container in a pod, restarts it when it looks dead, and pulls it out of load balancing when it says it is not ready. Those are two different probes, and confusing them causes most of the pain.

Probes

A failing livenessProbe kills the container and starts a fresh one. A failing readinessProbe only removes the pod from the Service's endpoint list; the process keeps running. So liveness should test the event loop and nothing else. A liveness probe that pings the database restarts every pod in the cluster the moment the database blips. Dependency checks belong in readiness. A startupProbe covers slow boots: liveness and readiness are suspended until it passes once.

The defaults are periodSeconds: 10, timeoutSeconds: 1 and failureThreshold: 3, so a pod takes about thirty seconds to leave rotation. One second is strict; a check that queries Postgres without a shorter timeout of its own will flap.

Config and resources

Config arrives as environment variables, which ConfigModule reads like any others. envFrom with a configMapRef is shorthand for one env entry per key with valueFrom.configMapKeyRef; the two produce the same variables. Secrets use secretRef. Set the memory request equal to the limit: a Node process that exceeds its memory limit is OOM-killed by the kernel, with no stack trace and no log line.

Example

Example · typescript
// src/health/health.controller.ts
import { Controller, Get } from '@nestjs/common';
import {
  HealthCheck,
  HealthCheckService,
  MemoryHealthIndicator,
  TypeOrmHealthIndicator,
} from '@nestjs/terminus';

@Controller('health')
export class HealthController {
  constructor(
    private readonly health: HealthCheckService,
    private readonly db: TypeOrmHealthIndicator,
    private readonly memory: MemoryHealthIndicator,
  ) {}

  // Liveness: no dependencies at all. If this handler runs, the event loop is
  // turning, which is the only question liveness should ask.
  // @HealthCheck() is not an interceptor and nothing of it runs per request:
  // it applies @Header('Cache-Control', 'no-cache, no-store, must-revalidate')
  // at class-definition time, and adds the 200/503 response documentation when
  // @nestjs/swagger is installed.
  @Get('live')
  @HealthCheck()
  live() {
    return this.health.check([]);
  }

  // Readiness: the things this pod needs in order to serve traffic.
  @Get('ready')
  @HealthCheck()
  ready() {
    return this.health.check([
      // withTimeout keeps the check inside the probe's own timeoutSeconds, so
      // a hanging database produces a 503 rather than a timed-out probe.
      // cacheFor shares one result between probes that arrive close together,
      // which matters when several replicas poll the same database.
      () => this.db.pingCheck('database').withTimeout(800).cacheFor(5_000),
      () => this.memory.checkHeap('memory_heap', 512 * 1024 * 1024),
    ]);
  }
}

When to use it

  • Stop a brief database outage from restarting every replica at once.
  • Give a service that loads 40 MB of reference data at boot enough time to start.
  • Diagnose a pod that keeps vanishing with exit code 137 and no error in the logs.
  • Move database credentials out of the image and into a Secret the Deployment reads.

More examples

The Deployment

The manifest that goes with the health controller above. maxUnavailable: 0 with maxSurge: 1 means a new pod must pass its readiness probe before an old one is removed, which is what makes a deploy invisible to callers. The grace period has to be longer than the preStop sleep plus the slowest request you are willing to wait for, or Kubernetes sends SIGKILL in the middle of your teardown.

Example · yaml
# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      # Must exceed the preStop sleep plus your longest request.
      terminationGracePeriodSeconds: 45
      containers:
        - name: api
          image: myorg/api:1.4.0
          ports:
            - containerPort: 3000
          envFrom:
            - configMapRef:
                name: api-config
            - secretRef:
                name: api-secrets
          # Up to 60s to boot (30 x 2s) without liveness interfering.
          startupProbe:
            httpGet:
              path: /health/live
              port: 3000
            periodSeconds: 2
            failureThreshold: 30
          livenessProbe:
            httpGet:
              path: /health/live
              port: 3000
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 3000
            periodSeconds: 5
            timeoutSeconds: 2
          lifecycle:
            preStop:
              # Keep serving while kube-proxy on every node catches up with
              # the endpoint removal. SIGTERM is sent after this returns.
              exec:
                command: ["sleep", "5"]
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: "1"
              memory: 512Mi

envFrom and its longhand

Both forms produce the same process.env, so ConfigModule cannot tell them apart. Use envFrom for the bulk of your config, and fall back to the longhand when you need to rename a key, read from a different ConfigMap, or override one value for a single Deployment. The precedence rule is worth remembering: an explicit env entry beats whatever envFrom brought in under the same name.

Example · yaml
# k8s/deployment.yaml — the container's env, written out both ways
env:
  # Longhand: one entry per variable, each naming its source key.
  - name: NODE_ENV
    valueFrom:
      configMapKeyRef:
        name: api-config
        key: NODE_ENV
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: api-secrets
        key: DATABASE_URL
  # A literal value, and an override: entries in env win over envFrom.
  - name: LOG_LEVEL
    value: debug

envFrom:
  # Shorthand: every key in the ConfigMap and the Secret becomes a variable
  # under its own name. The first two blocks above are what two of those keys
  # look like when you expand them by hand.
  - configMapRef:
      name: api-config
  - secretRef:
      name: api-secrets

A readiness indicator that reports the shutdown

This closes the window the preStop sleep opens from the other side: the moment SIGTERM arrives, readiness starts failing, and Kubernetes removes the pod from the Service without waiting for a dependency check to notice anything. Reach for HealthIndicatorService whenever you need a readiness signal that no shipped indicator covers, such as a feature flag service being reachable or a warm cache being populated.

Example · typescript
// src/health/shutdown.health.ts
import { type BeforeApplicationShutdown, Injectable } from '@nestjs/common';
import { HealthIndicatorService } from '@nestjs/terminus';

// NestJS 12 removed the old HealthIndicator base class. Custom indicators are
// now built from HealthIndicatorService, which hands you a session keyed by
// the name that will appear in the response body.
@Injectable()
export class ShutdownHealthIndicator implements BeforeApplicationShutdown {
  private shuttingDown = false;

  constructor(private readonly indicators: HealthIndicatorService) {}

  beforeApplicationShutdown(): void {
    this.shuttingDown = true;
  }

  isServing() {
    const serving = this.indicators.check('serving');
    // down('shutting down') is shorthand for down({ message: 'shutting down' }).
    // Both produce { serving: { status: 'down', message: 'shutting down' } }.
    return this.shuttingDown ? serving.down('shutting down') : serving.up();
  }
}

// In HealthController.ready(), add it to the list:
//   () => this.shutdown.isServing(),

Related

nest healthnest shutdownnest rolloutsnest confignest metrics

Discussion

  • Be the first to comment on this lesson.