Health Checks with Terminus

Expose separate liveness and readiness endpoints with @nestjs/terminus, and check dependencies only in readiness.

Syntaxhealth.check([() => this.db.pingCheck('database').withTimeout(1500)])

An orchestrator asks two different questions. Liveness: is this process broken beyond recovery, should it be killed and restarted? Readiness: can it serve a request right now, should traffic be sent to it? Answer them on separate endpoints, because the consequences are not the same.

What each one checks

Liveness checks the process and nothing else: if it answers, it is alive. Put no dependency in it. A database outage would otherwise fail liveness on every instance at once, and the orchestrator would restart the whole fleet while the database is already struggling. Readiness checks what a request cannot be served without — the primary database, a broker you publish to — and stops there.

Terminus

@nestjs/terminus gives you HealthCheckService, whose check() takes an array of functions, runs them all, and returns { status, info, error, details }. One down indicator makes the overall status error and the response 503. Terminus 12 adds a third state: degraded reports impairment while keeping the 200, which is what an optional cache deserves. It also adds .attempt(fn) with .withTimeout(ms) and .cacheFor(ms) for your own indicators; Terminus 11 offered only up() and down().

What trips people up

A readiness check heavy enough to hurt is its own outage. SELECT 1 with a timeout shorter than the probe interval is enough.

Example

Example · typescript
// src/health/health.controller.ts
import { Controller, Get } from '@nestjs/common';
import {
  HealthCheck,
  HealthCheckService,
  MemoryHealthIndicator,
  TypeOrmHealthIndicator,
} from '@nestjs/terminus';
import { RedisHealthIndicator } from './redis.health.js';

@Controller('health')
export class HealthController {
  constructor(
    private readonly health: HealthCheckService,
    private readonly db: TypeOrmHealthIndicator,
    private readonly memory: MemoryHealthIndicator,
    private readonly redis: RedisHealthIndicator,
  ) {}

  // Liveness: no dependency of any kind. The only failure it can report is
  // this process being past saving. @HealthCheck() sets
  // Cache-Control: no-cache, no-store, must-revalidate and, when
  // @nestjs/swagger is installed, documents the 200 and 503 bodies.
  @Get('live')
  @HealthCheck()
  live() {
    return this.health.check([
      // Resident set size over 1 GiB means this instance is heading for OOM.
      () => this.memory.checkRSS('memory_rss', 1024 * 1024 * 1024),
    ]);
  }

  // Readiness: everything a request needs. A down indicator answers 503, so
  // traffic is withdrawn without the process being restarted.
  @Get('ready')
  @HealthCheck()
  ready() {
    return this.health.check([
      // pingCheck runs SELECT 1 on the TypeORM connection of this injector.
      // withTimeout replaces the deprecated { timeout } option in Terminus 12.
      () => this.db.pingCheck('database').withTimeout(1500),
      () => this.redis.isHealthy('cache'),
    ]);
  }
}

When to use it

  • Give Kubernetes a livenessProbe that does not restart pods during a database outage.
  • Withdraw an instance from the load balancer while its broker connection is down.
  • Report a cache outage as degraded so requests keep being served from the origin.
  • Finish in-flight requests during a rollout instead of answering with resets.

More examples

Module wiring and draining on SIGTERM

Use the graceful window whenever a rollout drops requests. Between the SIGTERM and the actual close, readiness reports shutting_down, the orchestrator removes the instance from its pool, and in-flight requests finish on the old process instead of being cut off.

Example · typescript
// src/health/health.module.ts
import { Module } from '@nestjs/common';
import { TerminusModule } from '@nestjs/terminus';
import { HealthController } from './health.controller.js';
import { RedisHealthIndicator } from './redis.health.js';

@Module({
  imports: [
    // forRoot() is the dynamic-module form: a static method that returns a
    // module description, so options can be passed in.
    TerminusModule.forRoot({
      errorLogStyle: 'pretty', // 'json' is the default
      // On a termination signal the health endpoints answer
      // { status: 'shutting_down' } with 503 for this long before the app
      // actually closes, so a load balancer can drain. Default 0.
      gracefulShutdownTimeoutMs: 5_000,
    }),
  ],
  controllers: [HealthController],
  providers: [RedisHealthIndicator],
})
export class HealthModule {}

// src/main.ts (excerpt)
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module.js';

const app = await NestFactory.create(AppModule);
// Required: shutdown listeners are off by default, so without this the
// graceful window never starts and onApplicationShutdown never fires.
app.enableShutdownHooks();
await app.listen(3000);

A custom indicator, up, down and degraded

Write your own indicator for anything Terminus has no package for: a feature-flag service, a licence server, a mounted share. Choose degraded over down when the request still succeeds, slower or with fewer features, so readiness does not pull a working instance out of rotation.

Example · typescript
// src/health/redis.health.ts
import { Injectable } from '@nestjs/common';
import { HealthIndicatorService } from '@nestjs/terminus';
import Redis from 'ioredis';

@Injectable()
export class RedisHealthIndicator {
  constructor(
    private readonly healthIndicatorService: HealthIndicatorService,
    private readonly redis: Redis,
  ) {}

  // check(key) opens a session for one entry in the result's details object.
  isHealthy(key: string) {
    return this.healthIndicatorService
      .check(key)
      .attempt(async ({ signal }) => {
        // attempt() reports up if this resolves and down if it throws. The
        // AbortSignal is aborted when withTimeout expires, so pass it to
        // anything that accepts one.
        const pong = await this.redis.ping();
        signal.throwIfAborted();
        return { response: pong }; // merged into details[key]
      })
      .withTimeout(1000)
      // Terminus 12: share one result between probes that arrive together,
      // marked cachedResponse: true.
      .cacheFor(2000);
  }

  // The cache is optional, so losing it is impairment, not failure:
  // the response stays 200 and the overall status becomes 'degraded'.
  async isDegradableCache(key: string) {
    const indicator = this.healthIndicatorService.check(key);

    try {
      await this.redis.ping();
      return indicator.up({ mode: 'cached' });
    } catch {
      return indicator.degraded('cache unreachable, serving from origin');
    }
  }
}

A deep check that is not a probe

Keep an endpoint like this for humans and dashboards, and keep it out of your probe configuration. Checking downstream services in readiness couples your availability to theirs: one slow neighbour and every instance of your service is removed from rotation at the same moment.

Example · typescript
// src/health/dependencies.controller.ts
import { Controller, Get } from '@nestjs/common';
import {
  DiskHealthIndicator,
  HealthCheck,
  HealthCheckService,
  HttpHealthIndicator,
} from '@nestjs/terminus';

@Controller('health')
export class DependenciesController {
  constructor(
    private readonly health: HealthCheckService,
    private readonly http: HttpHealthIndicator,
    private readonly disk: DiskHealthIndicator,
  ) {}

  @Get('deep')
  @HealthCheck()
  deep() {
    return this.health.check([
      // pingCheck needs HttpModule from @nestjs/axios imported in this module.
      () =>
        this.http.pingCheck('payments', 'https://payments.internal/healthz', {
          timeout: 800,
        }),
      // responseCheck decides from the response itself rather than from the
      // absence of an error.
      () =>
        this.http.responseCheck(
          'search',
          'https://search.internal/status',
          (res) => res.status === 204,
        ),
      // thresholdPercent 0.9: down once the volume is 90% full.
      () =>
        this.disk.checkStorage('uploads', {
          path: '/var/lib/uploads',
          thresholdPercent: 0.9,
        }),
    ]);
  }
}

Related

nest shutdownnest k8snest rolloutsnest metricsnest logging

Discussion

  • Be the first to comment on this lesson.