Metrics
Counters, gauges and histograms: the handful of numbers every service should expose, and what each one answers.
new Counter({ name: 'http_requests_total', help: 'Requests served', labelNames: ['route', 'status'] })A metric is a number about your service, sampled over time: requests per second, errors per second, latency, memory in use. Logs tell you what happened in one request. Metrics tell you how all of them are going. A service exposes them as plain text on an HTTP route, usually /metrics, on a port: the numbered door on a machine that a server listens on. A collector fetches that route every few seconds and stores what it finds.
Three shapes
- Counter: only ever goes up. Total requests, total errors.
- Gauge: goes up and down. Queue depth, database connections in use.
- Histogram: counts observations into buckets, so percentiles can be computed later. Request duration.
The handful worth having
Request rate, error rate, a duration histogram, and one saturation number per limited resource. Those answer, in order: is it being used, is it broken, is it slow, is it about to run out of something. An average latency hides the slow tail, which is why duration is a histogram and not a gauge.
Node supplies some figures itself. process.memoryUsage() reports heap use, and monitorEventLoopDelay({ resolution: 10 }) from node:perf_hooks records how late the event loop is running. Event loop delay rises before users notice anything.
Labels
Labels split one metric by route or status code. Every distinct combination of label values is a separately stored series.
Example
When to use it
- Page someone when the 5xx rate for a route crosses one percent of its traffic.
- Show the 99th percentile checkout latency on a dashboard, not the average.
- Watch connection pool saturation to decide whether the pool is too small.
- Catch a memory leak as a slow rise in heap used across a week.
More examples
Exposing /metrics with prom-client
The whole metrics surface of a small service: default runtime metrics, one counter, one route that renders them. register.contentType carries the exposition format version, which the collector checks, so do not hand-write that header.
const http = require('node:http');
const client = require('prom-client');
// Heap, CPU, event loop lag and open handles, named the way collectors expect.
client.collectDefaultMetrics();
const requests = new client.Counter({
name: 'http_requests_total',
help: 'Requests served',
labelNames: ['route', 'status'],
});
http.createServer(async (req, res) => {
if (req.url === '/metrics') {
// register.metrics() returns a promise.
const body = await client.register.metrics();
res.writeHead(200, { 'content-type': client.register.contentType });
res.end(body);
return;
}
requests.inc({ route: '/', status: 200 });
res.writeHead(200).end('ok');
}).listen(3000);Timing a handler into a histogram
startTimer returns a function that records the elapsed seconds when called, and accepts the remaining label values at that point. The finally block matters: without it, a thrown error leaves the request uncounted, so errors look free.
const client = require('prom-client');
const duration = new client.Histogram({
name: 'http_request_duration_seconds',
help: 'Request duration',
labelNames: ['route', 'status'],
// Buckets are in seconds, and must straddle your latency target.
buckets: [0.01, 0.05, 0.1, 0.3, 1, 3],
});
function withTiming(route, handler) {
return async (req, res) => {
const end = duration.startTimer({ route }); // returns a function
try {
await handler(req, res);
} finally {
// Label values can be completed at the end, once the status is known.
end({ status: res.statusCode });
}
};
}Event loop delay as a gauge
Event loop delay is the queue of callbacks waiting for their turn. When it climbs, every request slows down at once, whatever the cause. This is the one Node-specific metric worth adding by hand if the default set is not collected.
const { monitorEventLoopDelay } = require('node:perf_hooks');
const client = require('prom-client');
// resolution is the sampling interval in milliseconds; default 10.
const histogram = monitorEventLoopDelay({ resolution: 10 });
histogram.enable();
new client.Gauge({
name: 'nodejs_event_loop_delay_p99_seconds',
help: '99th percentile event loop delay',
// collect() runs when the collector scrapes.
collect() {
// percentile() returns nanoseconds.
this.set(histogram.percentile(99) / 1e9);
histogram.reset(); // start a fresh window for the next scrape
},
});
Discussion