Metrics
Counters, gauges and histograms explained, plus the five numbers worth exporting from every Go service.
var requests = expvar.NewInt("http_requests_total") // then: requests.Add(1)A metric is a number about your program that something else collects over time. Logs tell you what happened in one request; metrics tell you what is happening across all of them.
The three shapes
- A counter only ever goes up. Requests served, errors returned, messages published. You read the rate of change, not the value.
- A gauge goes up and down. Requests in flight, items queued, open database connections.
- A histogram counts observations into buckets, so you can ask what fraction of requests finished under 100ms. Averages hide the slow tail; histograms do not.
What to export
Five numbers cover most incidents: requests served, requests that failed, request duration as a histogram, requests in flight, and the latency of each outside dependency. Label the first three by method, route pattern and status code, so you can see that one route is failing rather than that something, somewhere, is.
Where the numbers live
The standard library ships expvar, which keeps named values in memory and serves them as JSON. It has counters and gauges but no histograms, so production services usually use the Prometheus client library instead. Either way the pattern is the same: declare the metric once at package level, update it from a middleware, and expose one HTTP route that a collector scrapes.
What trips people up
Labels multiply. A label whose value is a user ID or a raw path containing an ID creates a new series per value, and the collector falls over. Label with the route pattern, never the request path.
Example
package main
import (
"expvar"
"fmt"
"net/http"
"net/http/httptest"
)
// Declared once, at package level. NewInt publishes the value under this name.
var (
requests = expvar.NewInt("http_requests_total") // a counter
inFlight = expvar.NewInt("http_in_flight") // used as a gauge
)
// measure wraps any handler so every request through it is counted.
func measure(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests.Add(1)
inFlight.Add(1)
defer inFlight.Add(-1) // runs when this request finishes
next.ServeHTTP(w, r)
})
}
func main() {
mux := http.NewServeMux()
mux.Handle("GET /debug/vars", expvar.Handler()) // every published value, as JSON
mux.Handle("GET /", measure(http.NotFoundHandler()))
// Three requests, in memory, so the numbers below are fixed.
// "for range 3" is shorthand for "for i := 0; i < 3; i++" (Go 1.22 and later).
for range 3 {
mux.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
}
fmt.Println("http_requests_total:", requests.Value())
fmt.Println("http_in_flight:", inFlight.Value())
}
// Output:
// http_requests_total: 3
// http_in_flight: 0When to use it
- Show that the 99th percentile of one API route went from 80ms to 2s after a deploy.
- Alert when the error counter's rate crosses a threshold, instead of when a human reads a log.
- Watch in-flight requests and pool wait counts to see whether a service is saturated or just slow.
- Prove that a background worker is still consuming its queue.
More examples
A gauge that is computed when it is read
Some numbers already exist somewhere and only need exposing. expvar.Func wraps a function so the value is produced on each scrape rather than tracked by your code. Reach for this for pool sizes, queue lengths and cache entry counts. Keep the function fast: it runs inside the HTTP handler that serves the metrics.
// expvar.NewInt stores a number you update. expvar.Publish stores anything
// that can turn itself into JSON, including a function called at read time.
func publishPoolStats(db *sql.DB) {
expvar.Publish("db_open_connections", expvar.Func(func() any {
return db.Stats().OpenConnections
}))
expvar.Publish("db_wait_count", expvar.Func(func() any {
return db.Stats().WaitCount // requests that had to wait for a connection
}))
}
// The longhand of expvar.NewInt("x") is these two lines:
// v := new(expvar.Int)
// expvar.Publish("x", v)
// NewInt does exactly that and returns v.Duration histogram with the Prometheus client
This is the one metric you cannot build with expvar, and the one you miss most during an incident, because it answers "how slow is the slowest tenth of requests" rather than "what is the average". The route label is the registered pattern so that a thousand order IDs produce one series. Mount prometheus/client_golang's promhttp.Handler() on /metrics to expose it.
// github.com/prometheus/client_golang is a third-party library, added with
// go get. Buckets are seconds; DefBuckets ranges from 5ms to 10s.
var duration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Help: "Time to serve a request.",
Buckets: prometheus.DefBuckets,
}, []string{"method", "route", "status"})
func init() { prometheus.MustRegister(duration) }
func instrument(route string, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
rec := &statusRecorder{ResponseWriter: w, status: 200}
next.ServeHTTP(rec, r)
// route is the pattern ("/orders/{id}"), never r.URL.Path.
duration.WithLabelValues(r.Method, route, strconv.Itoa(rec.status)).
Observe(time.Since(start).Seconds())
})
}Counting work in a background worker
Services without HTTP traffic still need numbers: a worker that quietly stops is invisible unless something counts the work. Two counters plus a depth gauge tell you throughput, failure rate and whether the queue is growing. The worker still needs one HTTP route serving expvar.Handler() so the collector can reach it.
var (
jobsDone = expvar.NewInt("jobs_done_total")
jobsFailed = expvar.NewInt("jobs_failed_total")
queueDepth = expvar.NewInt("jobs_queued")
)
func run(ctx context.Context, jobs <-chan Job) {
for job := range jobs {
queueDepth.Add(-1)
if err := handle(ctx, job); err != nil {
jobsFailed.Add(1) // one counter per outcome, not a label per error text
continue
}
jobsDone.Add(1)
}
}
Discussion