Health and Readiness Endpoints
Liveness says the process is alive, readiness says it can serve; checking a database in the wrong one restarts healthy pods.
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) })A health endpoint is an HTTP route that something outside your program calls to ask how the program is doing. Kubernetes calls two of them, and they ask different questions.
Liveness asks whether the process is still running. It should check almost nothing. Returning 200 is enough: the fact that the handler ran at all proves the process accepts connections and can still schedule work.
Readiness asks whether this instance can serve a request right now. It checks the things the service cannot work without, such as the database connection pool. When one of those is unusable it returns 503, the status code that means the service is temporarily unavailable.
Why the split matters
A failing liveness probe restarts the container. A failing readiness probe only removes the instance from the list of addresses that receive traffic; the instance keeps running and keeps being probed. So if liveness checks the database and the database stalls for ten seconds, every instance restarts at once, and a short dependency outage becomes a long one. Dependency checks belong in readiness only.
Keeping it cheap
A probe may run every two seconds on every instance. Give each check a deadline so one hanging dependency cannot pile up handlers. Do not run migrations, count rows, or take a lock. During shutdown, make readiness fail first, then keep serving the requests already in flight while traffic drains away.
Example
package main
import (
"fmt"
"net/http"
"net/http/httptest"
)
// server holds what the checks need. dbUp stands in for a real pool here so
// the printed output is fixed.
type server struct {
dbUp bool
}
// live checks nothing. Running at all is the answer.
func (s *server) live(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "ok")
}
// ready reports whether this instance can serve traffic right now.
func (s *server) ready(w http.ResponseWriter, r *http.Request) {
if !s.dbUp {
http.Error(w, "database unreachable", http.StatusServiceUnavailable) // 503
return
}
fmt.Fprintln(w, "ok")
}
func main() {
s := &server{dbUp: false}
mux := http.NewServeMux()
// HandleFunc is shorthand for mux.Handle(pattern, http.HandlerFunc(fn)).
mux.HandleFunc("GET /healthz", s.live)
mux.Handle("GET /readyz", http.HandlerFunc(s.ready))
// Call both routes in memory instead of starting a listener.
for _, path := range []string{"/healthz", "/readyz"} {
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, httptest.NewRequest("GET", path, nil))
fmt.Printf("%s -> %d\n", path, rec.Code)
}
}
// Output:
// /healthz -> 200
// /readyz -> 503When to use it
- Keep a pod serving traffic during a brief database outage instead of restarting every instance at once.
- Stop a rolling deploy from sending requests to an instance that has not finished loading its configuration.
- Drain a container cleanly on SIGTERM by failing readiness before the server stops listening.
- Give a slow-starting service room to warm up without the orchestrator calling it dead.
More examples
Readiness that checks the database with a deadline
PingContext borrows a connection from the pool and gives it up again, which is the cheapest honest check that the database is reachable. The context timeout means a dead database makes the probe fail fast instead of leaving handlers stuck. Reach for this on any service that cannot answer a single request without its database.
// readyz returns 200 only if the database answers within one second.
func readyz(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// The probe itself has a timeout; never wait longer than it does.
ctx, cancel := context.WithTimeout(r.Context(), time.Second)
defer cancel()
if err := db.PingContext(ctx); err != nil {
http.Error(w, "db: "+err.Error(), http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
fmt.Fprintln(w, "ok")
}
}Failing readiness first during shutdown
A container that stops listening the instant it is told to shut down drops requests that were already routed to it, because the thing routing them needs a few seconds to notice. Flipping readiness to failing first, sleeping, then shutting down gives those seconds. Use this in every deployment that does rolling updates.
// shuttingDown is read by the readiness handler and written by main.
// atomic.Bool is safe to use from two goroutines at once.
var shuttingDown atomic.Bool
func readyz(w http.ResponseWriter, r *http.Request) {
if shuttingDown.Load() {
http.Error(w, "shutting down", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
}
// On SIGTERM: stop being advertised, wait for traffic to move, then drain.
func onSignal(srv *http.Server) {
shuttingDown.Store(true)
time.Sleep(5 * time.Second) // let the load balancer notice
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
_ = srv.Shutdown(ctx)
}How Kubernetes is told about the two routes
The same program serves both routes; the difference lives in this configuration. The startup probe exists so a slow start does not look like a crash: liveness only begins once the startup probe passes. Write the Go side with these thresholds in mind, since failureThreshold times periodSeconds is how long a problem may last before the container is killed.
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
failureThreshold: 3 # 3 failures in a row restart the container
readinessProbe:
httpGet:
path: /readyz
port: 8080
periodSeconds: 2
failureThreshold: 1 # one failure is enough to stop sending traffic
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 30
periodSeconds: 2 # allows 60s to start before liveness applies
Discussion