Probes and Resources

Liveness restarts a wedged container, readiness takes a pod out of traffic, startup buys a slow boot time; requests and limits bound both.

SyntaxreadinessProbe.httpGet: {path: /readyz, port: http}; periodSeconds: 5; failureThreshold: 2

A probe is a question the cluster asks your container on a schedule, over the network. Your program answers with an HTTP status code. Any code from 200 to 399 passes. There are three probes and they do different things when they fail.

The three probes

The liveness probe asks whether the process is still working. After failureThreshold failures in a row the container is killed and started again. The readiness probe asks whether this pod should be sent requests. When it fails the pod is removed from its Service and traffic goes to the other pods; nothing is restarted. The startup probe covers a slow boot. While it is still failing the other two probes do not run at all, so a program that needs forty seconds to open its database pool is not killed for being slow.

The defaults are periodSeconds 10, timeoutSeconds 1, failureThreshold 3 and initialDelaySeconds 0. The time a startup probe allows is failureThreshold multiplied by periodSeconds.

Requests and limits

requests is the amount of CPU and memory the scheduler sets aside when it picks a machine for the pod. limits is the ceiling enforced while it runs. Past the CPU limit the container is throttled: it keeps running, more slowly. Past the memory limit it is killed, and the reason shown is OOMKilled.

Since Go 1.25 the runtime reads the cgroup CPU limit on Linux, and GOMAXPROCS defaults to the lower of that limit and the machine's CPU count. It ignores requests. Set GOMEMLIMIT, available since Go 1.19, a little under the memory limit so the garbage collector works harder instead of the pod being killed.

Example

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"
	"sync/atomic"
)

// health holds one flag per question the cluster can ask.
// atomic.Bool is safe to read and write from several goroutines.
type health struct {
	ready atomic.Bool // false until the service can serve traffic
}

// liveness answers "is this process wedged?" It stays cheap and
// checks nothing outside the process.
func (h *health) liveness(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK)
}

// readiness answers "should the Service send me requests?"
func (h *health) readiness(w http.ResponseWriter, r *http.Request) {
	if !h.ready.Load() {
		http.Error(w, "not ready", http.StatusServiceUnavailable)
		return
	}
	fmt.Fprintln(w, "ready")
}

// status calls a handler the way the probe would, and reports the code.
func status(h http.HandlerFunc) int {
	rec := httptest.NewRecorder()
	h(rec, httptest.NewRequest("GET", "/readyz", nil))
	return rec.Code
}

func main() {
	h := &health{}
	fmt.Println("before warm-up:", status(h.readiness))
	h.ready.Store(true) // set once the pool is open and migrations ran
	fmt.Println("after warm-up: ", status(h.readiness))
	h.ready.Store(false) // set on SIGTERM, before shutdown begins
	fmt.Println("draining:      ", status(h.readiness))
}

// Output:
// before warm-up: 503
// after warm-up:  200
// draining:       503

When to use it

Stop a fresh pod receiving traffic until its database pool is open and its cache is warm.
Let a service run migrations on boot without the cluster restarting it for taking too long.
Get a container that has deadlocked and stopped answering taken out of the Service automatically.
Size a container so the Go collector trims the heap instead of the pod being OOMKilled.

More examples

All three probes in a Deployment

The container part of a Deployment, with two endpoints doing two jobs: /healthz never touches anything outside the process, /readyz may. Because the startup probe covers the slow first minute, the liveness probe can stay strict afterwards; without it you would need a large initialDelaySeconds, which delays detection of a wedged container for the whole life of the pod. scheme: HTTP and periodSeconds: 10 are the defaults shown in full, so you can recognise the short form, which simply omits them. A gRPC service can use a grpc probe block instead of httpGet; that became a stable feature in Kubernetes 1.27.

          ports:
            - name: http
              containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz
              port: http          # the named port above; 8080 also works
            periodSeconds: 3
            failureThreshold: 20  # 3 x 20 = 60 seconds to finish booting
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
              scheme: HTTP        # HTTP is the default, written longhand
            periodSeconds: 10     # the default, written out for clarity
            timeoutSeconds: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
            failureThreshold: 2

Requests, limits, and telling the Go runtime about them

Memory request equals memory limit here on purpose. Memory cannot be throttled, so a container that is allowed more than it was promised eventually gets killed instead; asking for what you intend to use makes the pod's fate predictable, and a pod whose requests equal its limits on every resource is given the Guaranteed quality-of-service class, which is evicted last when a machine runs short. GOMEMLIMIT is a soft limit: when the heap approaches it the collector runs more often rather than letting the pod be OOMKilled. Set it below the real limit, because stacks and the runtime itself also use memory. Reach for a CPU limit when you need predictable GOMAXPROCS, and leave it off when you would rather the service use spare capacity on the machine.

          resources:
            requests:
              cpu: 100m         # 100m is one tenth of a core
              memory: 256Mi
            limits:
              cpu: "1"          # one core; the Go runtime reads this
              memory: 256Mi
          env:
            - name: GOMEMLIMIT
              value: "230MiB"   # about 90% of the memory limit

A readiness check that depends on the database

This is the handler you wire to readinessProbe, and the one place a dependency check belongs: a 503 here removes the pod from the Service until the database answers again, and no pod is restarted. Two details matter. The timeout is shorter than the probe's own timeoutSeconds, so you fail fast with a clear answer rather than being cut off mid-check. And the ready flag is checked before the ping, which is what lets a shutdown sequence flip one boolean and have the cluster stop sending work immediately.

type health struct {
	ready atomic.Bool
	db    *sql.DB
}

func (h *health) readyz(w http.ResponseWriter, r *http.Request) {
	// Still booting, or already draining.
	if !h.ready.Load() {
		http.Error(w, "not ready", http.StatusServiceUnavailable)
		return
	}
	// The probe gives up after timeoutSeconds, so bound our own wait
	// below that. r.Context() is cancelled if the probe hangs up.
	ctx, cancel := context.WithTimeout(r.Context(), 800*time.Millisecond)
	defer cancel()
	if err := h.db.PingContext(ctx); err != nil {
		http.Error(w, "database unreachable", http.StatusServiceUnavailable)
		return
	}
	w.WriteHeader(http.StatusOK)
}

Discussion

Be the first to comment on this lesson.