Health Checks and Metrics
Liveness asks whether to restart you; readiness asks whether to send you traffic. Mixing them restarts a healthy fleet.
GET /livez -> 200 always; GET /readyz -> 200 or 503Two probes, two different questions. Liveness asks "is this process wedged?", and the only remedy is a restart. Readiness asks "should traffic go here right now?", and the remedy is to take the instance out of rotation and put it back later. Answering both from one handler is how a thirty-second database blip restarts every replica at once.
What each endpoint checks
Liveness should touch nothing external: return 200 if the process is still serving. Readiness checks the dependencies you cannot work without and returns 503 when one is down, so the load balancer stops sending requests while the process keeps its connections and finishes what it has. For a slow boot use a startup probe rather than a long initialDelaySeconds on liveness, or the pod is killed while it is still warming up.
Checks need a budget
Give every dependency check its own timeout, and cache the result for a second or two. A probe every five seconds across fifty replicas is real, pointless load on the database you are asking about.
Counters worth exporting
Monotonic counters labelled by route and status code: requests total and errors total, plus a gauge for in-flight requests and a histogram for latency. One sample per line in the Prometheus text format, as in http_requests_total{route="/orders",status="200"} 2. Export raw totals and let the query compute rates and ratios; a pre-averaged number cannot be re-aggregated. Keep labels low-cardinality — a user id, or a path with an id still in it, multiplies your series count until the scrape times out.
Example
When to use it
- Keep a brief database outage from restarting every replica at the same moment.
- Drain traffic from one instance while it reconnects, without killing it.
- Give a service that runs migrations on boot five minutes to start without weakening liveness.
- Alert on an error ratio computed from exported totals rather than on log lines.
More examples
Serving both endpoints on the stdlib server
Running the probes on their own port and their own thread means a saturated application worker pool does not make the process look dead. Overriding log_message is not cosmetic: at one probe every two seconds, the default handler writes about 43,000 lines a day per replica.
# health.py - nothing to install, and independent of the app's framework
import json
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1" # requires an accurate Content-Length on every reply
def _json(self, status, body):
payload = json.dumps(body).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
def do_GET(self):
if self.path == "/livez":
self._json(200, {"status": "ok"}) # no dependency calls, ever
elif self.path == "/readyz":
checks = {"postgres": ping_postgres(), "redis": ping_redis()}
ok = all(checks.values())
self._json(200 if ok else 503, {"ready": ok, "checks": checks})
else:
self._json(404, {"error": "not found"})
def log_message(self, fmt, *args):
pass # probe traffic goes nowhere near your JSON logs
ThreadingHTTPServer(("", 8081), Handler).serve_forever()The probes in the deployment
Liveness is deliberately slower and more forgiving than readiness, because its action is destructive. The startup probe exists so you do not have to set initialDelaySeconds high enough for the worst boot, which would also delay detection of a genuine hang by the same amount.
containers:
- name: api
startupProbe:
# Covers a slow boot - migrations, a warm cache - without making
# liveness tolerant of a wedged process forever.
httpGet: { path: /livez, port: 8080 }
periodSeconds: 5
failureThreshold: 60 # up to five minutes to start
livenessProbe:
httpGet: { path: /livez, port: 8080 }
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3 # 30s of silence before a restart
readinessProbe:
httpGet: { path: /readyz, port: 8080 }
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 1 # leave the rotation fast
successThreshold: 2 # rejoin only once it is actually settledA dependency check with a timeout and a cache
The broad except is correct here and almost nowhere else: a readiness endpoint that raises gives the orchestrator a 500 it has to interpret, when what you want is an explicit false. The timeout passed into the real call is what keeps a hung socket from hanging the probe.
import threading
import time
class CachedCheck:
"""Fifty replicas probing every 5s is 10 checks a second on the database.
Cache the answer briefly; the probe interval already decides how fresh
the information needs to be."""
def __init__(self, fn, ttl=2.0, timeout=1.0):
self.fn, self.ttl, self.timeout = fn, ttl, timeout
self.lock = threading.Lock()
self.checked_at = 0.0
self.ok = False
def __call__(self):
now = time.monotonic() # monotonic: immune to a clock correction
with self.lock:
if now - self.checked_at < self.ttl:
return self.ok
try:
self.ok = bool(self.fn(timeout=self.timeout))
except Exception:
# A failed check is a "not ready", never a 500 from the probe
# endpoint - the orchestrator must get an answer it can read.
self.ok = False
self.checked_at = now
return self.ok
ping_postgres = CachedCheck(lambda timeout: db.execute("SELECT 1", timeout=timeout))
Discussion