Deploying and Operating It
Apply the Deployment, Service and Ingress, wire the probes, then use logs, metrics and kubectl to find what broke.
kubectl apply -f deploy/ && kubectl rollout status deploy/notes --timeout=120sDeploying is handing the cluster a description of what you want. Operating is everything after that. Both come down to a few files and a few numbers.
Applying it
Three objects describe the service. A Deployment names the image, how many copies to run, the environment variables, the probes and the resources. A Service gives those copies one stable address. An Ingress maps a public hostname onto the Service. kubectl apply -f deploy/ sends them, and kubectl rollout status deploy/notes waits and fails if the new pods never become ready. Plain settings come from a ConfigMap, the database password from a Secret; both arrive as environment variables, which is what the config loader already reads.
Probes and shutdown
The liveness path answers for the process alone and checks nothing outside it. The readiness path checks the database pool, so a pod with no working database leaves the Service instead of being restarted. On SIGTERM the process stops accepting connections, finishes the requests in flight and exits, which is what makes a rollout invisible to clients.
What to look at when it misbehaves
Four numbers cover most of it: request rate, error rate, latency, and saturation such as how often a request waited for a database connection. Then the commands. kubectl get pods shows restarts. kubectl logs --previous shows why the last container died. kubectl describe pod shows probe failures and an OOMKilled termination. kubectl get endpoints notes shows whether the Service has any pods at all, which is the usual answer to a 503 from the Ingress.
Example
// Command notes shows what an operator sees: a readiness gate, a counter
// and one structured log line per request.
// Against a real server: curl -i localhost:8080/readyz
package main
import (
"fmt"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"sync/atomic"
)
type app struct {
log *slog.Logger
ready atomic.Bool
requests atomic.Int64
}
// readyz is what the readiness probe calls. Failing it takes this pod out
// of the Service; nothing is restarted.
func (a *app) readyz(w http.ResponseWriter, r *http.Request) {
a.requests.Add(1)
if !a.ready.Load() {
http.Error(w, "not ready", http.StatusServiceUnavailable)
a.log.Warn("readiness failed", "reason", "database pool not open")
return
}
fmt.Fprintln(w, "ready")
a.log.Info("request handled", "path", r.URL.Path, "status", http.StatusOK)
}
func main() {
// Dropping the timestamp keeps this output fixed. A real service keeps
// it, because that is what you search logs by.
opts := &slog.HandlerOptions{ReplaceAttr: func(groups []string, at slog.Attr) slog.Attr {
if at.Key == slog.TimeKey {
return slog.Attr{}
}
return at
}}
a := &app{log: slog.New(slog.NewJSONHandler(os.Stdout, opts))}
call := func() int {
rec := httptest.NewRecorder()
a.readyz(rec, httptest.NewRequest(http.MethodGet, "/readyz", nil))
return rec.Code
}
fmt.Println("before the pool opens:", call())
a.ready.Store(true) // set once migrations ran and the pool answered
fmt.Println("after the pool opens:", call())
fmt.Println("requests served:", a.requests.Load())
}
// Output:
// {"level":"WARN","msg":"readiness failed","reason":"database pool not open"}
// before the pool opens: 503
// {"level":"INFO","msg":"request handled","path":"/readyz","status":200}
// after the pool opens: 200
// requests served: 2When to use it
- Roll out a new image and have the pipeline fail by itself if the new pods never become ready.
- Find out in under a minute why a pod is in CrashLoopBackOff after a deploy.
- Diagnose a 503 that no handler logged, by checking whether the Service has any ready pods.
More examples
The Deployment
envFrom and env are the two spellings of the same thing: envFrom takes every key in a ConfigMap or Secret as a variable, while env names them one at a time. Use envFrom for the bulk and env for the few you want visible in the file. Three other choices are deliberate. There is a memory limit but no CPU limit, because exceeding a memory limit kills the container while exceeding a CPU limit only throttles it, which looks like unexplained latency. The preStop sleep gives the cluster five seconds to stop sending traffic before the process is asked to stop; the sleep handler is enabled by default from Kubernetes 1.30, and an exec handler could not be used here because a distroless image has no /bin/sleep. runAsUser is the number, not the name, because the kubelet cannot look a name up.
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 3
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
terminationGracePeriodSeconds: 30
securityContext:
runAsNonRoot: true
runAsUser: 65532
containers:
- name: api
image: registry.example.com/notes:9f3c1ab
ports:
- containerPort: 8080
# envFrom turns every key in these objects into a variable.
envFrom:
- configMapRef:
name: notes-config
- secretRef:
name: notes-db
# The same thing written out one variable at a time.
env:
- name: PORT
value: "8080"
readinessProbe:
httpGet:
path: /readyz
port: 8080
periodSeconds: 5
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
memory: 256Mi
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
lifecycle:
preStop:
sleep:
seconds: 5Rolling it out, and reading a failure
This is the first ninety seconds of every bad deploy. rollout status is the part people skip; without it the apply succeeds, the pipeline goes green and the pods never start. The three commands after it answer three different questions: get pods says whether the container stays up, logs --previous says why the last one died, and describe says what the cluster thinks rather than what the program printed. A startup that exits on a missing setting, as here, makes this a thirty-second fix instead of an afternoon.
kubectl apply -f deploy/
# deploy/notes, deployment/notes and deployments/notes all name the same
# object; deploy is kubectl's shorthand for the long form.
kubectl rollout status deploy/notes --timeout=120s
# Waiting for deployment "notes" rollout to finish: 1 of 3 updated replicas are available...
kubectl get pods -l app=notes
# NAME READY STATUS RESTARTS AGE
# notes-7c9d8b6f45-2xq8r 0/1 CrashLoopBackOff 3 70s
# The container that failed is already gone, so ask for its output.
kubectl logs deploy/notes --previous | tail -3
# config: DATABASE_URL is required
# Events say what the kubelet saw: failed probes, OOMKilled, image pull errors.
kubectl describe pod -l app=notes | tail -20
# Put the previous image back. The tag is a commit, so there is one to go to.
kubectl rollout undo deploy/notesLooking at a service that is up but wrong
A service that is running and still wrong needs different tools from one that will not start. Empty endpoints is the single most common cause of a 503 that no pod logged, and it means readiness is failing; endpointslices is the newer object holding the same information. A goroutine count that only grows is the signature of a leak, usually a request whose context was never cancelled, and the goroutine profile prints the stacks so you can see which. The last command is why request ids are worth the middleware: one grep returns every line belonging to one request, across all three replicas.
# 503 from the Ingress, pods look fine? Ask whether the Service has any
# pods it is willing to send to. Readiness failing empties this list.
kubectl get endpoints notes
# NAME ENDPOINTS AGE
# notes <none> 6d
kubectl get endpointslices -l kubernetes.io/service-name=notes
# Reach one pod's own numbers without exposing them to the internet.
kubectl port-forward deploy/notes 8080:8080
# In another terminal. These paths exist because main registers the metrics
# handler and imports net/http/pprof.
curl -s localhost:8080/metrics | grep -E 'http_requests_total|go_goroutines'
# go_goroutines 2184
# Goroutines that climb and never fall are a leak; the stacks name the line.
curl -s 'localhost:8080/debug/pprof/goroutine?debug=1' | head -20
# One request's whole story, by the id the middleware put on every line.
kubectl logs deploy/notes --since=15m | grep 7f3c1ab2
Discussion