Writing a Service Kubernetes Can Run
Read config from the environment, log to standard output, handle SIGTERM, and keep no state on local disk.
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)Kubernetes is a system that runs programs for you across a pool of machines. It decides which machine runs your program, and it will stop that copy and start a fresh one somewhere else whenever it wants to. A service that survives that treatment follows four habits. They are often grouped under the name twelve-factor, but these four are the ones Kubernetes actually enforces.
Config comes from the environment
An environment variable is a named piece of text the operating system hands your process when it starts. Kubernetes sets them for you. Read them with os.Getenv and supply a default when the variable is empty. A config file that only exists on your laptop will not exist in the container.
Logs go to standard output
Standard output is the stream that normally reaches your terminal. Write every log line there and nowhere else. Kubernetes collects that stream and stores it. A log file written inside the container disappears when the container does. log/slog, added in Go 1.21 and still the standard logger in Go 1.27, does not write there by default: its default handler hands records to the log package, whose standard logger writes to standard error. Build a handler with os.Stdout explicitly, as the example below does.
SIGTERM means finish up
A signal is a one-word message the operating system sends a process. To remove your program Kubernetes sends SIGTERM, waits (30 seconds unless you change terminationGracePeriodSeconds), then sends SIGKILL, which cannot be caught. Use the SIGTERM window to finish requests already in flight and then exit.
Nothing important lives on local disk
The container's filesystem is thrown away on restart. Uploads, sessions and caches belong in Postgres or object storage.
The same thing, written two ways
signal.NotifyContext is shorthand. The longhand is a channel, signal.Notify, a goroutine waiting on that channel, and a call to cancel. Both give you a context that is cancelled when the signal arrives.
Example
package main
import (
"log/slog"
"os"
"strconv"
)
// getenv reads an environment variable, falling back to def when it is unset.
func getenv(key, def string) string {
if v := os.Getenv(key); v != "" {
return v
}
return def
}
// dropTime removes the timestamp so this example prints the same line every run.
// Real services keep the timestamp.
func dropTime(groups []string, a slog.Attr) slog.Attr {
if len(groups) == 0 && a.Key == slog.TimeKey {
return slog.Attr{}
}
return a
}
func main() {
// os.Stdout is the stream Kubernetes collects.
log := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{ReplaceAttr: dropTime}))
port, err := strconv.Atoi(getenv("PORT", "8080"))
if err != nil {
log.Error("PORT is not a number", "value", os.Getenv("PORT"))
os.Exit(1)
}
log.Info("starting", "port", port, "log_level", getenv("LOG_LEVEL", "info"))
}
// Output:
// {"level":"INFO","msg":"starting","port":8080,"log_level":"info"}When to use it
- Turn a working local Go server into something that can be deployed as a Kubernetes Deployment without losing requests on every rollout.
- Stop a rolling update from returning 502s by draining in-flight requests on SIGTERM.
- Move hardcoded database URLs and signing keys out of the binary and into environment variables set by a ConfigMap and a Secret.
- Make a crashing pod explain itself by failing fast with a list of missing configuration.
More examples
Catching SIGTERM and draining the server
This is the shape every Kubernetes-hosted Go server ends up with. Shutdown stops accepting new connections and waits for handlers already running to return. Reach for it the moment your service does anything a client would mind losing halfway through. Run it and curl http://localhost:8080/healthz, then press Ctrl-C to see the drain.
package main
import (
"context"
"errors"
"log/slog"
"net/http"
"os"
"os/signal"
"syscall"
"time"
)
func main() {
// Shorthand: NotifyContext cancels ctx the first time one of these signals arrives.
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer stop()
mux := http.NewServeMux()
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("ok"))
})
srv := &http.Server{Addr: ":8080", Handler: mux}
go func() {
// ListenAndServe returns ErrServerClosed after a clean Shutdown. That is not a failure.
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
slog.Error("serve failed", "err", err)
os.Exit(1)
}
}()
slog.Info("listening", "addr", srv.Addr)
<-ctx.Done() // SIGTERM arrived
slog.Info("draining")
// Must be shorter than terminationGracePeriodSeconds, or SIGKILL interrupts the drain.
drain, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
if err := srv.Shutdown(drain); err != nil {
slog.Error("shutdown", "err", err)
}
slog.Info("stopped")
}Required config that fails at startup, not at midnight
A pod that exits immediately with a clear message is far easier to diagnose than one that starts fine and then panics on the first request that needs the missing secret. Kubernetes will report CrashLoopBackOff and the reason sits in the pod logs.
package main
import (
"fmt"
"os"
"strings"
)
// mustEnv collects every missing variable instead of stopping at the first one,
// so one restart tells the operator everything that is wrong.
type loader struct {
missing []string
}
func (l *loader) require(key string) string {
// LookupEnv also reports whether the variable was set at all,
// which os.Getenv cannot tell you apart from an empty value.
v, ok := os.LookupEnv(key)
if !ok || v == "" {
l.missing = append(l.missing, key)
}
return v
}
func main() {
var l loader
dbURL := l.require("DATABASE_URL")
jwtKey := l.require("JWT_SIGNING_KEY")
if len(l.missing) > 0 {
fmt.Fprintf(os.Stderr, "missing config: %s\n", strings.Join(l.missing, ", "))
os.Exit(1)
}
_, _ = dbURL, jwtKey
fmt.Println("config ok")
}A readiness flag that turns off before the drain
Kubernetes removes a pod from its load balancer only after readiness fails, and that removal is not instant. Flipping readiness to false and pausing for a second before draining stops the last few requests being routed to a server that is already closing.
package main
import (
"net/http"
"sync/atomic"
)
// ready starts false, flips true once dependencies are reachable, and flips
// back to false the instant SIGTERM arrives.
var ready atomic.Bool
func main() {
mux := http.NewServeMux()
// Liveness: is the process alive at all? Keep it trivial.
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("ok"))
})
// Readiness: should traffic be sent here right now?
mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, r *http.Request) {
if !ready.Load() {
http.Error(w, "not ready", http.StatusServiceUnavailable)
return
}
w.Write([]byte("ready"))
})
ready.Store(true)
http.ListenAndServe(":8080", mux)
// On SIGTERM, call ready.Store(false) first, pause a second, then Shutdown.
}
Discussion