Deploying It
Manifests with probes, config, secrets and limits, a rollout that fails safely, and migrations applied before the new pods arrive.
kubectl set image deployment/shortly-api api=registry.example.com/shortly:1.4.0 && kubectl rollout status deployment/shortly-api --timeout=180sDeploying means handing the cluster a description of what should be running and letting it make that true. Four objects do it: a Deployment that says which image and how many copies, a Service that gives those copies one stable address, an Ingress that routes a public hostname to the Service, and a ConfigMap and Secret holding the values the image deliberately does not contain.
Two probes, two questions
A probe is an HTTP request the cluster makes to a pod. They answer different questions and must not share an endpoint. /healthz/live answers "is this process still working", and checks nothing external, because a failed liveness probe restarts the pod. /healthz/ready answers "may this pod receive traffic", and does check the database, because a failed readiness probe only takes the pod out of the Service. A startup probe covers the slow first few seconds so the liveness probe can stay impatient afterwards.
The rollout
With maxUnavailable: 0 the cluster starts a new pod, waits for it to report ready, then removes an old one. A new image that never becomes ready therefore serves no traffic at all. kubectl rollout status returns non-zero when that happens, which is how a pipeline notices.
Migrations, separately
Run the schema change as its own step before the new image rolls out, never from the application's startup path where every replica would race. During the rollout old and new pods run together, so each change must be additive: add a nullable column, deploy code that writes it, backfill, and only then make it required.
Example
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using Npgsql;
WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
builder.Services.AddHealthChecks()
.AddCheck<DatabaseCheck>("database", tags: ["ready"], timeout: TimeSpan.FromSeconds(2));
WebApplication app = builder.Build();
// Predicate: _ => false runs no checks at all. Liveness says only that the
// process is answering, so a database outage cannot restart every pod.
app.MapHealthChecks("/healthz/live", new HealthCheckOptions { Predicate = _ => false });
// Readiness runs the checks tagged "ready": 200 when healthy, 503 when not.
app.MapHealthChecks("/healthz/ready", new HealthCheckOptions
{
Predicate = registration => registration.Tags.Contains("ready")
});
app.Run();
// The primary constructor (IConfiguration configuration) is shorthand for a
// constructor that stores its argument in a private field.
sealed class DatabaseCheck(IConfiguration configuration) : IHealthCheck
{
public async Task<HealthCheckResult> CheckHealthAsync(
HealthCheckContext context,
CancellationToken cancellationToken = default)
{
await using NpgsqlConnection connection =
new(configuration.GetConnectionString("Shortly"));
await connection.OpenAsync(cancellationToken);
return HealthCheckResult.Healthy("connected");
}
}When to use it
- Rolling out a new image without dropping a request.
- Adding a column to a live table while the previous version is still serving.
- Making a bad release stop itself instead of quietly serving errors.
More examples
The Deployment and its Service
This is the manifest to start from for any .NET service. The memory limit is there because exceeding it kills one pod, which is what you want; a CPU limit is left off because it throttles the process instead of killing it, and a throttled thread pool shows up as mysterious latency rather than an obvious failure. envFrom pulls every key in the ConfigMap and Secret in as environment variables, which is why the names use double underscores.
apiVersion: apps/v1
kind: Deployment
metadata:
name: shortly-api
spec:
replicas: 3
strategy:
rollingUpdate:
maxUnavailable: 0 # never fewer than 3 ready pods during a rollout
maxSurge: 1
selector:
matchLabels: { app: shortly-api }
template:
metadata:
labels: { app: shortly-api }
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1654 # APP_UID from the .NET base image
containers:
- name: api
image: registry.example.com/shortly:1.4.0 # a version, never latest
ports:
- containerPort: 8080
envFrom:
- configMapRef: { name: shortly-config }
- secretRef: { name: shortly-secrets }
startupProbe: # up to 60s to start, then liveness takes over
httpGet: { path: /healthz/live, port: 8080 }
periodSeconds: 2
failureThreshold: 30
livenessProbe:
httpGet: { path: /healthz/live, port: 8080 }
periodSeconds: 10
readinessProbe:
httpGet: { path: /healthz/ready, port: 8080 }
periodSeconds: 5
resources:
requests: { cpu: 100m, memory: 192Mi }
limits: { memory: 384Mi } # no cpu limit: see the explanation
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
---
apiVersion: v1
kind: Service
metadata:
name: shortly-api
spec:
selector: { app: shortly-api }
ports:
- port: 80
targetPort: 8080Migrations as their own job
Use this instead of calling Database.Migrate() when the app starts. Three replicas starting at once would each try to take the migration lock, and the one that waits may fail its startup probe while it does. A Job also gives the pipeline something to gate on: if the schema change fails, the new image is never rolled out.
# A Job runs once to completion, unlike a Deployment which runs for ever.
# One Job per release, named after the version, so re-applying is harmless.
apiVersion: batch/v1
kind: Job
metadata:
name: shortly-migrate-1-4-0
spec:
backoffLimit: 2 # two retries, then stop and fail the deploy
template:
spec:
restartPolicy: Never
containers:
- name: migrate
# An image containing only the EF Core migration bundle: no SDK,
# no application code, nothing that can serve a request.
image: registry.example.com/shortly-migrate:1.4.0
args: ["--connection", "$(ConnectionStrings__Shortly)"]
env:
- name: ConnectionStrings__Shortly
valueFrom:
secretKeyRef:
name: shortly-secrets
key: ConnectionStrings__ShortlyThe release, start to finish
This is the whole deploy, in the order that makes a failure survivable. The two commands worth memorising are rollout status, because a deploy that does not check is not a deploy, and describe pod, because a pod stuck in CrashLoopBackOff or ImagePullBackOff explains itself in the Events section at the bottom rather than in the application log.
# In CI: build the bundle. It is a single executable that needs no SDK,
# no EF Core tools and no source code at the time it runs.
dotnet ef migrations bundle --project Shortly.Api --self-contained --target-runtime linux-x64 --output artifacts/efbundle
# Schema first, and stop here if it fails.
kubectl apply -f k8s/migrate-job.yaml
kubectl wait --for=condition=complete job/shortly-migrate-1-4-0 --timeout=300s
# Then the image. rollout status exits non-zero if the new pods never
# become ready, so the pipeline can undo the change by itself.
kubectl set image deployment/shortly-api api=registry.example.com/shortly:1.4.0
kubectl rollout status deployment/shortly-api --timeout=180s || kubectl rollout undo deployment/shortly-api
# What to read when it did not work.
kubectl get pods -l app=shortly-api
kubectl describe pod -l app=shortly-api | grep -A5 Events
kubectl logs -l app=shortly-api --tail=50
Discussion