Rolling Out and Rolling Back
How a rolling update replaces pods one group at a time, how to roll back, and how to migrate a schema two versions can share.
kubectl set image deployment/orders api=registry.example.com/orders:v2 && kubectl rollout status deployment/ordersChange the image in a Deployment and Kubernetes does not restart anything. It creates a second ReplicaSet for the new version and moves pods from the old one to the new one a few at a time. That is a rolling update, and it is why the service stays up.
Surge and unavailability
maxSurge is how many pods may exist above the desired count, and maxUnavailable is how many of the desired count may be missing. Both default to 25%. Each new pod has to pass its readiness probe before the next old one is removed, so a readiness endpoint that lies turns a rolling update into an outage. minReadySeconds makes a pod prove itself for a while before it counts as available. If nothing progresses for progressDeadlineSeconds, 600 by default, the rollout is marked failed. It is not undone for you.
Rolling back
kubectl rollout undo deployment/orders returns to the previous revision by scaling the old ReplicaSet back up, which is fast because the old pod template was kept. revisionHistoryLimit, 10 by default, decides how many are kept.
Two versions at once
During every rollout both versions run against one database, so the schema must satisfy both. Work in three steps. Expand: add the new column or table, nullable, and ship code that writes both the old and the new shape while still reading the old. Migrate: backfill, then ship code that reads the new shape. Contract: only once no running version needs it, drop the old column. Run the migration itself as a separate Job, not at startup in every replica.
Example
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;
var builder = WebApplication.CreateBuilder(args);
// False until warm-up finishes. Volatile, because the check below runs on
// request threads while this is written on the startup thread.
bool warm = false;
builder.Services.AddHealthChecks()
.AddCheck("warmup", () => Volatile.Read(ref warm)
? HealthCheckResult.Healthy("warm")
: HealthCheckResult.Unhealthy("still warming up"), tags: ["ready"]);
var app = builder.Build();
app.MapGet("/orders/1", () => new { id = 1, total = 42 });
// The rollout only replaces the next old pod once this returns 200.
app.MapHealthChecks("/readyz", new HealthCheckOptions
{
Predicate = registration => registration.Tags.Contains("ready")
});
await app.StartAsync(); // listening, but /readyz is 503
await Task.Delay(TimeSpan.FromSeconds(2)); // stands in for priming a cache
Volatile.Write(ref warm, true); // /readyz now returns 200
await app.WaitForShutdownAsync();When to use it
- Ship a new image to production without dropping a request.
- Get back to the previous version in under a minute when errors spike.
- Add a non-nullable column to a busy table without breaking the version that is still running.
More examples
Controlling the pace
maxUnavailable: 0 with maxSurge: 1 is the safe, slow setting: capacity never drops, and one extra pod is added before an old one goes. The grace period must be longer than your longest request plus the host's shutdown timeout, or in-flight work is cut off when the container is killed.
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders
spec:
replicas: 4
minReadySeconds: 10 # stay healthy 10s before counting as available
progressDeadlineSeconds: 300 # give up after 5 minutes of no progress
revisionHistoryLimit: 5
strategy:
type: RollingUpdate # the default; the other is Recreate
rollingUpdate:
maxSurge: 1 # at most 5 pods during the rollout
maxUnavailable: 0 # never fewer than 4 serving pods
selector:
matchLabels:
app: orders
template:
metadata:
labels:
app: orders
spec:
terminationGracePeriodSeconds: 45
containers:
- name: api
image: registry.example.com/orders:v2Migrations as a Job
The image holds a migration bundle, built with dotnet ef migrations bundle, so no SDK is needed at runtime. Run the Job to completion, then roll out the application image. Calling Database.Migrate at startup instead means every replica tries to migrate at once during a rollout, and a long migration fails the startup probe.
apiVersion: batch/v1
kind: Job
metadata:
name: orders-migrate-v2
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: registry.example.com/orders-migrations:v2
args: ["--connection", "$(ConnectionStrings__Default)"]
env:
- name: ConnectionStrings__Default
valueFrom:
secretKeyRef:
name: orders-secrets
key: ConnectionStrings__DefaultCode that both versions can run
Writing both shapes for one release is the price of never taking the service down. The methods are in a static class rather than written at the top level because two of them share a scope and a local function cannot be called from another file. Display shows the matching read rule: prefer the new column, fall back to the old one while rows may still be unmigrated.
namespace Orders.Data;
// Expand phase. LegalName is new and nullable, so the previous version,
// which does not know the column exists, still inserts and updates rows.
public class Customer
{
public int Id { get; set; }
public string Name { get; set; } = "";
public string? LegalName { get; set; }
}
public static class CustomerUpdates
{
// v2 writes both columns: the old one for v1 readers, the new one
// for itself. The backfill fills LegalName for existing rows.
public static void Rename(Customer customer, string newName)
{
customer.Name = newName;
customer.LegalName = newName;
}
// Only after every pod runs v2, and the backfill has finished, does a
// later release stop writing Name and a migration drop the column.
public static string Display(Customer customer) =>
customer.LegalName ?? customer.Name;
}
Discussion