Safe Upgrades, Atomic Installs, and Rollbacks

Make upgrades all-or-nothing with --atomic, wait for real readiness with --wait, and know exactly how rollback behaves before you need it in an incident.

An upgrade that half-applies is worse than one that fails cleanly. The senior move is to make every production upgrade transactional: it either fully succeeds and is healthy, or it automatically returns to the last good state.

The flags that make upgrades safe

--atomic — if any part of the upgrade fails or times out, Helm automatically rolls the whole release back to the previous revision. This implies --wait.
--wait — Helm blocks until the new Pods, PVCs, and Services actually report ready, not just "created." Without it, helm upgrade returns the instant the API accepts the manifests, long before your app is actually up.
--timeout — how long to wait before declaring failure (default 5m). Set it to something realistic for your startup time so --atomic triggers instead of hanging.
--cleanup-on-fail — delete resources a failed upgrade newly created, so a retry starts clean.

Rollback, understood before the incident

helm rollback creates a new forward revision whose contents match an old one — history only grows. But it rolls back the manifests, not the world around them: a database migration a hook already ran, or a PVC that was expanded, does not un-happen. Know your stateful blast radius in advance.

The idempotent workhorse for pipelines: helm upgrade --install --atomic --wait --timeout 5m. It installs on first run, upgrades after, and self-heals on failure — the same command in every environment.

Example

# Production-grade, self-healing deploy -- same command everywhere
helm upgrade --install my-web ./mychart \
  -f values-prod.yaml \
  --namespace web --create-namespace \
  --atomic \            # roll back automatically if anything fails
  --wait \              # block until resources are actually Ready
  --timeout 5m \        # give startup a realistic budget
  --cleanup-on-fail     # remove half-created resources on failure

# If you ever need to step back manually:
helm history my-web                 # find the last good revision
helm rollback my-web 7 --wait       # forward to a new revision matching rev 7

When to use it

A platform team runs helm upgrade --atomic in production so a deployment that fails to become healthy within the timeout window is automatically rolled back, limiting blast radius.
A developer uses --wait to block the CI pipeline until all Pods are ready after an upgrade, ensuring the success signal reflects actual cluster health rather than just manifest acceptance.
An SRE rehearses rollback in staging by running helm history and helm rollback to confirm the process is fast and correct before a high-risk production upgrade.

More examples

Atomic upgrade with rollback

Runs the upgrade atomically: if any resource fails to become ready within 10 minutes, Helm automatically rolls back to the previous revision.

helm upgrade my-app ./myapp \
  -f values-prod.yaml \
  --atomic \
  --timeout 10m \
  --cleanup-on-fail \
  -n production

Wait for real readiness

Waits for all Pods and Jobs to become ready before returning, then prints the new revision number to confirm success.

helm upgrade my-app ./myapp \
  -f values-prod.yaml \
  --wait \
  --wait-for-jobs \
  --timeout 8m \
  -n production

echo "Upgrade revision: $(helm list -n production -f my-app -o json | jq '.[0].revision')"

Emergency rollback workflow

Shows the full emergency rollback workflow: inspect history, roll back to a specific revision, and verify the release status.

# 1. See all revisions
helm history my-app -n production

# 2. Roll back to last known-good revision
helm rollback my-app 5 \
  --wait --timeout 5m \
  -n production

# 3. Confirm cluster state
helm status my-app -n production

Discussion

Be the first to comment on this lesson.