Deploying Without an Outage

Rolling, blue-green and canary bound who meets a bad version; feature flags and compatible schemas are what make rollback cheap.

Syntaxkubectl rollout undo deployment/<name> [--to-revision=<n>]

Most outages arrive with a deployment, because that is when the system changes. So the job of a deployment strategy is not speed. It is to bound how many users meet a bad version, and to make undoing it a routine command instead of an improvisation.

Three mechanisms

Strategy	How it works	What it costs
Rolling	Replace instances in batches; Kubernetes defaults to `maxSurge` 25% and `maxUnavailable` 25%	Both versions serve at once, so every change must be compatible in both directions
Blue-green	Run a second full environment, flip the router, flip back to revert	Double capacity, and the flip does not duplicate shared state such as the database
Canary	Send a small share of real traffic to the new version and compare it against the old	Traffic shaping plus an automated comparison, or it is only a slow rolling deploy

Feature flags cut the last tie: deploy the code dark, switch behaviour by config, and the release becomes reversible without a rebuild.

Rollback as a decision

Write the abort criteria before the rollout — error ratio, p99, saturation — and how long you will watch. Keep the previous version ready (revisionHistoryLimit defaults to 10) so kubectl rollout undo deployment/api is one step. Set minReadySeconds above its default of 0 so a pod that crashes after ten seconds still counts as a failure, and leave progressDeadlineSeconds (default 600) to mark a stuck rollout failed rather than pending.

What trips people up

Rollback only reverts code. A dropped column, a queue message in a new format, or a cache entry the old version cannot parse all survive the rollback and keep the outage running.

Example

// Percentage rollout with a stable assignment: the same user must land in the
// same group on every request, or the feature flickers mid-session.
import { createHash } from 'node:crypto';

// Flags are data loaded at runtime, never constants. Changing one must not
// require a deploy - that separation is the entire point of a flag.
const flags = {
  'checkout.new-tax-engine': {
    enabled: true,
    percentage: 5,
    killSwitch: false,
    allowSegments: ['staff'],
  },
};

function bucketOf(flagKey, subjectId) {
  // Hash the flag key TOGETHER with the subject id. Hash the id alone and
  // every 5% flag selects the same unlucky 5% of users, so one cohort absorbs
  // every experiment you ever run.
  const digest = createHash('sha256').update(`${flagKey}:${subjectId}`).digest();
  return digest.readUInt32BE(0) % 100; // 0-99, uniform and stable
}

export function isEnabled(flagKey, user) {
  const flag = flags[flagKey];
  if (!flag || !flag.enabled || flag.killSwitch) return false; // kill switch wins
  if (flag.allowSegments?.includes(user.segment)) return true; // staff go first
  return bucketOf(flagKey, user.id) < flag.percentage;
}

// Call sites keep BOTH paths until the flag is deleted. That is what makes a
// rollback a config change rather than a redeploy - and why a flag left in
// place for a year is a maintenance cost, not a safety net.
export function taxFor(order, user) {
  return isEnabled('checkout.new-tax-engine', user)
    ? newTaxEngine.quote(order)
    : legacyTaxEngine.quote(order);
}

When to use it

Shipping a rewrite of the pricing path with no maintenance window.
A deploy has doubled the error rate and the on-call needs it reverted in under a minute.
Releasing a schema change that both the old and the new version must tolerate while they run side by side.
Exposing a feature to staff, then 5% of users, then everyone, with one switch and no rebuild.

More examples

Rolling update: the shorthand and what it expands to

Both specs are the same object once the API server fills in defaults, so you need to read the short form as the long one when reviewing someone else's manifest. Write the long form whenever capacity matters: `maxUnavailable: 0` keeps full serving capacity through the rollout, and a non-zero `minReadySeconds` is what stops a crash-looping image from being declared a success.

# SHORTHAND - naming the type only. This is identical to the spec below: a
# RollingUpdate with no rollingUpdate block takes maxSurge 25% and
# maxUnavailable 25%.
spec:
  strategy:
    type: RollingUpdate

---
# LONGHAND - the same defaults written out, which is where you change them.
spec:
  replicas: 12
  minReadySeconds: 30          # default 0: a pod counts as available the
                               # instant it is Ready, before it can crash
  progressDeadlineSeconds: 600 # default: after this the rollout is Failed
  revisionHistoryLimit: 10     # default: how many revisions you can undo to
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%            # extra pods above replicas; % rounds UP
      maxUnavailable: 0        # never dip below replicas; % rounds DOWN.
                               # 0 needs spare cluster capacity, and both
                               # fields may not be 0 at the same time.

A canary with explicit pause points

Shows a canary as a sequence of decisions rather than one switch: each `setWeight` raises exposure and each `pause` buys observation time, with the final open-ended pause putting a person in front of the 100% step. Use it for changes whose failure mode is gradual — a slow query, a memory leak, a pricing error — where a thirty-second smoke test would pass.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 20
  strategy:
    canary:
      maxSurge: "25%"
      maxUnavailable: 0
      steps:
        - setWeight: 5
        - pause: { duration: 10m }   # long enough to collect a real sample at
                                     # 5% of traffic, not just a smoke test
        - setWeight: 25
        - pause: { duration: 30m }
        - setWeight: 50
        - pause: {}                  # no duration: waits indefinitely for a
                                     # human to run
                                     #   kubectl argo rollouts promote checkout
# The weights are a blast-radius schedule. 5% for ten minutes means a bad
# version reaches ~0.8% of the hour's requests before anyone has to react.

Deciding the verdict from metrics, not from nerve

Written before the rollout, this turns the abort criteria into code the pipeline can act on, which is the difference between a rollback that is a decision and one that is an emergency. The sample-size guard matters most: at 5% weight a short window produces too few requests for a ratio to mean anything, and teams routinely promote on that noise.

// Compare the canary against the CURRENT stable version, never against
// yesterday's numbers: traffic mix and payload sizes change by the hour.
const WINDOW = '10m';
const MIN_SAMPLE = 1000; // an error ratio over 20 requests is noise

const errorRatio = (version) => `
  sum(rate(http_requests_total{job="checkout",version="${version}",status=~"5.."}[${WINDOW}]))
  / sum(rate(http_requests_total{job="checkout",version="${version}"}[${WINDOW}]))`;

const p99 = (version) => `
  histogram_quantile(0.99, sum by (le) (
    rate(http_request_duration_seconds_bucket{job="checkout",version="${version}"}[${WINDOW}])))`;

const sampleCount = (version) =>
  `sum(increase(http_requests_total{job="checkout",version="${version}"}[${WINDOW}]))`;

export async function canaryVerdict(promql) {
  const [n, errC, errS, p99C, p99S] = await Promise.all([
    promql(sampleCount('canary')),
    promql(errorRatio('canary')), promql(errorRatio('stable')),
    promql(p99('canary')),        promql(p99('stable')),
  ]);

  if (n < MIN_SAMPLE) return { verdict: 'wait', reason: `only ${n} requests` };
  // Absolute floor plus a relative margin: a stable version at 0.1% errors
  // should not license a canary at 0.2%.
  if (errC > 0.005 && errC > errS * 1.5) return { verdict: 'abort', reason: 'errors' };
  if (p99C > p99S * 1.3) return { verdict: 'abort', reason: 'latency' };
  return { verdict: 'promote' };
}

Discussion

Be the first to comment on this lesson.