Rollouts and Migrations

Old and new pods share one database during a rolling update, so schema changes ship as expand, backfill, switch, contract.

Syntaxtypeorm migration:run -d dist/data-source.js [-t|--transaction all|each|none] [-f|--fake]

A rolling update replaces pods a few at a time. For a minute or two the old image and the new image are both serving, against one database. Every schema change therefore has to be valid for both. That single fact is the whole subject.

Expand and contract

Split a breaking change into deploys that are each backwards compatible.

  1. Expand. Add the new nullable column or table. Old code ignores it.
  2. Write both. Ship code that writes the old and new shape and still reads the old one.
  3. Backfill. Copy existing rows in batches, outside the request path.
  4. Switch reads. Ship code that reads the new column, then add NOT NULL.
  5. Contract. Once no running pod mentions it, drop the old column.

Renaming a column in one step is the opposite of this: the moment the migration lands, the old pods query a column that is gone.

Where migrations run

Not inside the application. TypeORM's migrationsRun: true executes migrations during bootstrap, so three replicas start three migration runs; one takes the lock and the others wait or fail their readiness probe. Run them once before the rollout, from a Kubernetes Job or a release step, and keep synchronize: false everywhere but your laptop, because synchronize infers DDL from your entities and will drop a column it cannot account for.

Example

Example · typescript
// src/migrations/1738000000000-AddUserEmailNormalized.ts
import type { MigrationInterface, QueryRunner } from 'typeorm';

// Deploy 1 of expand/contract: additive only. Pods still running the previous
// image never mention this column, so applying it mid-rollout is safe.
export class AddUserEmailNormalized implements MigrationInterface {
  // Optional. The class name is used as the migration name when this is
  // omitted, and the name is what gets recorded in the migrations table.
  name = 'AddUserEmailNormalized1738000000000';

  public async up(queryRunner: QueryRunner): Promise<void> {
    // Nullable and with no DEFAULT: Postgres writes a catalogue entry only, so
    // the table is not rewritten and concurrent readers are not blocked.
    await queryRunner.query(
      `ALTER TABLE "users" ADD COLUMN "email_normalized" varchar(320)`,
    );
  }

  // down() has to undo exactly this much. Dropping a column the old code still
  // reads is why a rollback hurts more than a roll-forward; prefer shipping a
  // corrective migration over reverting one.
  public async down(queryRunner: QueryRunner): Promise<void> {
    await queryRunner.query(
      `ALTER TABLE "users" DROP COLUMN "email_normalized"`,
    );
  }
}

When to use it

  • Rename a column across three deploys without a maintenance window.
  • Stop three replicas racing to run the same migration at startup.
  • Add a unique index to a 50-million-row table without blocking writes.
  • Set up a release pipeline that fails before the rollout when a migration is broken.

More examples

The DataSource the CLI and the app share

The CLI needs a file that exports a DataSource instance, and pointing both the CLI and TypeOrmModule at the same options is what stops the two drifting apart. The glob paths reference dist, because by the time this file runs in production it is itself compiled JavaScript. migrationsTransactionMode defaults to wrapping the entire run in one transaction; 'each' is the setting you want as soon as one migration cannot live inside a transaction.

Example · typescript
// src/data-source.ts
// Compiled to dist/data-source.js, which is the file the TypeORM CLI loads.
import { DataSource } from 'typeorm';

export default new DataSource({
  type: 'postgres',
  url: process.env.DATABASE_URL,
  entities: ['dist/**/*.entity.js'],
  migrations: ['dist/migrations/*.js'],
  migrationsTableName: 'typeorm_migrations',
  // One transaction per migration rather than one around all of them, so a
  // single migration can opt out with `transaction = false`.
  migrationsTransactionMode: 'each',
  // Never run migrations from the application process: every replica would
  // start its own run during bootstrap.
  migrationsRun: false,
  // Never infer DDL from entities outside local development.
  synchronize: false,
});

A migration that cannot be wrapped in a transaction

The ordinary CREATE INDEX takes a lock that blocks writes to the table for as long as the build takes, which on a large table means a visible outage. CONCURRENTLY avoids that at the cost of being slower and not transactional. Reach for this pattern for index creation on any table big enough to matter, and remember that a CONCURRENTLY build which fails leaves an invalid index behind that you have to drop before retrying.

Example · typescript
// src/migrations/1738000600000-IndexUserEmailNormalized.ts
import type { MigrationInterface, QueryRunner } from 'typeorm';

export class IndexUserEmailNormalized implements MigrationInterface {
  // Required here: TypeORM takes the recorded name from this property, or from
  // the class name when it is omitted, and either way the last 13 characters
  // have to parse as a timestamp. Without it the run aborts with
  // "IndexUserEmailNormalized migration name is wrong".
  name = 'IndexUserEmailNormalized1738000600000';

  // CREATE INDEX CONCURRENTLY cannot run inside a transaction block.
  // This flag is only honoured when migrationsTransactionMode is 'each' or
  // 'none'. Under the default 'all', TypeORM refuses the whole run up front
  // with ForbiddenTransactionModeOverrideError, so Postgres never sees it.
  transaction = false;

  public async up(queryRunner: QueryRunner): Promise<void> {
    await queryRunner.query(
      `CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS "uq_users_email_normalized"
       ON "users" ("email_normalized")`,
    );
  }

  public async down(queryRunner: QueryRunner): Promise<void> {
    await queryRunner.query(
      `DROP INDEX CONCURRENTLY IF EXISTS "uq_users_email_normalized"`,
    );
  }
}

Running the migration before the rollout

One Job, one migration run, one chance to fail loudly before any traffic sees the new image. The Job uses the same image as the Deployment, so the migrations it runs are exactly the ones the new code expects. Naming the Job after the version keeps the history readable and stops a re-apply silently reusing a completed Job; if the Job fails, the Deployment is untouched and still serving the old image.

Example · yaml
# k8s/migrate-job.yaml
# Apply this, wait for it, and only then update the Deployment's image.
apiVersion: batch/v1
kind: Job
metadata:
  name: api-migrate-1-4-0
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: myorg/api:1.4.0
          # -d is shorthand for --dataSource; both name the compiled file that
          # exports the DataSource. The other two flags migration:run accepts
          # are -t/--transaction and -f/--fake.
          command:
            - npx
            - typeorm
            - migration:run
            - -d
            - dist/data-source.js
          envFrom:
            - secretRef:
                name: api-secrets
---
# The release sequence, in order:
#   kubectl apply -f k8s/migrate-job.yaml
#   kubectl wait --for=condition=complete job/api-migrate-1-4-0 --timeout=10m
#   kubectl set image deploy/api api=myorg/api:1.4.0
#   kubectl rollout status deploy/api

Related

nest migrationsnest k8snest transactionsnest buildnest entities relations

Discussion

  • Be the first to comment on this lesson.