Deploying from a Pipeline
Promote one image digest from staging to production behind an approval, migrate the database first, verify the rollout and roll back safely.
kubectl -n orders set image deployment/orders-api api=$IMAGE
kubectl -n orders rollout status deployment/orders-api --timeout=10m
kubectl -n orders rollout undo deployment/orders-apiA deploy pipeline takes an image that CI already built and tested and moves it through environments. The same digest goes to staging and then to production; nothing is rebuilt along the way, so what was tested is what ships.
Environments and approvals
A GitHub Actions job with environment: production waits for the environment's protection rules: required reviewers, a wait timer, and a rule that only main may deploy there. The environment's secrets and variables become readable only after approval, so a pull request cannot read production credentials. Add concurrency per environment with cancel-in-progress: false: two deploys to one cluster must queue, and cancelling one halfway leaves a migration or rollout in an unknown state. For cloud access use OIDC (permissions: id-token: write) and a role scoped to that environment instead of a stored kubeconfig or access key.
Migrations go first, and must be backward compatible
Apply schema changes before the new pods start, as a separate step: an EF Core migration bundle run as a Kubernetes Job. Database Migrations in Deployments covers how the bundle, the migrator image and the Job are built; the pipeline's part is to apply that Job, wait until it completes or fails, and stop before the rollout if it fails. Never migrate on application start: ten replicas race for the same lock, and the app needs DDL rights it should not have. During the rollout old and new pods serve traffic together, so every migration must work with the previous release too: add a nullable column now, backfill, make it required in a later release (expand and contract). A migration that renames a column breaks the old pods mid-rollout and makes rollback impossible.
Rollout and verification
Update the image (kubectl set image, kubectl apply or helm upgrade --install), then wait for it: kubectl rollout status --timeout=10m or Helm's --wait. Without the wait the job is green the moment the API server accepts the change, even if every new pod then crash-loops. Rollout status only proves pods passed readiness. A smoke check then asks the running service through its public route whether it is the expected commit and whether a critical read works.
Rollback
| Option | Speed | Watch out for |
|---|---|---|
kubectl rollout undo | Seconds | Cluster now differs from the manifests in git |
helm rollback <release> | Seconds | Restores values too, including any old config |
| Redeploy the previous digest through the pipeline | Minutes | Needs the previous digest recorded per environment |
| Roll forward with a fix | Slowest | The right choice once data written by the new version depends on it |
Automate the first one for a failed rollout or smoke check, and use the pipeline for deliberate rollbacks so git stays the source of truth. None of these undo a migration: rollback is safe only because the schema change was backward compatible. With GitOps (Argo CD, Flux) the pipeline commits the new digest to a config repository and the controller applies it; rollback is a revert commit.
Production failure modes
- A rollback step with
if: failure()that also runs when the migration failed, undoing the previous good release. - A deploy that pulls
:latest, so the rollback target is the same broken image. - A smoke check that hits a single pod port-forward instead of the ingress, and misses a routing or TLS mistake.
Example
When to use it
- Promote ghcr.io/acme/orders-api@sha256 digest from staging to production after a release manager approves the production environment.
- Add a nullable discount_code column with a migration Job before rolling out the API version that writes it, so old pods keep working.
- Undo a rollout automatically when /version on the public host still reports the old commit after kubectl rollout status has passed.
More examples
Deploy workflow with approval, migration, smoke check and rollback
The production environment carries a required-reviewer rule, so the job pauses before its first step until someone approves, and only then can it read <code>DEPLOY_ROLE_ARN</code>. Inputs are validated as a full SHA and a digest, the commit must be on <code>main</code>, and the digest must be the one tagged <code>sha-</code> plus that commit, so a typed value can neither inject shell code nor promote an unreviewed build. The rollback returns to the revision recorded before the change, and only if the revision moved: when the migration or <code>set image</code> fails, the cluster still runs the previous good release and a bare <code>rollout undo</code> would deploy the one before it. Rolling back the image is safe only because the migration was additive.
name: deploy
on:
workflow_dispatch:
inputs:
environment:
type: choice
options: [staging, production]
required: true
digest:
description: "Image digest from the image workflow (sha256:...)"
required: true
commit:
description: "Full commit SHA the image was built from"
required: true
permissions:
contents: read
packages: read
id-token: write
concurrency:
group: deploy-orders-api-${{ inputs.environment }}
cancel-in-progress: false
jobs:
deploy:
runs-on: ubuntu-24.04
timeout-minutes: 30
environment:
name: ${{ inputs.environment }}
url: https://${{ vars.API_HOST }}
env:
# Inputs reach scripts only as environment variables, never as ${{ }} inside run:.
COMMIT: ${{ inputs.commit }}
DIGEST: ${{ inputs.digest }}
steps:
# Deploy tooling comes from main, not from whatever commit the dispatcher typed.
- uses: actions/checkout@v5
with:
ref: main
fetch-depth: 0
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Validate inputs
run: |
[[ "$COMMIT" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::commit must be a full SHA"; exit 1; }
[[ "$DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] || { echo "::error::digest must be sha256:<64 hex>"; exit 1; }
git merge-base --is-ancestor "$COMMIT" origin/main || { echo "::error::$COMMIT is not on main"; exit 1; }
built=$(docker buildx imagetools inspect "ghcr.io/acme/orders-api:sha-$COMMIT" --format '{{json .Manifest}}' | jq -r .digest)
[[ "$built" == "$DIGEST" ]] || { echo "::error::$DIGEST was not built from $COMMIT"; exit 1; }
migrator=$(docker buildx imagetools inspect "ghcr.io/acme/orders-migrator:sha-$COMMIT" --format '{{json .Manifest}}' | jq -r .digest)
echo "IMAGE=ghcr.io/acme/orders-api@$DIGEST" >> "$GITHUB_ENV"
echo "MIGRATOR_IMAGE=ghcr.io/acme/orders-migrator@$migrator" >> "$GITHUB_ENV"
- uses: actions/setup-dotnet@v5
with:
global-json-file: global.json
- uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
with:
role-to-assume: ${{ vars.DEPLOY_ROLE_ARN }}
aws-region: eu-west-1
- run: aws eks update-kubeconfig --name ${{ vars.CLUSTER_NAME }}
- id: before
name: Record the running revision
run: |
rev=$(kubectl -n orders get deployment/orders-api -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')
echo "revision=$rev" >> "$GITHUB_OUTPUT"
- name: Migrate (expand only)
run: |
kubectl -n orders delete job orders-migrate --ignore-not-found
# deploy/migrate-job.yaml is the bundle Job from Database Migrations in Deployments,
# with image: ${MIGRATOR_IMAGE} so it runs the migrator built from the same commit.
envsubst < deploy/migrate-job.yaml | kubectl -n orders apply -f -
# kubectl wait --for=condition=complete would sit out its whole timeout on a failed Job.
for attempt in $(seq 1 130); do
state=$(kubectl -n orders get job/orders-migrate -o jsonpath='{.status.conditions[?(@.status=="True")].type}')
case "$state" in
*Complete*) exit 0 ;;
*Failed*) kubectl -n orders logs job/orders-migrate --tail=50; exit 1 ;;
esac
sleep 5
done
echo "::error::migration did not finish in time"; exit 1
- name: Roll out
run: |
kubectl -n orders set image deployment/orders-api api="$IMAGE"
kubectl -n orders annotate deployment/orders-api kubernetes.io/change-cause="commit $COMMIT" --overwrite
kubectl -n orders rollout status deployment/orders-api --timeout=10m
- name: Smoke check
env:
API_HOST: ${{ vars.API_HOST }}
run: dotnet run tools/SmokeCheck.cs -- "https://$API_HOST" "$COMMIT"
- name: Roll back
if: failure() && steps.before.outputs.revision != ''
env:
PREVIOUS: ${{ steps.before.outputs.revision }}
run: |
now=$(kubectl -n orders get deployment/orders-api -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')
if [[ "$now" == "$PREVIOUS" ]]; then echo "Nothing was rolled out; revision $PREVIOUS stays."; exit 0; fi
kubectl -n orders rollout undo deployment/orders-api --to-revision="$PREVIOUS"
kubectl -n orders rollout status deployment/orders-api --timeout=10mStaging, then production, with one digest
For this, <code>image.yml</code> and <code>deploy.yml</code> also declare <code>on: workflow_call</code>, with string inputs (a <code>choice</code> input exists only for <code>workflow_dispatch</code>) and <code>digest</code> as an output of the image workflow. Both deploy jobs receive the digest of the single build, so production gets the bytes staging tested. The production job still pauses for the environment's required reviewers, and a failed staging migration or smoke check stops the run before anyone is asked to approve. A called workflow cannot raise the permissions of its caller, which is why <code>packages: write</code> and <code>id-token: write</code> are granted here.
name: release
on:
push:
tags: ["v*.*.*"]
permissions:
contents: read
packages: write
id-token: write
jobs:
image:
uses: ./.github/workflows/image.yml
staging:
needs: image
uses: ./.github/workflows/deploy.yml
with:
environment: staging
digest: ${{ needs.image.outputs.digest }}
commit: ${{ github.sha }}
secrets: inherit
production:
# Runs only after staging deployed, migrated and passed its smoke check.
needs: [image, staging]
uses: ./.github/workflows/deploy.yml
with:
environment: production
digest: ${{ needs.image.outputs.digest }}
commit: ${{ github.sha }}
secrets: inheritGitOps: the pipeline commits the digest instead of applying it
With Argo CD or Flux the cluster pulls its state from a config repository, so the pipeline never holds cluster credentials: it writes the new digest into the production overlay and pushes. The controller applies it, runs the migration Job first if it is a pre-sync hook, and reports health. Rollback is <code>git revert</code> of that commit. The job ends when the push succeeds, so either wait for the controller (for example <code>argocd app wait --health</code>) or let the controller's own notifications report a failed sync. The approval rule stays on the <code>production</code> environment, which guards the token that can push.
name: promote-gitops
on:
workflow_dispatch:
inputs:
digest:
description: "Image digest (sha256:...)"
required: true
commit:
description: "Full commit SHA the image was built from"
required: true
permissions:
contents: read
concurrency:
group: promote-orders-api-production
cancel-in-progress: false
jobs:
promote:
runs-on: ubuntu-24.04
timeout-minutes: 10
environment: production
env:
DIGEST: ${{ inputs.digest }}
COMMIT: ${{ inputs.commit }}
steps:
- uses: actions/checkout@v5
with:
repository: acme/orders-config
token: ${{ secrets.CONFIG_REPO_TOKEN }}
- name: Pin the digest in the production overlay
working-directory: overlays/production
run: |
[[ "$DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] || { echo "::error::bad digest"; exit 1; }
[[ "$COMMIT" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::bad commit"; exit 1; }
kustomize edit set image "ghcr.io/acme/orders-api=ghcr.io/acme/orders-api@$DIGEST"
git config user.name "orders-deploy"
git config user.email "[email protected]"
git commit -am "orders-api production: $COMMIT"
git push
Discussion