Secrets in CI/CD and Infrastructure

Secrets leak from pipelines, images and repositories far more often than from running applications.

Application code usually handles secrets carefully. The pipeline that builds it, the image it ships in and the repository it lives in are where they actually leak.

Where they end up

Committed to git — permanently, in every clone and fork, whatever you do to history later.
In CI logs — echoed by a debug flag, or printed by a tool that does not know it is a secret.
In Docker image layers — a COPY .env or a build argument persists in the layer even if a later step deletes the file.
In build arguments, which are visible in image metadata.
In Terraform state, which stores values in plaintext.
In a Kubernetes Secret, which is base64, not encryption.

The hierarchy

Best is no secret at all — workload identity and OIDC federation. Then a managed secret store with short-lived credentials. Then injected environment variables. Then an encrypted file with the key held elsewhere. Never in the repository.

CI/CD specifically

Prefer OIDC federation over long-lived cloud keys: the pipeline exchanges its identity token for short-lived credentials, so there is no static key to steal. Restrict which branches and environments can assume which role. And treat a pull request from a fork as untrusted — it must never receive production secrets.

Assume a leak and plan the response

Enable push protection and secret scanning. When a secret is found: revoke first, investigate second. Rewriting git history does not un-leak anything, because clones, forks and caches keep the old value.

Example

# Docker: a secret in ANY layer is in the image, even if deleted later
# ❌
COPY .env /app/.env
RUN npm ci && rm /app/.env         # the file is still in the earlier layer

# ❌ build args are stored in image metadata
ARG NPM_TOKEN
RUN npm ci
# docker history --no-trunc <image> shows it

# ✅ BuildKit secret mounts — never written to a layer
# syntax=docker/dockerfile:1
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci
# docker build --secret id=npmrc,src=$HOME/.npmrc .

# Check what you already shipped
docker history --no-trunc myimage:latest | grep -iE 'secret|token|key|password'

When to use it

A cloud key committed in a Dockerfile build argument is found by scanning image metadata months after the image shipped.
A CI pipeline switches to OIDC federation, removing the long-lived cloud credentials that were stored as repository secrets.
Push protection blocks a private key from being committed, avoiding a permanent presence in the repository history.

More examples

OIDC federation instead of stored cloud keys

The pull_request_target pattern is worth recognising on sight: it is the single most common way CI secrets are stolen from open-source repositories.

# The best CI secret is no CI secret. GitHub Actions can exchange its own
# identity token for short-lived cloud credentials.

# ── The IAM trust policy: which repo, which branch, which environment ─
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
        # Pin to the exact repository AND ref. Without the ref condition, ANY
        # branch — including one an attacker opens a PR from — can assume it.
        "token.actions.githubusercontent.com:sub":
          "repo:acme/api:environment:production"
      }
    }
  }]
}

# ── The workflow ─────────────────────────────────────────────────────
name: deploy
on:
  push: { branches: [main] }

permissions:
  id-token: write        # required to request the OIDC token
  contents: read         # and nothing more

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production        # gated: requires approval, holds the secrets
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-deploy
          aws-region: eu-west-1
          role-duration-seconds: 900        # 15 minutes, not 12 hours
      # No AWS_ACCESS_KEY_ID anywhere. Nothing static to steal.

      - run: aws ecs update-service --cluster prod --service api --force-new-deployment

# ── Forks are untrusted ──────────────────────────────────────────────
# pull_request from a fork gets NO secrets by default — keep it that way.
# NEVER use pull_request_target with a checkout of the PR head: it runs the
# fork's code WITH your secrets, and it has caused real supply-chain breaches.
#
# ❌ on: pull_request_target
#    steps: [{ uses: actions/checkout@v4, with: { ref: '${{ github.event.pull_request.head.sha }}' } }]

# ── Pin actions to a SHA, not a tag ──────────────────────────────────
# A tag can be moved. A SHA cannot.
# ❌ uses: some/action@v1
# ✅ uses: some/action@a1b2c3d4e5f6...   # v1.2.3

Keeping secrets out of images and state

The `docker save | grep` check is a blunt but effective audit — it finds credential-shaped strings anywhere in any layer, including ones you did not know were there.

# ── Docker: BuildKit secret mounts ───────────────────────────────────
# syntax=docker/dockerfile:1
FROM node:22-alpine AS deps
WORKDIR /app
COPY package*.json ./
# The secret is mounted for THIS command only and never written to a layer.
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci --omit=dev --ignore-scripts

FROM node:22-alpine
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
# .dockerignore MUST exclude: .env* .git **/*.pem **/*.key .aws
USER node
CMD ["node", "server.js"]

# docker build --secret id=npmrc,src=$HOME/.npmrc -t api .

# Verify nothing leaked into the image
docker history --no-trunc api | grep -iE 'secret|token|npm_|key|password'
docker run --rm api env | grep -iE 'secret|token|key'
docker save api | tar -xO | grep -aoE '(sk_live|ghp_|AKIA)[A-Za-z0-9]{10,}' | head

# ── Terraform state is PLAINTEXT ─────────────────────────────────────
# Anything you pass as a variable ends up readable in the state file.
terraform {
  backend "s3" {
    bucket  = "tf-state"
    key     = "prod/api.tfstate"
    encrypt = true                       # at rest
    kms_key_id = "arn:aws:kms:..."
    dynamodb_table = "tf-locks"
  }
}

# ✅ Reference secrets rather than passing them in
data "aws_secretsmanager_secret_version" "db" {
  secret_id = "prod/database"
}
resource "aws_ecs_task_definition" "api" {
  container_definitions = jsonencode([{
    # The ARN is in the state; the VALUE is not.
    secrets = [{ name = "DATABASE_URL",
                 valueFrom = data.aws_secretsmanager_secret_version.db.arn }]
  }])
}

# ── Kubernetes Secrets are base64, not encryption ────────────────────
kubectl get secret db -o jsonpath='{.data.password}' | base64 -d
# Anyone with get-secret RBAC reads it. Options:
#   - enable encryption at rest for etcd
#   - External Secrets Operator, syncing from a real secret manager
#   - Sealed Secrets, so only the cluster can decrypt what is in git
#   - or best: workload identity, so there is no Secret object at all

Scanning, and what to do on a hit

The revoke-before-rewrite ordering is the whole lesson: history rewriting addresses your copy of the repository and nobody else's.

# ── Prevent: block the commit before it happens ──────────────────────
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks: [{ id: gitleaks }]
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.5.0
    hooks: [{ id: detect-secrets, args: ['--baseline', '.secrets.baseline'] }]

# Enable push protection on the platform too — it blocks the push itself,
# which catches anyone who skipped the hook.

# ── Detect: scan history, not just the working tree ──────────────────
gitleaks detect --source . --report-format json --report-path leaks.json
trufflehog git file://. --only-verified        # verifies keys are LIVE

# Scan built images and running containers too
trufflehog docker --image api:latest

# ── CI gate ──────────────────────────────────────────────────────────
- name: secret scan
  run: |
    docker run --rm -v "$PWD:/repo" zricethezav/gitleaks:latest \
      detect --source /repo --verbose --redact

# ── RESPOND: the order matters ───────────────────────────────────────
# 1. REVOKE. Immediately. Before anything else.
aws iam delete-access-key --access-key-id AKIA...
# stripe: roll the key in the dashboard
# database: change the password
# JWT signing key: rotate, which invalidates every outstanding token

# 2. Rotate to a new value, deployed from a real secret store.

# 3. Investigate: was it used?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA... \
  --start-time "$(date -d '90 days ago' -Iseconds)"

# 4. Only NOW consider history rewriting — and understand it does not help
#    much: forks, clones, CI caches and the platform's own cache retain the
#    old objects.
git filter-repo --path .env --invert-paths

# 5. Record it: what leaked, when, when it was revoked, whether it was used.

# ⚠️ The single most common mistake is doing step 4 first and skipping step 1.
#    Cleaning history feels like fixing it. Revocation is what fixes it.

Discussion

Be the first to comment on this lesson.

Where they end up

The hierarchy

CI/CD specifically

Assume a leak and plan the response

Example

When to use it

More examples

OIDC federation instead of stored cloud keys

Keeping secrets out of images and state

Scanning, and what to do on a hit

Related

Discussion