Secrets Management & External Secrets

Base64 is not encryption. Production secret handling means encryption at rest, tight RBAC, and pulling real secrets from an external vault at runtime.

A Kubernetes Secret is just base64-encoded data in etcd — trivially decodable by anyone with read access. Treating Secrets as if they were secure is a classic rookie mistake. Here is the grown-up version.

Layer your defenses

Encrypt etcd at rest with an EncryptionConfiguration (ideally KMS-backed), so a stolen etcd backup is not a breach.
Lock down RBAC so only the workloads that need a Secret can get it — see the least-privilege lesson.
Keep secrets out of Git. Never commit a raw Secret manifest. Use Sealed Secrets (encrypted, safe to commit) or an external store.
Prefer an external vault. Tools like the External Secrets Operator (ESO) or the Secrets Store CSI Driver sync from AWS Secrets Manager, Vault, or GCP Secret Manager into short-lived Kubernetes Secrets — so the source of truth lives outside the cluster and rotates centrally.

How External Secrets Operator works

You define a SecretStore (where the real secrets live) and an ExternalSecret (which keys to pull and how often to refresh). ESO reconciles them into a normal Secret your Pods consume as usual — your app code never changes, but the value now originates from and rotates in the vault.

Example

# External Secrets Operator: pull a DB password from AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-store
  namespace: prod
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: config-reader   # IRSA-annotated SA, no static keys
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: prod
spec:
  refreshInterval: 1h            # re-sync (and pick up rotation) hourly
  secretStoreRef:
    name: aws-store
    kind: SecretStore
  target:
    name: db-credentials         # ESO creates this native Secret
  data:
    - secretKey: password
      remoteRef:
        key: prod/db
        property: password

When to use it

A team integrates Sealed Secrets so encrypted secrets can be committed to Git safely, letting GitOps tools sync them without exposing plaintext.
A company uses the External Secrets Operator to sync secrets from AWS Secrets Manager into Kubernetes Secrets, removing the need to manage secret rotation manually.
An SRE enables envelope encryption on the etcd Secret store using a KMS key so that Secret data at rest is encrypted beyond just base64 encoding.

More examples

Sealed Secret encryption

Uses kubeseal to encrypt a secret's data before committing it to Git; only the in-cluster Sealed Secrets controller can decrypt it.

# Encrypt a secret with kubeseal
kubectl create secret generic db-creds \
  --from-literal=password='S3cr3t' \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > sealed-db-creds.yaml

# Commit sealed-db-creds.yaml to Git safely
git add sealed-db-creds.yaml

External Secrets Operator

Defines an ExternalSecret that syncs a value from AWS Secrets Manager into a Kubernetes Secret every hour automatically.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-creds
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: db-creds
  data:
    - secretKey: password
      remoteRef:
        key: prod/myapp/db_password

Avoid secrets in pod environment

Mounts the secret as a read-only file volume instead of an environment variable, reducing exposure in process listings and crash dumps.

# Mount secret as a file instead of env var
spec:
  containers:
    - name: app
      image: myapp:1.0
      volumeMounts:
        - name: secrets
          mountPath: /run/secrets
          readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: db-creds

Discussion

Be the first to comment on this lesson.