Configuration in Containers

Configure a containerised API through ports, environment, ConfigMaps, Secret files and a read-only file system without baking settings into the image.

Syntax

env:
  - name: ASPNETCORE_ENVIRONMENT
    value: Staging
  - name: ConnectionStrings__Shop
    valueFrom:
      secretKeyRef: { name: orders-db, key: connection-string }

One image is built once and promoted from staging to production; everything that differs between environments comes from outside it. In Kubernetes that means environment variables and mounted files, which the default ASP.NET Core configuration already understands.

Ports and URLs

Setting	Example	Precedence
`Kestrel:Endpoints` in configuration	`Kestrel__Endpoints__Http__Url=http://*:8080`	Wins over both below
`ASPNETCORE_URLS`	`http://+:8080`	Overrides the ports variables, with a warning in the log
`ASPNETCORE_HTTP_PORTS` / `ASPNETCORE_HTTPS_PORTS`	`8080` (set by the image)	Ports only, bound on all interfaces

The classic mistake is ASPNETCORE_URLS=http://localhost:8080 inside a container: Kestrel binds the loopback interface only, so the kubelet's probes and the Service get "connection refused" although the app logs that it is listening. Leave the image's ASPNETCORE_HTTP_PORTS=8080 alone, and terminate TLS at the ingress rather than in the pod.

Environment name

The images do not set ASPNETCORE_ENVIRONMENT, so a container runs as Production unless the manifest says otherwise. Set it in the Deployment, never in the Dockerfile: the same image must behave as Staging in staging.

ConfigMaps and Secrets

Kubernetes delivers settings in the two forms the default configuration already reads. The mechanics are covered in the configuration stage; here is where each one is taught:

Environment variables (env, envFrom, secretKeyRef): __ naming, read once at start, so a ConfigMap change needs a rollout restart. See Environment Variables and the Double Underscore.
Secret volumes: one file per key, read with AddKeyPerFile, source order decides who wins. See Configuration in Production.
Mounted JSON and live reload: reloadOnChange, IOptionsMonitor<T>, the symlink swap and the subPath caveat. See IOptionsSnapshot and IOptionsMonitor.

Read-only root file system

readOnlyRootFilesystem: true stops an attacker from dropping files, and exposes everything the app writes:

/tmp: large form uploads are buffered to temporary files once they exceed 64 KB, and the diagnostics socket used by dotnet-counters and dotnet-trace lives there. Mount an emptyDir.
Data Protection keys default to the user profile. If it is not writable, keys are kept in memory and every restart invalidates cookies and antiforgery tokens. Persist them to storage every replica shares (a ReadWriteMany volume, the database or Redis) and encrypt them at rest with ProtectKeysWithCertificate or a key vault; on Linux they are otherwise written as plain XML.

To audit what a pod actually loaded, use the start-up audit from Configuration in Production, which logs the winning provider and an HMAC fingerprint of each value. The main example below prints only the provider column of that audit.

Example

Loading editor…

Press Run to execute the code.

When to use it

Promote the same orders-api:1.4.0 image from staging to production with only the Deployment's ASPNETCORE_ENVIRONMENT and ConfigMap changing.
Mount the payment provider key from a Kubernetes Secret as a file so it never appears in the pod's environment or a crash dump.
Run the API with readOnlyRootFilesystem: true and an emptyDir on /tmp so 5 MB invoice uploads still buffer correctly.

More examples

Deployment wiring: env, Secret files, read-only root

Non-secret settings arrive through <code>envFrom</code> with keys such as <code>Orders__MaxItemsPerOrder</code>, while the Secret is mounted as files that <code>AddKeyPerFile</code> reads. The read-only root plus an <code>emptyDir</code> on <code>/tmp</code> gives the runtime the one writable place it needs, capped so a runaway upload cannot fill the node. The Data Protection key ring needs its own writable mount, and it must be a ReadWriteMany claim, because three replicas read and write the same keys.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1654
      containers:
        - name: orders-api
          image: registry.example.com/shop/orders-api:1.4.0
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: ASPNETCORE_ENVIRONMENT
              value: Production
          envFrom:
            - configMapRef:
                name: orders-api-settings
          volumeMounts:
            - name: secrets
              mountPath: /run/secrets/orders
              readOnly: true
            - name: tmp
              mountPath: /tmp
            - name: dp-keys
              mountPath: /var/lib/orders/dp-keys
            - name: dp-cert
              mountPath: /run/secrets/dp-cert
              readOnly: true
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
      volumes:
        - name: secrets
          secret:
            secretName: orders-api-secrets
        - name: tmp
          emptyDir:
            sizeLimit: 256Mi
        - name: dp-keys
          persistentVolumeClaim:
            claimName: orders-dp-keys    # ReadWriteMany: all three replicas share the key ring
        - name: dp-cert
          secret:
            secretName: orders-dp-cert

Program.cs for the mounted secrets and keys

Ordering is the decision here: sources added later win, so <code>AddKeyPerFile</code> comes after the ConfigMap's JSON and the Secret files override anything an operator left in it. The Data Protection call matters on a read-only file system, where the default key folder is not writable and keys would otherwise live only in memory until the next restart; <code>ProtectKeysWithCertificate</code> stops them sitting on the shared volume as plain XML.

using System.Security.Cryptography.X509Certificates;
using Microsoft.AspNetCore.DataProtection;

var builder = WebApplication.CreateBuilder(args);

// Optional mounted ConfigMap file; polling is set via DOTNET_USE_POLLING_FILE_WATCHER.
builder.Configuration.AddJsonFile("/config/orders.json", optional: true, reloadOnChange: true);

// Added last, so a mounted Secret overrides the ConfigMap file, env vars and appsettings.
builder.Configuration.AddKeyPerFile("/run/secrets/orders", optional: true, reloadOnChange: true);

// Keys on a ReadWriteMany volume so cookies survive restarts and work on every replica,
// encrypted at rest with a certificate from another Secret.
var keyCertificate = X509CertificateLoader.LoadPkcs12FromFile(
    "/run/secrets/dp-cert/dp-keys.pfx", builder.Configuration["DataProtection:CertificatePassword"]);
builder.Services.AddDataProtection()
    .SetApplicationName("orders-api")
    .PersistKeysToFileSystem(new DirectoryInfo("/var/lib/orders/dp-keys"))
    .ProtectKeysWithCertificate(keyCertificate);

var app = builder.Build();

var shop = app.Configuration.GetConnectionString("Shop");
app.Logger.LogInformation("ConnectionStrings:Shop present: {Present}", !string.IsNullOrEmpty(shop));

app.MapGet("/orders/{id:int}", (int id) => Results.Ok(new { id, status = "Shipped" }));
app.Run();

Creating and rolling the settings

The <code>--dry-run=client -o yaml | kubectl apply</code> pattern makes the commands idempotent, so a pipeline can rerun them. In real clusters the Secret usually comes from an external secret operator rather than a literal; the restart is still needed because environment variables never change in a running process.

kubectl create configmap orders-api-settings \
  --from-literal=Orders__MaxItemsPerOrder=50 \
  --from-literal=Logging__LogLevel__Default=Information \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic orders-api-secrets \
  --from-literal=ConnectionStrings__Shop='Host=postgres.shop.svc;Database=shop;Username=orders;Password=<from-secret-store>' \
  --dry-run=client -o yaml | kubectl apply -f -

# Env vars are only read at start: restart the pods to pick up the change
kubectl rollout restart deployment/orders-api
kubectl rollout status deployment/orders-api --timeout=5m

Discussion

Be the first to comment on this lesson.