Configuration in Production

Deliver secrets and per-environment settings to a production API safely and audit what it loaded without leaking values.

Syntaxbuilder.Configuration.AddKeyPerFile("/run/secrets/orders-api", optional: true, reloadOnChange: true); builder.Services.AddOptions<PaymentOptions>().BindConfiguration("Payments") .ValidateDataAnnotations().ValidateOnStart();

In production the code should not care where a value came from: it reads IConfiguration and options as it does locally. What changes is the delivery. Non-secret settings come from the platform per environment; secrets come from a secret store; nothing sensitive is in the repository or the image.

Delivery options

MechanismIn the appRotationTrade-off
Kubernetes Secret as env varsDefault env providerRestart neededSimplest; inherited by child processes, visible in process dumps and printenv
Kubernetes Secret as a volumeAddKeyPerFileFile updates in placeOne file per key; avoid subPath
Azure Key VaultAddAzureKeyVault from Azure.Extensions.AspNetCore.Configuration.SecretsOptional reload interval-- in a secret name becomes :
AWS Secrets ManagerSynced into Kubernetes by External Secrets Operator or the Secrets Store CSI driverSync intervalApp code stays provider-neutral

Anyone who can exec into the pod can read either form. The key-per-file provider maps a file name to a key (Payments__ApiKey becomes Payments:ApiKey), uses the file content as the value with one trailing newline removed (other whitespace, including a , is kept, so write secrets with stringData or printf '%s'), and skips files starting with ignore.. Authenticate to vaults with a workload or managed identity, so there is no bootstrap secret to manage.

Rules a team agrees on

  • No secret in appsettings*.json, Dockerfiles, Helm values or CI logs. A secret pushed to git is compromised even after a force-push: rotate it.
  • One image for every environment. The environment name and the platform supply the differences; ASPNETCORE_ENVIRONMENT defaults to Production when unset, which is the safe default.
  • Every required value is validated at start-up with ValidateOnStart().
  • Rotation has an owner: something must reload or restart the app after a secret changes.

Auditing what the app sees

The question during an incident is always "what value did this pod actually load, and from where?". Log, at start-up, each key in your own sections with the provider that supplied it, and for everything not on an allow-list of known non-secret keys a keyed HMAC-SHA256 fingerprint (the key comes from the secret store, never the logs) or the vault's secret version. A bare SHA-256 of a password lets anyone with log access test guesses offline. Two pods with different fingerprints have different secrets. IConfigurationRoot.GetDebugView() prints every key with its provider and accepts a function to redact values. Never log it unfiltered, and never expose a configuration endpoint: it will eventually be reachable by someone it should not be.

Production failure modes

  • A rotated database password: the app reads it once into a connection string, and new connections fail after the old password is revoked. Overlap old and new credentials during rotation, and restart or reload deliberately.
  • A secret file mounted 0400 owned by root, while the container runs as the non-root app user: AddKeyPerFile throws UnauthorizedAccessException while configuration loads and the pod crash-loops; optional: true only tolerates a missing directory. Use fsGroup with mode 0440, and ValidateOnStart() for a missing file or key.
  • A connection string in an exception message, a log scope or a health check response. Redact before anything leaves the process.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Mount the Orders API's database password from a Kubernetes Secret volume and read it with AddKeyPerFile instead of an env var.
  • Prove during an incident that all three pods loaded the same Payments:ApiKey by comparing start-up log fingerprints.
  • Fail the CI build when someone pastes a real connection string into appsettings.Production.json.

More examples

Secrets as files, settings as variables

In a real cluster the Secret is created by External Secrets Operator or a sealed-secrets workflow from the vault, not committed with real values. Each key becomes a file under <code>/run/secrets/orders-api</code>, which <code>AddKeyPerFile</code> maps back to configuration keys. <code>runAsUser: 1654</code> matches the <code>app</code> user of the .NET images, and <code>fsGroup</code> with mode <code>0440</code> lets that user read the files; with root-only permissions <code>AddKeyPerFile</code> throws <code>UnauthorizedAccessException</code> while configuration loads, because <code>optional: true</code> only covers a missing directory.

Example · yaml
apiVersion: v1
kind: Secret
metadata:
  name: orders-api-secrets
type: Opaque
stringData:
  ConnectionStrings__Shop: "Host=orders-db;Database=shop;Username=orders_api;Password=<from-secret-store>"
  Payments__ApiKey: "<from-secret-store>"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1654
        fsGroup: 1654
      containers:
        - name: api
          image: registry.example.com/shop/orders-api:1.14.2
          env:
            - name: ASPNETCORE_ENVIRONMENT
              value: Production
            - name: Payments__Provider
              value: Adyen
          volumeMounts:
            - name: secrets
              mountPath: /run/secrets/orders-api
              readOnly: true
      volumes:
        - name: secrets
          secret:
            secretName: orders-api-secrets
            defaultMode: 0440

A configuration provider for a secret store

This is the shape every vault integration follows: an <code>IConfigurationSource</code> that builds a <code>ConfigurationProvider</code>, a blocking first <code>Load</code> bounded by a timeout (the same one guards each refresh), and <code>OnReload()</code> when values change so <code>IOptionsMonitor</code> listeners see a rotation. Comparing before reloading avoids rebinding every options type every five minutes. Swallowing refresh errors is deliberate: a vault outage should not take down an API that already holds valid secrets, but in a real provider log the failure through a logger.

Example · csharp
var builder = WebApplication.CreateBuilder(args);

// In production the store wraps your vault SDK client; the provider does not care which.
ISecretStore store = new EnvironmentFallbackSecretStore();
builder.Configuration.Sources.Add(new SecretStoreConfigurationSource(
    store, "orders-api/", refreshInterval: TimeSpan.FromMinutes(5), fetchTimeout: TimeSpan.FromSeconds(20)));

var app = builder.Build();
app.MapGet("/", () => "orders-api");
app.Run();

public interface ISecretStore
{
    Task<IReadOnlyDictionary<string, string>> GetSecretsAsync(string prefix, CancellationToken cancellationToken);
}

public sealed class EnvironmentFallbackSecretStore : ISecretStore
{
    public Task<IReadOnlyDictionary<string, string>> GetSecretsAsync(string prefix, CancellationToken cancellationToken) =>
        Task.FromResult<IReadOnlyDictionary<string, string>>(new Dictionary<string, string>
        {
            [prefix + "Payments--ApiKey"] = Environment.GetEnvironmentVariable("LOCAL_PAYMENTS_KEY") ?? ""
        });
}

public sealed class SecretStoreConfigurationSource(ISecretStore store, string prefix, TimeSpan refreshInterval, TimeSpan fetchTimeout)
    : IConfigurationSource
{
    public IConfigurationProvider Build(IConfigurationBuilder builder) =>
        new SecretStoreConfigurationProvider(store, prefix, refreshInterval, fetchTimeout);
}

public sealed class SecretStoreConfigurationProvider(ISecretStore store, string prefix, TimeSpan refreshInterval, TimeSpan fetchTimeout)
    : ConfigurationProvider, IDisposable
{
    private Timer? _timer;

    public override void Load()
    {
        // Start-up must see the secrets, so the first load blocks; refreshes run in the background.
        Data = FetchAsync().GetAwaiter().GetResult();
        _timer ??= new Timer(_ => _ = RefreshAsync(), null, refreshInterval, refreshInterval);
    }

    private async Task RefreshAsync()
    {
        try
        {
            var fresh = await FetchAsync();
            if (fresh.Count != Data.Count || fresh.Any(kv => !Data.TryGetValue(kv.Key, out var old) || old != kv.Value))
            {
                Data = fresh;
                OnReload();
            }
        }
        catch (Exception)
        {
            // Keep serving with the last good values; the next tick tries again.
        }
    }

    private async Task<IDictionary<string, string?>> FetchAsync()
    {
        // Bounded, so a hung vault fails start-up with a clear message instead of hanging until a probe kills the pod.
        using var timeout = new CancellationTokenSource(fetchTimeout);
        IReadOnlyDictionary<string, string> secrets;
        try
        {
            secrets = await store.GetSecretsAsync(prefix, timeout.Token);
        }
        catch (OperationCanceledException) when (timeout.IsCancellationRequested)
        {
            throw new TimeoutException($"Secret store did not return '{prefix}*' within {fetchTimeout.TotalSeconds}s.");
        }
        // Vault names cannot contain ':', so "orders-api/Payments--ApiKey" becomes "Payments:ApiKey".
        return secrets.ToDictionary(
            s => s.Key[prefix.Length..].Replace("--", ConfigurationPath.KeyDelimiter),
            s => (string?)s.Value,
            StringComparer.OrdinalIgnoreCase);
    }

    public void Dispose() => _timer?.Dispose();
}

A CI test that keeps secrets out of settings files

The test loads every committed settings file except the Development one through the real JSON provider and fails if a key that looks secret has a value other than the agreed placeholder. It needs the API's <code>appsettings*.json</code> copied into the test output, which a linked <code>Content</code> item in the test project does. It does not replace a secret scanner in CI, but it catches the commonest slip: a real connection string pasted into <code>appsettings.Production.json</code>.

Example · csharp
using Xunit;

public class CommittedSettingsTests
{
    private static readonly string[] SecretWords = ["password", "apikey", "secret", "token", "connectionstring"];

    public static TheoryData<string> SettingsFiles()
    {
        var files = new TheoryData<string>();
        foreach (var path in Directory.EnumerateFiles(AppContext.BaseDirectory, "appsettings*.json"))
        {
            if (!path.EndsWith(".Development.json", StringComparison.OrdinalIgnoreCase))
            {
                files.Add(Path.GetFileName(path));
            }
        }
        return files;
    }

    [Theory]
    [MemberData(nameof(SettingsFiles))]
    public void Settings_files_hold_no_secret_values(string fileName)
    {
        var config = new ConfigurationBuilder()
            .AddJsonFile(Path.Combine(AppContext.BaseDirectory, fileName))
            .Build();

        var leaks = config.AsEnumerable()
            .Where(e => !string.IsNullOrEmpty(e.Value))
            .Where(e => SecretWords.Any(w => e.Key.Replace(":", "").Contains(w, StringComparison.OrdinalIgnoreCase)))
            .Where(e => e.Value != "<from-secret-store>")
            .Select(e => e.Key)
            .ToList();

        Assert.Empty(leaks);
    }
}

Discussion

  • Be the first to comment on this lesson.