Configuration in Production
Deliver secrets and per-environment settings to a production API safely and audit what it loaded without leaking values.
builder.Configuration.AddKeyPerFile("/run/secrets/orders-api", optional: true, reloadOnChange: true);
builder.Services.AddOptions<PaymentOptions>().BindConfiguration("Payments")
.ValidateDataAnnotations().ValidateOnStart();In production the code should not care where a value came from: it reads IConfiguration and options as it does locally. What changes is the delivery. Non-secret settings come from the platform per environment; secrets come from a secret store; nothing sensitive is in the repository or the image.
Delivery options
| Mechanism | In the app | Rotation | Trade-off |
|---|---|---|---|
| Kubernetes Secret as env vars | Default env provider | Restart needed | Simplest; inherited by child processes, visible in process dumps and printenv |
| Kubernetes Secret as a volume | AddKeyPerFile | File updates in place | One file per key; avoid subPath |
| Azure Key Vault | AddAzureKeyVault from Azure.Extensions.AspNetCore.Configuration.Secrets | Optional reload interval | -- in a secret name becomes : |
| AWS Secrets Manager | Synced into Kubernetes by External Secrets Operator or the Secrets Store CSI driver | Sync interval | App code stays provider-neutral |
Anyone who can exec into the pod can read either form. The key-per-file provider maps a file name to a key (Payments__ApiKey becomes Payments:ApiKey), uses the file content as the value with one trailing newline removed (other whitespace, including a
, is kept, so write secrets with stringData or printf '%s'), and skips files starting with ignore.. Authenticate to vaults with a workload or managed identity, so there is no bootstrap secret to manage.
Rules a team agrees on
- No secret in
appsettings*.json, Dockerfiles, Helm values or CI logs. A secret pushed to git is compromised even after a force-push: rotate it. - One image for every environment. The environment name and the platform supply the differences;
ASPNETCORE_ENVIRONMENTdefaults toProductionwhen unset, which is the safe default. - Every required value is validated at start-up with
ValidateOnStart(). - Rotation has an owner: something must reload or restart the app after a secret changes.
Auditing what the app sees
The question during an incident is always "what value did this pod actually load, and from where?". Log, at start-up, each key in your own sections with the provider that supplied it, and for everything not on an allow-list of known non-secret keys a keyed HMAC-SHA256 fingerprint (the key comes from the secret store, never the logs) or the vault's secret version. A bare SHA-256 of a password lets anyone with log access test guesses offline. Two pods with different fingerprints have different secrets. IConfigurationRoot.GetDebugView() prints every key with its provider and accepts a function to redact values. Never log it unfiltered, and never expose a configuration endpoint: it will eventually be reachable by someone it should not be.
Production failure modes
- A rotated database password: the app reads it once into a connection string, and new connections fail after the old password is revoked. Overlap old and new credentials during rotation, and restart or reload deliberately.
- A secret file mounted
0400owned by root, while the container runs as the non-rootappuser:AddKeyPerFilethrowsUnauthorizedAccessExceptionwhile configuration loads and the pod crash-loops;optional: trueonly tolerates a missing directory. UsefsGroupwith mode0440, andValidateOnStart()for a missing file or key. - A connection string in an exception message, a log scope or a health check response. Redact before anything leaves the process.
Example
When to use it
- Mount the Orders API's database password from a Kubernetes Secret volume and read it with AddKeyPerFile instead of an env var.
- Prove during an incident that all three pods loaded the same Payments:ApiKey by comparing start-up log fingerprints.
- Fail the CI build when someone pastes a real connection string into appsettings.Production.json.
More examples
Secrets as files, settings as variables
In a real cluster the Secret is created by External Secrets Operator or a sealed-secrets workflow from the vault, not committed with real values. Each key becomes a file under <code>/run/secrets/orders-api</code>, which <code>AddKeyPerFile</code> maps back to configuration keys. <code>runAsUser: 1654</code> matches the <code>app</code> user of the .NET images, and <code>fsGroup</code> with mode <code>0440</code> lets that user read the files; with root-only permissions <code>AddKeyPerFile</code> throws <code>UnauthorizedAccessException</code> while configuration loads, because <code>optional: true</code> only covers a missing directory.
apiVersion: v1
kind: Secret
metadata:
name: orders-api-secrets
type: Opaque
stringData:
ConnectionStrings__Shop: "Host=orders-db;Database=shop;Username=orders_api;Password=<from-secret-store>"
Payments__ApiKey: "<from-secret-store>"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-api
spec:
replicas: 3
selector:
matchLabels:
app: orders-api
template:
metadata:
labels:
app: orders-api
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1654
fsGroup: 1654
containers:
- name: api
image: registry.example.com/shop/orders-api:1.14.2
env:
- name: ASPNETCORE_ENVIRONMENT
value: Production
- name: Payments__Provider
value: Adyen
volumeMounts:
- name: secrets
mountPath: /run/secrets/orders-api
readOnly: true
volumes:
- name: secrets
secret:
secretName: orders-api-secrets
defaultMode: 0440A configuration provider for a secret store
This is the shape every vault integration follows: an <code>IConfigurationSource</code> that builds a <code>ConfigurationProvider</code>, a blocking first <code>Load</code> bounded by a timeout (the same one guards each refresh), and <code>OnReload()</code> when values change so <code>IOptionsMonitor</code> listeners see a rotation. Comparing before reloading avoids rebinding every options type every five minutes. Swallowing refresh errors is deliberate: a vault outage should not take down an API that already holds valid secrets, but in a real provider log the failure through a logger.
var builder = WebApplication.CreateBuilder(args);
// In production the store wraps your vault SDK client; the provider does not care which.
ISecretStore store = new EnvironmentFallbackSecretStore();
builder.Configuration.Sources.Add(new SecretStoreConfigurationSource(
store, "orders-api/", refreshInterval: TimeSpan.FromMinutes(5), fetchTimeout: TimeSpan.FromSeconds(20)));
var app = builder.Build();
app.MapGet("/", () => "orders-api");
app.Run();
public interface ISecretStore
{
Task<IReadOnlyDictionary<string, string>> GetSecretsAsync(string prefix, CancellationToken cancellationToken);
}
public sealed class EnvironmentFallbackSecretStore : ISecretStore
{
public Task<IReadOnlyDictionary<string, string>> GetSecretsAsync(string prefix, CancellationToken cancellationToken) =>
Task.FromResult<IReadOnlyDictionary<string, string>>(new Dictionary<string, string>
{
[prefix + "Payments--ApiKey"] = Environment.GetEnvironmentVariable("LOCAL_PAYMENTS_KEY") ?? ""
});
}
public sealed class SecretStoreConfigurationSource(ISecretStore store, string prefix, TimeSpan refreshInterval, TimeSpan fetchTimeout)
: IConfigurationSource
{
public IConfigurationProvider Build(IConfigurationBuilder builder) =>
new SecretStoreConfigurationProvider(store, prefix, refreshInterval, fetchTimeout);
}
public sealed class SecretStoreConfigurationProvider(ISecretStore store, string prefix, TimeSpan refreshInterval, TimeSpan fetchTimeout)
: ConfigurationProvider, IDisposable
{
private Timer? _timer;
public override void Load()
{
// Start-up must see the secrets, so the first load blocks; refreshes run in the background.
Data = FetchAsync().GetAwaiter().GetResult();
_timer ??= new Timer(_ => _ = RefreshAsync(), null, refreshInterval, refreshInterval);
}
private async Task RefreshAsync()
{
try
{
var fresh = await FetchAsync();
if (fresh.Count != Data.Count || fresh.Any(kv => !Data.TryGetValue(kv.Key, out var old) || old != kv.Value))
{
Data = fresh;
OnReload();
}
}
catch (Exception)
{
// Keep serving with the last good values; the next tick tries again.
}
}
private async Task<IDictionary<string, string?>> FetchAsync()
{
// Bounded, so a hung vault fails start-up with a clear message instead of hanging until a probe kills the pod.
using var timeout = new CancellationTokenSource(fetchTimeout);
IReadOnlyDictionary<string, string> secrets;
try
{
secrets = await store.GetSecretsAsync(prefix, timeout.Token);
}
catch (OperationCanceledException) when (timeout.IsCancellationRequested)
{
throw new TimeoutException($"Secret store did not return '{prefix}*' within {fetchTimeout.TotalSeconds}s.");
}
// Vault names cannot contain ':', so "orders-api/Payments--ApiKey" becomes "Payments:ApiKey".
return secrets.ToDictionary(
s => s.Key[prefix.Length..].Replace("--", ConfigurationPath.KeyDelimiter),
s => (string?)s.Value,
StringComparer.OrdinalIgnoreCase);
}
public void Dispose() => _timer?.Dispose();
}A CI test that keeps secrets out of settings files
The test loads every committed settings file except the Development one through the real JSON provider and fails if a key that looks secret has a value other than the agreed placeholder. It needs the API's <code>appsettings*.json</code> copied into the test output, which a linked <code>Content</code> item in the test project does. It does not replace a secret scanner in CI, but it catches the commonest slip: a real connection string pasted into <code>appsettings.Production.json</code>.
using Xunit;
public class CommittedSettingsTests
{
private static readonly string[] SecretWords = ["password", "apikey", "secret", "token", "connectionstring"];
public static TheoryData<string> SettingsFiles()
{
var files = new TheoryData<string>();
foreach (var path in Directory.EnumerateFiles(AppContext.BaseDirectory, "appsettings*.json"))
{
if (!path.EndsWith(".Development.json", StringComparison.OrdinalIgnoreCase))
{
files.Add(Path.GetFileName(path));
}
}
return files;
}
[Theory]
[MemberData(nameof(SettingsFiles))]
public void Settings_files_hold_no_secret_values(string fileName)
{
var config = new ConfigurationBuilder()
.AddJsonFile(Path.Combine(AppContext.BaseDirectory, fileName))
.Build();
var leaks = config.AsEnumerable()
.Where(e => !string.IsNullOrEmpty(e.Value))
.Where(e => SecretWords.Any(w => e.Key.Replace(":", "").Contains(w, StringComparison.OrdinalIgnoreCase)))
.Where(e => e.Value != "<from-secret-store>")
.Select(e => e.Key)
.ToList();
Assert.Empty(leaks);
}
}
Discussion