Authentication in Microservices
Edge authentication, internal identity, and the mistake of trusting the network.
One monolith has one authentication check. Forty services have forty opportunities to get it wrong. The senior question is not "how do I verify a token" but where does verification happen, and what does a service trust?
Three architectures
1. Verify everywhere
Every service verifies the incoming token independently. Most secure, no implicit trust, and it costs a signature verification per hop — which is microseconds with a cached JWKS. This is the default answer.
2. Verify at the edge, trust internally
The gateway verifies and forwards an identity header. Fast, and it means anything that reaches a service directly is authenticated. That is true only while the network is perfect: one misconfigured ingress, one pod with a public service, one SSRF, and every internal API is open.
3. Verify at the edge, re-sign internally
The gateway verifies the external token and mints a short-lived internal token, signed with an internal key. Services verify that. You get edge normalisation plus real internal verification — this is the pragmatic answer at scale, and it is what most large deployments converge on.
The rule that survives everything
Never trust an inbound header. If a service reads X-User-Id, that value must have been set by something that verified it — and the gateway must strip any copy the client sent. Two lines of proxy config separate a working system from one where anyone can be anyone.
Two identities per request
Internal calls carry both a workload identity (which service is calling, usually mTLS from the mesh) and a user identity (whom the call is for, from the token). They answer different questions and you need both: mTLS alone cannot express "Alice's data", and a user token alone cannot say which service asked.
Failure modes to have an answer for
- The identity provider is down. Cached JWKS keeps verification working; new logins fail. That is the correct degradation, and it is worth stating in an interview.
- Clock skew across nodes rejects valid tokens. NTP everywhere, plus a small tolerance.
- Key rotation mid-flight. Publish before signing; retire after expiry.
Example
# Pattern 3 — verify at the edge, re-sign for the inside
Internet
│ Authorization: Bearer <external token, RS256, iss=auth.dfg.com>
▼
API Gateway
│ 1. verify signature, iss, aud, exp
│ 2. STRIP every X-Internal-* header the client may have sent
│ 3. mint an internal token: 60s TTL, iss=gateway, aud=internal
▼
Service A ── mTLS ──▶ Service B ── mTLS ──▶ Service C
│ │ │
└─ verifies the internal token on every hop, every time
# Two identities travel together:
# mTLS peer certificate → WHICH SERVICE is calling
# internal token → WHOM the call is for
#
# Neither substitutes for the other.When to use it
- A gateway mints 60-second internal tokens so a service reached directly through a misconfigured ingress still rejects the request.
- A service mesh supplies workload identity by mTLS while the internal token carries the user, letting a service authorize on both.
- An identity provider outage leaves existing sessions working because every service caches the JWKS, and only new logins fail.
More examples
Minting and verifying an internal token
The 60-second lifetime is deliberate: an internal token that escapes the cluster in a log or a trace is expired long before anyone could use it.
// ---------- at the gateway ----------
import jwt from 'jsonwebtoken';
const INTERNAL_TTL_S = 60; // short: it never leaves the cluster
export async function edgeAuth(req, res, next) {
// 1. Verify the EXTERNAL token properly.
let claims;
try {
claims = await verifyExternalToken(readBearer(req));
} catch {
return res.status(401).json({ error: 'invalid_token' });
}
// 2. Remove anything the client tried to inject. Do this by allowlist —
// a denylist of header names is always one header short.
for (const name of Object.keys(req.headers)) {
if (name.toLowerCase().startsWith('x-internal-')) delete req.headers[name];
}
// 3. Mint a short-lived internal assertion.
req.headers['x-internal-token'] = jwt.sign(
{
sub: claims.sub,
scope: claims.scope,
tenant: claims.tenant,
rid: req.get('x-request-id') ?? crypto.randomUUID(),
},
INTERNAL_PRIVATE_KEY,
{ algorithm: 'RS256', expiresIn: INTERNAL_TTL_S,
issuer: 'gateway', audience: 'internal', keyid: INTERNAL_KID },
);
next();
}
// ---------- in every service ----------
export async function internalAuth(req, res, next) {
const token = req.get('x-internal-token');
if (!token) return res.status(401).json({ error: 'authentication_required' });
try {
const claims = await jwtVerify(token, INTERNAL_JWKS, {
algorithms: ['RS256'],
issuer: 'gateway', // ← only the gateway may mint these
audience: 'internal',
clockTolerance: 5,
});
req.user = { id: claims.payload.sub, tenant: claims.payload.tenant };
req.requestId = claims.payload.rid;
next();
} catch {
// A service reached directly, bypassing the gateway, lands here.
return res.status(401).json({ error: 'invalid_internal_token' });
}
}The header-trust bug, and how to see it
Point 2 is what makes point 1 non-critical. A service that verifies a signature cannot be fooled by a forged header even if the gateway forgets to strip it.
# ❌ The vulnerable pattern: the service trusts a plain header.
#
# app.use((req, res, next) => {
# req.user = { id: req.get('x-user-id') }; // ← set by "the gateway"
# next();
# });
# Reaching the service directly — through a misconfigured ingress, a NodePort,
# an SSRF, or from any other pod in the namespace:
curl http://orders-service.default.svc.cluster.local:3000/api/orders \
-H "X-User-Id: 1"
# → 200, full access as user 1. No credential involved anywhere.
# And even THROUGH the gateway, if it forwards rather than strips:
curl https://dfg.com/api/orders \
-H "Authorization: Bearer $MY_REAL_TOKEN" \
-H "X-User-Id: 1"
# → the gateway authenticates you as yourself, then passes YOUR header through.
# Test both of these against your own cluster. They take a minute and they are
# the single most common microservice authentication finding.
# The fix is two things, and you need both:
# 1. the gateway STRIPS x-internal-* / x-user-* before setting its own
# 2. the service VERIFIES a signature rather than reading a headerDegrading correctly when the identity provider is down
Naming what keeps working and what correctly stops is the answer interviewers are listening for — 'we cache the JWKS' alone does not describe the failure mode.
import { createRemoteJWKSet } from 'jose';
// jose caches the key set and applies a cooldown, so an unknown kid cannot be
// used to hammer the issuer. But an IdP outage still needs a decision.
const remoteJwks = createRemoteJWKSet(new URL(JWKS_URL), {
cooldownDuration: 30_000,
cacheMaxAge: 10 * 60_000,
});
// Keep a last-known-good copy on disk so a cold start during an outage still
// verifies existing tokens.
let lastGoodKeys = loadCachedKeys();
export async function getKey(protectedHeader, token) {
try {
const key = await remoteJwks(protectedHeader, token);
persistCachedKeys(remoteJwks.jwks()); // refresh the on-disk copy
return key;
} catch (err) {
if (!lastGoodKeys) throw err;
// Verification continues from the cached set. Existing tokens keep working;
// anything signed with a NEW key fails, which is the correct blast radius.
metrics.increment('auth.jwks.stale_fallback');
logger.warn({ err: err.message }, 'JWKS unreachable — using cached keys');
return keyFromSet(lastGoodKeys, protectedHeader.kid);
}
}
// The degradation to state in an interview:
// IdP down → existing sessions keep working (cached keys verify fine)
// → token refresh keeps working if the AS is separate
// → NEW LOGINS fail — and that is correct, because you cannot
// authenticate someone without the authority that authenticates.
//
// What you must NOT do: fail open. "IdP unreachable, allow the request" turns
// an availability incident into a total authorization bypass.
Discussion