VPC CNI IP Exhaustion & Prefix Delegation

Real pod IPs are wonderful until your /24 runs dry. Prefix delegation multiplies pod density; secondary CIDRs buy headroom.

Because the VPC CNI gives every pod a real VPC IP, a busy cluster can devour a subnet. The classic failure is pods stuck ContainerCreating with failed to assign an IP address — you've run out of addresses, not compute.

Why the ceiling is lower than you think

Without prefix delegation, each instance's pod count is capped by how many secondary IPs its ENIs can hold. A modest instance might top out around 17–29 pods regardless of spare CPU or memory.

Prefix delegation — the big lever

Enable ENABLE_PREFIX_DELEGATION=true on the aws-node DaemonSet and each ENI is assigned /28 prefixes (16 IPs each) instead of single addresses. Pod density per node jumps dramatically (often to the 110/250 Kubernetes limits) using the same ENIs. This is the first thing to turn on for dense clusters.

When you're simply out of subnet space

Add a secondary CIDR to the VPC (e.g. from the 100.64.0.0/10 carrier-grade range) and give the CNI custom networking so pods draw from big, dedicated subnets while nodes stay on the primary range.
Plan subnets generously from day one — a /19 per AZ is not excessive for a growing cluster.

Example

# Turn on prefix delegation to massively raise pods-per-node
kubectl set env daemonset aws-node -n kube-system \
  ENABLE_PREFIX_DELEGATION=true

# Keep warm-IP hoarding in check
kubectl set env daemonset aws-node -n kube-system \
  WARM_PREFIX_TARGET=1

# Diagnose exhaustion: how many IPs are free on the subnet?
aws ec2 describe-subnets --subnet-ids subnet-abc \
  --query 'Subnets[0].AvailableIpAddressCount'

# See why a pod is stuck
kubectl describe pod stuck-pod | grep -A3 Events

When to use it

A platform team enables prefix delegation on a /24 VPC subnet to increase pod capacity from 110 to 1750 pods per m5.xlarge node without requesting new IP ranges.
A network engineer moves EKS node groups to dedicated /20 subnets separate from the application VPC to prevent pods from consuming IPs needed by RDS and EC2.
A team hitting subnet IP exhaustion uses custom networking to assign pods from a secondary CIDR block while keeping node IPs in the original subnet.

More examples

Enable prefix delegation on VPC CNI

Enables prefix delegation on the VPC CNI so each ENI attachment assigns a /28 CIDR block (16 IPs) instead of individual IPs, multiplying pod density per node.

kubectl set env daemonset aws-node -n kube-system \
  ENABLE_PREFIX_DELEGATION=true \
  WARM_PREFIX_TARGET=1 \
  MINIMUM_IP_TARGET=3
kubectl rollout restart daemonset/aws-node -n kube-system
kubectl rollout status daemonset/aws-node -n kube-system

Check pod IP availability on node

Shows the allocatable pod count for a node and the current pod count to determine how close the node is to IP exhaustion.

NODE=ip-10-0-1-100.ec2.internal
kubectl describe node $NODE | grep -E 'Allocatable|pods'
kubectl get pods --field-selector spec.nodeName=$NODE | wc -l

Custom networking for secondary CIDR

Enables custom networking mode so pods are assigned IPs from ENIConfig-defined subnets (typically a secondary VPC CIDR) instead of the node's primary subnet.

kubectl set env daemonset aws-node -n kube-system \
  AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true \
  ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

Discussion

Be the first to comment on this lesson.