Resources, Limits and Scaling

Every container declares what it needs. A request is the amount reserved for it: the scheduler only places the pod on a machine with that much CPU and memory still unreserved. A limit is the ceiling it may not pass. Requests decide where a pod runs, limits decide what happens when it gets greedy.

What each limit does when it is hit

A CPU limit does not fail anything. The kernel simply stops giving the process time slices for the rest of each short accounting period, which is called throttling: the app gets slower and CPU use looks capped rather than busy. A memory limit is harder. Cross it and the kernel kills the process, and the pod reports OOMKilled with exit code 137.

What .NET sees

Environment.ProcessorCount returns the smallest of the machine's logical processors, the affinity mask, and the CPU limit rounded up to a whole number. It is fixed at startup. With a memory limit set and no GC settings of your own, the garbage collector's heap hard limit defaults to 75% of that limit, leaving the rest for stacks, the JIT and native buffers. ASP.NET Core apps use server GC by default, which is not used on a single processor, so a CPU limit of 1 means workstation GC. Since .NET 9, server GC also runs DATAS by default, which grows the heap with the live data instead of with the core count.

Scaling out

A HorizontalPodAutoscaler changes the replica count from CPU use measured as a percentage of the request. More replicas only help if no request depends on which pod answered.

// The container's memory limit from the pod spec, in mebibytes.
int limitMiB = 512;

// With a container memory limit set and no GC configuration of your own,
// the GC's heap hard limit defaults to 75% of it (.NET Core 3.0 onwards,
// or 20 MB, whichever is larger).
int defaultHeapMiB = limitMiB * 75 / 100;

// What is left has to hold everything outside the managed heap: thread
// stacks, the JIT, and the native buffers used by sockets and TLS.
int nativeRoomMiB = limitMiB - defaultHeapMiB;

// Lowering the percentage leaves more of the limit for that native memory
// and makes the GC collect sooner.
int tunedPercent = 60;
int tunedHeapMiB = limitMiB * tunedPercent / 100;

Console.WriteLine($"memory limit: {limitMiB} MiB");
Console.WriteLine($"default gc heap hard limit: {defaultHeapMiB} MiB");
Console.WriteLine($"left for native memory: {nativeRoomMiB} MiB");
Console.WriteLine($"at {tunedPercent} percent: {tunedHeapMiB} MiB");

// Output:
// memory limit: 512 MiB
// default gc heap hard limit: 384 MiB
// left for native memory: 128 MiB
// at 60 percent: 307 MiB

containers:
  - name: api
    image: registry.example.com/orders:v1
    resources:
      requests:
        cpu: 250m          # 250m means 250 millicores, the same as 0.25
        memory: 256Mi      # Mi is 1024-based; M would be 1000-based
      limits:
        cpu: "1"           # the same as 1000m
        memory: 512Mi      # exceed this and the container is OOM killed

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70   # 70% of the 250m request, not of the limit

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

// Log this once at startup. When a pod is throttled or OOM killed, the
// first question is whether the runtime saw the limits you set.
GCMemoryInfo memory = GC.GetGCMemoryInfo();
app.Logger.LogInformation(
    "processors {Processors}, memory available to the GC {AvailableBytes} bytes",
    Environment.ProcessorCount,
    memory.TotalAvailableMemoryBytes);

app.MapGet("/orders/1", () => new { id = 1, total = 42 });

app.Run();