Rate Limiting

Cap how many requests each client may send per period, answer the rest with 429 and Retry-After, and keep the limit consistent across replicas.

Syntaxclass RateLimitFilter extends OncePerRequestFilter { protected void doFilterInternal(req, res, chain) { if (!limiter.tryAcquire(keyFor(req))) { res.setStatus(429); return; } chain.doFilter(req, res); } }

A rate limit protects your service and your other clients from one caller that sends too much: a buggy retry loop, a scraper, or a customer's script run in a tight loop. Without one, a single client can use every thread and connection, and everyone else sees timeouts.

Where to enforce it

  • At the edge (API gateway, reverse proxy, CDN): cheapest, stops abusive traffic before it reaches the JVM. Good for crude per-IP limits.
  • In the application: knows who the caller is (API key, user, tenant) and which endpoint is expensive, so it can apply per-plan or per-route limits.

Most production APIs use both. Spring Boot has no built-in rate limiter, so in-application limits are a servlet filter you write, or a library such as Bucket4j or Resilience4j.

Algorithms

A fixed window counts requests per calendar minute; it is a single Redis INCR, but allows a burst of double the limit across a window boundary. A token bucket refills at a steady rate up to a capacity: it allows short bursts and smooths the long-run rate, which is what most APIs want.

Identifying the caller

Key limits by API key or authenticated user wherever possible. IP addresses are shared by whole offices and mobile networks, and behind a load balancer every request comes from the balancer unless you set server.forward-headers-strategy and trust only your own proxy. Never key on a header the client can set freely, such as an API key that has not been checked yet: a client can send a new value with every request. Run the limiter after authentication and key on the verified identity.

Answering correctly

Reject with 429 Too Many Requests and a Retry-After header in seconds. Well-behaved clients and SDKs back off on it; without it they retry immediately and make the overload worse. Return a problem detail body like any other error.

One instance or many

An in-memory limiter allows limit times replicas, a restart resets it, and its map of buckets must expire idle keys or it grows with every new client. That is fine for protecting one instance from overload. For a contractual per-customer quota, keep the counters in Redis so every replica sees the same count, and decide whether to allow or reject requests when Redis is unreachable.

Rate is not concurrency

A rate limit counts requests per second. A slow report endpoint can still be overwhelmed by five requests a second if each takes a minute. For that, limit how many run at once with @ConcurrencyLimit from Spring Framework 7, covered in its own lesson.

Example

Example · java
import java.io.IOException;
import java.security.Principal;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;

import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;

final class TokenBucket {

    private final long capacity;
    private final double refillPerNano;
    private double tokens;
    private long lastRefill;

    TokenBucket(long capacity, long refillPerSecond) {
        this.capacity = capacity;
        this.refillPerNano = refillPerSecond / 1_000_000_000.0;
        this.tokens = capacity;
        this.lastRefill = System.nanoTime();
    }

    synchronized boolean tryConsume() {
        long now = System.nanoTime();
        tokens = Math.min(capacity, tokens + (now - lastRefill) * refillPerNano);
        lastRefill = now;
        if (tokens >= 1) {
            tokens -= 1;
            return true;
        }
        return false;
    }
}

@Component
public class RateLimitFilter extends OncePerRequestFilter {

    // 20-request burst, 10 requests per second sustained, per caller.
    // This map never forgets a key; in production use an expiring cache or Bucket4j.
    private final ConcurrentMap<String, TokenBucket> buckets = new ConcurrentHashMap<>();

    @Override
    protected boolean shouldNotFilter(HttpServletRequest request) {
        return !request.getRequestURI().startsWith("/api/");
    }

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response,
                                    FilterChain chain) throws ServletException, IOException {
        // Runs after Spring Security, so the principal is the verified caller.
        // Never key on a raw header: a client could rotate values to dodge the limit.
        Principal user = request.getUserPrincipal();
        String key = user != null ? "user:" + user.getName() : "ip:" + request.getRemoteAddr();

        if (buckets.size() > 100_000) {
            buckets.clear();  // crude bound so rotating clients cannot exhaust memory
        }
        TokenBucket bucket = buckets.computeIfAbsent(key, k -> new TokenBucket(20, 10));
        if (bucket.tryConsume()) {
            chain.doFilter(request, response);
            return;
        }
        response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
        response.setHeader("Retry-After", "1");
        response.setContentType(MediaType.APPLICATION_PROBLEM_JSON_VALUE);
        response.getWriter().write("""
                {"type":"about:blank","title":"Too Many Requests","status":429,\
                "detail":"Rate limit exceeded, retry after 1 second"}""");
    }
}

When to use it

  • A partner's integration retries in a tight loop after an outage; per-key limits return 429 to them while everyone else is served normally.
  • Free-plan API keys get 60 requests a minute and paid keys 6,000, enforced in Redis so the quota is the same on all replicas.
  • A login endpoint is limited per account and per IP to slow down password guessing.

More examples

A shared fixed-window limit in Redis

Each tenant gets one counter per minute. The Lua script makes the increment and the expiry one atomic step, so a crash between them can never leave a counter without a TTL. Every replica talks to the same key, so the quota holds however many instances run. Wrap the call in a try/catch and decide deliberately whether a Redis outage should let traffic through or block it.

Example · java
import java.time.Instant;
import java.util.List;

import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.RedisScript;
import org.springframework.stereotype.Component;

@Component
public class RedisRateLimiter {

    // INCR and EXPIRE run atomically on the Redis server.
    private static final RedisScript<Long> INCREMENT = RedisScript.of("""
            local count = redis.call('INCR', KEYS[1])
            if count == 1 then
              redis.call('EXPIRE', KEYS[1], ARGV[1])
            end
            return count
            """, Long.class);

    private final StringRedisTemplate redis;

    RedisRateLimiter(StringRedisTemplate redis) {
        this.redis = redis;
    }

    /** Allows {@code limit} calls per minute for the tenant, across every replica. */
    public boolean allow(String tenantId, long limit) {
        long minute = Instant.now().getEpochSecond() / 60;
        String key = "rate:" + tenantId + ":" + minute;
        Long count = redis.execute(INCREMENT, List.of(key), "60");
        return count != null && count <= limit;
    }
}

Limiting concurrency instead of rate

A rate limit of 10 per second does not protect an endpoint where each call takes 40 seconds: 400 would pile up. <code>@ConcurrencyLimit</code> caps how many calls run at the same time and makes the rest wait, which keeps the connection pool and CPU free for other endpoints. Use both together on expensive routes: the rate limit for fairness between clients, the concurrency limit for the server's own safety.

Example · java
import org.springframework.context.annotation.Configuration;
import org.springframework.resilience.annotation.ConcurrencyLimit;
import org.springframework.resilience.annotation.EnableResilientMethods;
import org.springframework.stereotype.Service;

@Configuration
@EnableResilientMethods
class ResilienceConfig {
}

@Service
public class ReportGenerator {

    // At most 3 reports build at once on this instance; extra callers wait.
    @ConcurrencyLimit(3)
    public byte[] generate(long accountId) {
        return new byte[0];  // heavy query and PDF rendering
    }
}

Coarse per-IP limiting at the reverse proxy

An edge limit drops floods before they cost a JVM thread, a TLS handshake on the app, or a database query. It cannot tell paying customers apart, so keep it generous and use it as a backstop. The finer, per-key limits belong in the application, where the caller's identity and plan are known.

Example · text
# nginx: 20 requests per second per client IP, bursts of 40 absorbed.
limit_req_zone $binary_remote_addr zone=api:10m rate=20r/s;

server {
    location /api/ {
        limit_req zone=api burst=40 nodelay;
        limit_req_status 429;
        proxy_pass http://app:8080;
    }
}

Related

sb limiting concurrency with concurrencylimitsb a redis cachesb running behind a reverse proxysb problem details rfc 9457sb how spring security works

Discussion

  • Be the first to comment on this lesson.