Retries, Timeouts & Rate Limits
LLM APIs are network calls over slow, occasionally overloaded services — build in backoff, timeouts, and graceful limits.
An LLM call is a network request to a shared, sometimes-busy service. Treat it with the same resilience you would any external dependency, plus a few LLM-specific twists.
Retry the right errors, with backoff
Transient failures — rate limits (429), overloads, and 5xx server errors — are safe to retry. Client errors (a malformed request, a bad API key, a 400) are not — retrying just wastes time and money. Use exponential backoff with jitter so a burst of failures doesn't turn into a synchronized retry stampede that makes things worse.
Respect the Retry-After header
When you're rate-limited, the response usually tells you exactly how long to wait. Honour it instead of guessing. Most official SDKs retry 429 and 5xx automatically with sane defaults — lean on that before hand-rolling your own loop.
Set timeouts — and stream long calls
A request that hangs forever is worse than one that fails fast. Set a timeout appropriate to the task. For long outputs, stream the response: a streaming connection stays active token-by-token and sidesteps the idle-connection timeouts that kill large non-streaming requests.
Degrade gracefully at the limit
When you're persistently throttled, have a plan: queue non-urgent work, fall back to a smaller/less-loaded model, or show the user a clear "busy, try again shortly" message. Silent failures and infinite spinners are the worst outcome.
Example
async function callWithRetry(fn, { maxRetries = 4, baseMs = 500 } = {}) {
for (let attempt = 0; ; attempt++) {
try {
return await fn();
} catch (err) {
const status = err.status;
const retryable = status === 429 || status >= 500;
if (!retryable || attempt >= maxRetries) throw err; // 4xx: don't retry
// Honour Retry-After when present; otherwise exponential backoff + jitter.
const retryAfter = Number(err.headers?.["retry-after"]) * 1000;
const backoff = baseMs * 2 ** attempt;
const jitter = Math.random() * baseMs;
const waitMs = retryAfter || backoff + jitter;
console.warn(`retry ${attempt + 1}/${maxRetries} in ${Math.round(waitMs)}ms`);
await new Promise((r) => setTimeout(r, waitMs));
}
}
}
// const res = await callWithRetry(() => llm.create({ model, messages }));When to use it
- A production service wraps every API call in an exponential-backoff retry so a temporary rate-limit error retries automatically without user impact.
- A batch pipeline sets a per-request timeout of 30 seconds and retries timed-out requests up to three times before logging the failure.
- A multi-tenant SaaS tracks per-user request counts and pauses AI features gracefully when a tenant approaches the provider's rate limit.
More examples
Exponential backoff retry loop
Retries on RateLimitError and 5xx errors with exponential backoff, raising only after all retries are exhausted.
import time
from openai import OpenAI, RateLimitError, APIError
client = OpenAI()
def call_with_retry(messages, max_retries=4):
delay = 1.0
for attempt in range(max_retries):
try:
return client.chat.completions.create(
model='gpt-4o-mini', messages=messages
)
except RateLimitError:
if attempt == max_retries - 1:
raise
print(f'Rate limited — retrying in {delay:.1f}s')
time.sleep(delay)
delay *= 2
except APIError as e:
if e.status_code >= 500 and attempt < max_retries - 1:
time.sleep(delay); delay *= 2
else:
raise
r = call_with_retry([{'role':'user','content':'Hello!'}])
print(r.choices[0].message.content)Request timeout with httpx
Configures a 30-second total timeout and a 5-second connect timeout so hung requests fail fast.
from openai import OpenAI
import httpx
client = OpenAI(
http_client=httpx.Client(timeout=httpx.Timeout(30.0, connect=5.0))
)
try:
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':'Summarise the water cycle.'}]
)
print(r.choices[0].message.content)
except httpx.TimeoutException:
print('Request timed out after 30 seconds')Rate-limit requests with a token bucket
Implements a client-side token bucket to self-throttle at 2 requests per second, staying within rate limits.
import time
from openai import OpenAI
client = OpenAI()
class TokenBucket:
def __init__(self, rate, capacity):
self.rate = rate # tokens added per second
self.capacity = capacity
self.tokens = capacity
self.last = time.time()
def acquire(self):
now = time.time()
self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
self.last = now
if self.tokens >= 1:
self.tokens -= 1
return True
return False
bucket = TokenBucket(rate=2, capacity=5) # 2 requests/sec max
for i in range(6):
if bucket.acquire():
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':f'Question {i}'}],
max_tokens=5
)
print(f'Request {i}: {r.choices[0].message.content.strip()}')
else:
print(f'Request {i}: throttled')
time.sleep(0.3)
Discussion