Rate Limits, Retries and Timeouts

Obey the provider's retry-after header, back off with jitter, budget a timeout per call, and keep a retry from duplicating work.

Syntaxclient.with_options(timeout=20.0, max_retries=5).messages.create(...)

Model APIs rate-limit per minute on both requests and tokens, and they are slower and more variable than a database. Every call therefore needs three decisions made up front: how long to wait, what to retry, and what a retry might duplicate.

429 and what the provider tells you

A 429 reply carries a retry-after header in seconds. Obey it in preference to your own arithmetic, because it reflects the window the server is actually enforcing. Retry 429, 408, 409 and 5xx plus connection errors. Never retry a 400 or a 404: the request is wrong and sending it again just wastes the budget.

Backoff with jitter

Exponential backoff doubles the wait after each failure; jitter picks a random point inside that window. Without jitter, every worker that failed in the same second retries in the same later second and recreates the spike. Full jitter, sleep(random.uniform(0, min(cap, base * 2 ** attempt))), is the simple default.

Idempotency

A timeout does not tell you whether the server did the work. If the call has a side effect of your own, such as charging an account or writing a row, derive a stable key from the inputs and make the side effect conditional on that key, so the second attempt lands on the same row rather than a new one.

What trips people up

The SDK already retries 429 and 5xx twice with backoff, so hand-rolled retries around it multiply: five of yours around two of theirs is fifteen attempts. Set max_retries explicitly, or max_retries=0 if your own loop owns the policy.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Keeping a web handler responsive when the provider is slow
  • Draining a backlog of prompts without tripping the per-minute limit
  • Making sure a timed-out call does not charge a customer twice
  • Diagnosing why one request took half an hour before failing

More examples

Let the SDK do it, but bound it

The usual correct answer: configure the SDK rather than wrapping it. A classification call behind a web request gets a short timeout and no retries so the handler stays fast, and the work goes to a queue on failure. Catch the typed exceptions, most specific first, instead of matching on error strings.

Example · python
import anthropic

# Client-wide policy: 20s per attempt, 4 retries of 429/5xx with backoff.
client = anthropic.Anthropic(timeout=20.0, max_retries=4)

# Per-request override, without mutating the client.
quick = client.with_options(timeout=5.0, max_retries=0)

try:
    reply = quick.messages.create(
        model="claude-opus-5-5", max_tokens=256,
        messages=[{"role": "user", "content": "Classify: " + text}],
    )
except anthropic.RateLimitError as exc:
    wait = int(exc.response.headers.get("retry-after", "60"))
    enqueue_for_later(text, delay=wait)
except anthropic.APITimeoutError:
    enqueue_for_later(text, delay=30)

An explicit loop when you own the policy

Reach for this when the SDK's policy does not fit: a worker that must respect a shared token budget, or logging of every attempt. The two things that make it correct are disabling the SDK's retries and raising immediately on a 4xx that is not 429.

Example · python
import random, time, anthropic

def call(client, **kwargs):
    # max_retries=0: this loop is the only retry policy, so attempts do not
    # multiply with the SDK's own.
    client = client.with_options(max_retries=0)
    last = None
    for attempt in range(5):
        try:
            return client.messages.create(**kwargs)
        except anthropic.RateLimitError as exc:
            last = exc
            header = exc.response.headers.get("retry-after")
            delay = float(header) if header else None
        except anthropic.APIStatusError as exc:
            if exc.status_code < 500:
                raise                      # 400/404: retrying cannot help
            last, delay = exc, None
        if delay is None:
            delay = random.uniform(0, min(30.0, 0.5 * 2 ** attempt))
        time.sleep(delay)
    raise last

A retried call that must not double-charge

The pattern for a call with a side effect. The key is derived from the inputs, so a retry after a timeout computes the same key, finds the existing row and returns it. INSERT OR IGNORE closes the race where two workers finish at once. Without this, every timeout risks a second paid call and a duplicate row.

Example · python
def summarise_and_bill(conn, user_id, document):
    key = idempotency_key(user_id, document)
    row = conn.execute(
        "SELECT summary FROM summaries WHERE idem_key = ?", (key,)
    ).fetchone()
    if row:
        return row[0]            # a previous attempt already paid for this

    summary = call(client, model="claude-opus-5-5", max_tokens=1024,
                   messages=[{"role": "user", "content": document}])
    conn.execute(
        "INSERT OR IGNORE INTO summaries (idem_key, summary) VALUES (?, ?)",
        (key, summary),
    )
    conn.commit()
    return summary

Related

py ai callpy ai batchpy web retriespy async timeoutspy quality logging

Discussion

  • Be the first to comment on this lesson.