Paging Through a Large API

Cursor against offset paging, and a generator that yields items so the caller never holds the whole result set.

Syntaxfor item in iter_items(client): ... # generator yields; one page is in memory

No API will hand you 400,000 rows in one response. It gives you a page and a way to ask for the next one, and your job is to turn that into something a caller can iterate without knowing it is happening.

Offset against cursor

Offset paging takes ?limit=100&offset=200. It is easy to reason about and jumps to any page, but it is unstable: if rows are inserted or deleted while you are walking, items shift between pages and you will read one twice or miss it entirely. It also degrades, because the database has to count past every skipped row.

Cursor (keyset) paging takes an opaque ?cursor=... that encodes where the last page stopped. It is stable under concurrent writes and stays fast at depth, at the cost of only moving forwards. Prefer it when the API offers both.

Yield, do not collect

Write the walk as a generator and yield each item. The caller writes an ordinary for loop, memory holds one page at a time, and anything that stops early — a break, itertools.islice — stops the requests as well. Inside the loop, yield from page["items"] is shorthand for for item in page["items"]: yield item; identical behaviour, one line.

Knowing when to stop

Stop on what the API documents: a null or absent next cursor, or no next link. Many APIs send RFC 8288 Link headers, which both requests and httpx expose as resp.links. Add a guard for the pathological cases — an empty page and a cursor that repeats itself — because the alternative is an infinite loop that quietly burns your rate limit.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Exporting every customer from a CRM into a local file without exhausting memory.
  • Summing a field across an API you cannot query directly.
  • Taking the first 50 matches from an endpoint that only pages 20 at a time.
  • Explaining why a nightly export has duplicate rows.

More examples

The real cursor loop

The shape to copy. max_pages is not paranoia: a server bug that returns the same cursor forever is a loop that stops only when your rate limit or your bill does.

Example · python
import httpx

def iter_orders(client, page_size=100, max_pages=1000):
    cursor, seen_pages = None, 0
    while True:
        params = {"limit": page_size}
        if cursor is not None:
            params["cursor"] = cursor          # omit the key entirely on page one
        r = client.get("/orders", params=params, timeout=10.0)
        r.raise_for_status()
        page = r.json()
        yield from page["items"]
        cursor = page.get("next_cursor")
        seen_pages += 1
        if not cursor or seen_pages >= max_pages:   # documented end, or the guard
            return

with httpx.Client(base_url="https://api.example.com/v1") as client:
    for order in iter_orders(client):
        print(order["id"])                      # one page in memory, however many rows

Following a Link header instead

Use it whenever the API sends Link headers — GitHub's does. The appeal is that the server owns the next URL completely, so filters and sort order carry across pages without you re-encoding them.

Example · python
import requests

def iter_items(url, session):
    while url:
        r = session.get(url, timeout=10)
        r.raise_for_status()
        yield from r.json()["items"]
        # requests parses RFC 8288 Link headers into resp.links:
        #   Link: <https://api/...?page=2>; rel="next"
        # becomes {"next": {"url": "https://api/...?page=2", "rel": "next"}}
        nxt = r.links.get("next")
        url = nxt["url"] if nxt else None   # absent rel=next means last page

# GitHub, Stripe-style link paging: you never build the next URL yourself.

Offset paging, and the drift it causes

Written out because plenty of internal APIs only offer offsets. If you must use one, sort by something immutable and accept that a snapshot taken during writes is approximate; say so wherever the numbers get reported.

Example · python
def iter_offset(fetch, limit=100):
    offset = 0
    while True:
        rows = fetch(limit=limit, offset=offset)
        if not rows:                  # an empty page is the only reliable end here
            return
        yield from rows
        offset += limit               # NOT len(rows), or a short page shifts the window

# The hazard, concretely: you read rows 0-99, someone inserts a row that sorts
# into position 10, and the row previously at 99 moves to 100. Your next request
# starts at offset 100 and you read it twice. Ordering by an immutable key and
# filtering with `WHERE id > last_id` -- keyset paging -- avoids it entirely.

Related

py web clientpy generatorspy web httppy web retriespy async gather

Discussion

  • Be the first to comment on this lesson.