Calling an API

GET and POST from Python, reading the response, and the timeout you must pass because the default is to wait for ever.

Syntaxhttpx.get(url, params={"page": 2}, headers={...}, timeout=10.0).json()

Calling an HTTP API is three decisions: what goes in the URL, what goes in the body, and how long you are willing to wait. httpx and requests are the two libraries people reach for, and the standard library's urllib.request does the same job without a dependency.

Query parameters and bodies

httpx.get(url, params={"status": "open"}, timeout=10.0) and requests.get(url, params=..., timeout=10) both take a mapping and build the query string for you; the longhand is urllib.parse.urlencode(query, doseq=False, safe='', ...) joined to the URL with a ?. Likewise json=payload is shorthand for two steps you can write yourself: json.dumps(payload).encode() as the body, plus a Content-Type: application/json header. They are the same request. With urllib.request.Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None) you do that by hand; method defaults to GET when data is None and POST otherwise, and if you supply data without a Content-Type, urllib adds application/x-www-form-urlencoded — which is wrong for JSON, so set the header.

Reading the response, and the timeout

urlopen returns an http.client.HTTPResponse: .status (added in 3.9), .headers, .read(). A 4xx or 5xx raises urllib.error.HTTPError, which is both an exception and a readable response, so the error body is still available through .read(). In the libraries, resp.json() decodes and resp.raise_for_status() turns a bad status into an exception.

requests applies no timeout unless you pass one; urlopen without a timeout uses the global socket default, which is None. Both mean a hung server hangs your process. httpx is the exception, raising TimeoutException after 5 seconds of network inactivity by default.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Pulling a report from a third-party API into a script.
  • Posting a JSON payload to an internal service and acting on the id it returns.
  • Replacing a hand-built query string with params= and getting the escaping right.
  • Stopping a job that hangs for hours because one upstream call never answered.

More examples

The real call, with both libraries

What the runnable example stands in for. Note the tuple form of requests' timeout: the first number caps the TCP connect, the second caps how long the server may go quiet mid-response — a single float applies to both.

Example · python
import httpx, requests

# httpx: timeout defaults to 5s of inactivity, but say it out loud anyway.
r = httpx.get("https://api.example.com/v1/orders",
               params={"status": "open", "limit": 50},
               headers={"Accept": "application/json"},
               timeout=10.0)
r.raise_for_status()           # 4xx/5xx becomes httpx.HTTPStatusError
orders = r.json()

# requests: timeout is (connect, read) when you want them separate.
r = requests.post("https://api.example.com/v1/orders",
                  json={"sku": "A-1", "qty": 2},   # sets Content-Type for you
                  timeout=(3.05, 27))
r.raise_for_status()           # 4xx/5xx becomes requests.HTTPError
print(r.status_code, r.json()["id"])

The same POST with no dependencies

Use this in a script, a Lambda, or anywhere adding a dependency is more trouble than it is worth. It also shows the exception split worth knowing: HTTPError means the server answered and said no, URLError means you never got an answer at all.

Example · python
import json
import urllib.error
import urllib.request

body = json.dumps({"sku": "A-1", "qty": 2}).encode()
req = urllib.request.Request(
    "https://api.example.com/v1/orders",
    data=body,                                        # data present => method POST
    headers={"Content-Type": "application/json"},     # or urllib guesses form-encoded
    method="POST",                                    # state it anyway, for the reader
)
try:
    with urllib.request.urlopen(req, timeout=10) as resp:
        print(resp.status, json.loads(resp.read()))
except urllib.error.HTTPError as exc:
    # HTTPError IS the response: the server's error body is still readable.
    print("failed", exc.code, exc.read().decode()[:200])
except urllib.error.URLError as exc:
    print("could not connect:", exc.reason)           # DNS, TLS, refused, timeout

One client for many calls

Reach for a client or session as soon as you make more than one request to the same host. Creating a fresh client per call pays for a new TLS handshake every time, and it is where a forgotten timeout usually sneaks in.

Example · python
import httpx

# A client keeps the TCP and TLS connection open between calls and carries
# base_url, headers and timeout so no individual call can forget them.
with httpx.Client(base_url="https://api.example.com/v1",
                  headers={"Accept": "application/json"},
                  timeout=10.0) as client:
    for page in (1, 2, 3):
        r = client.get("/orders", params={"page": page})
        r.raise_for_status()
        print(page, len(r.json()["items"]))
# requests spells the same thing requests.Session(), with s.headers.update(...)
# and the timeout passed per call rather than held on the session.

Related

py web httppy web authpy web retriespy web paginationpy data json

Discussion

  • Be the first to comment on this lesson.