Testing Code That Calls a Model
Push the deterministic logic out of the call so unit tests cover it, assert on shape rather than wording, and keep one real smoke test.
with unittest.mock.patch.object(app, "client", StubClient()): ...The model is the only non-deterministic part of the system, and it is a small part. Everything around it, prompt construction, parsing, validation, retry policy, dispatch, is ordinary code that can be tested the ordinary way. Testing strategy here is mostly an architecture decision: make the untestable part as small as you can.
Split the call out
Write build_prompt(data) -> str and parse_reply(text) -> Result as pure functions, with the API call as a thin layer between them. Those two functions get real unit tests with real assertions and no network. In the tests that must cover the layer itself, inject a stub client with unittest.mock.patch.object or a fixture, and assert on the request you built as well as on what you did with the reply.
Golden files
Store captured real replies as files, one per interesting case, including the ugly ones: fenced JSON, a refusal, a truncated reply, an invented enum value. Your parser runs against the whole directory, so a reply shape that broke production once can never break it silently again.
Assert on shape, not wording
Never assert that a reply equals a sentence. Assert the fields exist, the types are right, the label is in the allowed set, the number is inside a plausible range. These hold across model versions; exact wording does not.
One smoke test, and a cost
Keep a single marked test that makes one real call with a tiny prompt and a small max_tokens, and run it on a schedule rather than on every push. It costs a fraction of a cent and it is the only thing that catches an expired key, a renamed model or a changed response shape.
Example
When to use it
- Getting a model-backed feature under test without paying per test run
- Catching the day a provider change breaks your reply parser
- Writing assertions that survive a model version upgrade
- Noticing an expired API key before a customer does
More examples
Unit tests with the client stubbed out
Covering the layer between the pure functions. The stub records the request, so one test asserts both halves: that the ticket reached the prompt and that the reply was parsed. The second test is the one people skip, and truncation is the failure that reaches production most often.
# tests/test_triage.py
import pytest
from types import SimpleNamespace
from unittest.mock import patch
from app import triage
def reply(text, stop_reason="end_turn"):
return SimpleNamespace(
stop_reason=stop_reason,
content=[SimpleNamespace(type="text", text=text)],
usage=SimpleNamespace(input_tokens=10, output_tokens=5),
)
@pytest.fixture
def stub_client():
client = SimpleNamespace(messages=SimpleNamespace(create=None))
with patch.object(triage, "client", client):
yield client
def test_sends_the_ticket_and_parses_the_label(stub_client):
seen = {}
def create(**kwargs):
seen.update(kwargs)
return reply('{"label": "billing", "urgency": 2}')
stub_client.messages.create = create
assert triage.classify({"subject": "Card declined"}) == ("billing", 2)
assert "Card declined" in seen["messages"][0]["content"]
assert seen["model"] == "claude-opus-5-5"
def test_truncated_reply_is_an_error(stub_client):
stub_client.messages.create = lambda **kw: reply('{"lab', "max_tokens")
with pytest.raises(RuntimeError):
triage.classify({"subject": "x"})A golden-file directory the parser must survive
Every captured reply, replayed on every push, for free. The assertion is deliberately weak: each file must either parse into something valid or raise ValueError. That is the real contract of a defensive parser, and it catches the KeyError or TypeError that would otherwise reach a user. Add a file every time production surprises you.
# tests/golden/fenced.txt, refusal.txt, truncated.txt, bad_enum.txt, ...
import json
from pathlib import Path
import pytest
GOLDEN = sorted((Path(__file__).parent / "golden").glob("*.txt"))
@pytest.mark.parametrize("path", GOLDEN, ids=lambda p: p.stem)
def test_parser_either_parses_or_raises_valueerror(path):
text = path.read_text(encoding="utf-8")
try:
label, urgency = parse_reply(text)
except ValueError:
return # a clean rejection is a pass
assert label in LABELS # never a crash, never a silent wrong value
assert 1 <= urgency <= 5The one test that costs money
Marked, skipped without credentials, excluded from the default run, and asserting only that a real call still returns a usable shape. This is what catches an expired key, a retired model id or a changed response shape, none of which any stubbed test can see. addopts keeps it out of the normal suite so nobody has a reason to delete it.
# tests/test_smoke.py - run nightly: pytest -m smoke
import os, pytest, anthropic
@pytest.mark.smoke
@pytest.mark.skipif(not os.environ.get("ANTHROPIC_API_KEY"),
reason="no credentials in this environment")
def test_one_real_call():
client = anthropic.Anthropic(timeout=30.0, max_retries=1)
reply = client.messages.create(
model="claude-opus-5-5", max_tokens=16, # tiny: fractions of a cent
messages=[{"role": "user", "content": "Reply with the word ok."}],
)
assert reply.stop_reason in {"end_turn", "max_tokens"}
assert any(b.type == "text" and b.text.strip() for b in reply.content)
# In pyproject.toml:
# [tool.pytest.ini_options]
# markers = ["smoke: makes a real, billable API call"]
# addopts = "-m 'not smoke'"
Discussion