Controlling Cost & Latency
The three biggest levers on your LLM bill and response time: model choice, output caps, and caching.
In production, the difference between a feature that ships and one that gets cut is almost always cost and latency, not model quality. The good news: a handful of habits get you most of the way there.
1. Use the smallest model that passes your evals
Bigger models are slower and more expensive per token. Start with a small or mid-tier model, measure it against a real test set (see the evaluation lesson), and only reach for a larger model on the tasks that genuinely need it. A common production pattern is routing: a cheap model handles the easy 80%, and hard cases escalate to a stronger one.
2. Always cap output tokens
Output tokens are usually the most expensive part of a call and the main driver of latency, because the model generates them one at a time. Set a max_tokens that fits the task. An extraction that returns a short JSON object never needs thousands of tokens.
3. Cache the stable prefix
If every request shares a large fixed preamble — a long system prompt, few-shot examples, a retrieved document — prompt caching lets the provider reuse that work instead of reprocessing it each time. Cached input is dramatically cheaper and faster. The rule is simple: put the stable content first and the varying content (the user's question) last, so the cacheable prefix stays byte-for-byte identical across calls.
Where the tokens go
| Lever | Effect on cost | Effect on latency |
|---|---|---|
| Smaller model | Large | Large |
Lower max_tokens | Medium | Large |
| Prompt caching | Large (repeat prefixes) | Medium |
| Trimming history / context | Medium | Medium |
Example
{
"model": "small-fast-model",
"max_tokens": 120,
"system": [
{
"type": "text",
"text": "<long, STABLE instructions + few-shot examples>",
"cache_control": {"type": "ephemeral"}
}
],
"messages": [
{"role": "user", "content": "<the VARYING question goes last>"}
],
"_comment": "Stable prefix cached; output capped; cheap model by default."
}When to use it
- A high-traffic product switches from gpt-4o to gpt-4o-mini for its FAQ classifier, cutting per-request cost by 15x with no measurable accuracy drop.
- A real-time suggestion feature sets max_tokens=30 so the model never produces a long reply, keeping p99 latency under 400ms.
- A batch-processing pipeline uses prompt caching so the 8k-token system prompt is charged only once per cache lifetime instead of on every call.
More examples
Pick cheapest model for the task
Selects a small model and caps output tokens for a simple classification task to minimise cost and latency.
from openai import OpenAI
client = OpenAI()
# Use the smallest model that meets accuracy requirements
MODEL = 'gpt-4o-mini' # 15x cheaper than gpt-4o for simple tasks
r = client.chat.completions.create(
model=MODEL,
messages=[{'role':'user','content':'Classify: "I love this!" — POSITIVE or NEGATIVE?'}],
max_tokens=5 # one-word answer needs very few tokens
)
print(r.choices[0].message.content.strip())
print(f'Tokens used: {r.usage.total_tokens}')Time a request and log cost
Measures wall-clock latency and computes exact dollar cost after each request for ongoing monitoring.
import time
from openai import OpenAI
INPUT_RATE = 0.15 / 1_000_000 # gpt-4o-mini USD per input token
OUTPUT_RATE = 0.60 / 1_000_000 # gpt-4o-mini USD per output token
client = OpenAI()
start = time.time()
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':'Explain recursion briefly.'}],
max_tokens=80
)
elapsed = time.time() - start
u = r.usage
cost = u.prompt_tokens * INPUT_RATE + u.completion_tokens * OUTPUT_RATE
print(f'Latency: {elapsed:.2f}s | Cost: ${cost:.6f} | Tokens: {u.total_tokens}')Cache stable prompts to avoid repeat cost
Caches identical (system, user) pairs in memory so repeated calls cost nothing and return instantly.
import hashlib
from openai import OpenAI
client = OpenAI()
_cache: dict = {}
def cheap_call(system: str, user: str) -> str:
key = hashlib.md5((system + user).encode()).hexdigest()
if key in _cache:
return _cache[key] # free — no API call
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[
{'role': 'system', 'content': system},
{'role': 'user', 'content': user}
],
max_tokens=100
)
reply = r.choices[0].message.content
_cache[key] = reply
return reply
SYS = 'You are a concise tech support agent.'
print(cheap_call(SYS, 'How do I reset my password?'))
print(cheap_call(SYS, 'How do I reset my password?')) # cached
Discussion