Latency & Cost
Keep AI features fast and affordable at scale.
In production, latency (how long a response takes) and cost (tokens x price) make or break the experience. Both grow with the number of tokens.
Levers to pull
- Smaller models for simple steps.
- Shorter prompts — trim history and context to what matters.
- Lower max_tokens for concise outputs.
- Streaming to improve perceived speed.
- Caching for repeated work.
- Parallelism — run independent calls at the same time.
Example
{
"before": {"model": "llm-large", "input_tokens": 4000, "latency_ms": 5200},
"after": {"model": "llm-medium", "input_tokens": 1200, "latency_ms": 1400, "note": "trimmed context, smaller model"}
}When to use it
- A real-time autocomplete feature caps max_tokens at 40 and uses a small model to keep median latency under 300ms.
- A nightly batch-summarisation job uses the largest model without a time constraint because it runs offline and quality matters more than speed.
- A high-traffic chatbot caches responses for popular questions to reduce both API spend and p95 latency by 60%.
More examples
Reduce latency with token cap
Compares latency at two max_tokens settings to show that capping output length reduces time to completion.
import time
from openai import OpenAI
client = OpenAI()
for max_tok in [500, 50]:
start = time.time()
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':'Explain TCP/IP networking.'}],
max_tokens=max_tok
)
elapsed = time.time() - start
print(f'max_tokens={max_tok}: {elapsed:.2f}s, {r.usage.completion_tokens} output tokens')Estimate cost before sending a request
Counts prompt tokens and estimates total cost before sending, helping stay within a per-request budget.
import tiktoken
enc = tiktoken.encoding_for_model('gpt-4o-mini')
INPUT_PRICE = 0.15 / 1_000_000 # USD per token
OUTPUT_PRICE = 0.60 / 1_000_000
def estimate_cost(prompt, expected_output_tokens=200):
input_tokens = len(enc.encode(prompt))
cost = input_tokens * INPUT_PRICE + expected_output_tokens * OUTPUT_PRICE
return input_tokens, cost
prompt = 'Explain the concept of recursion with an example.'
input_tok, estimated = estimate_cost(prompt)
print(f'Input tokens: {input_tok}, Estimated cost: ${estimated:.6f}')Simple in-process response cache
Caches responses by prompt hash so identical requests are served instantly without an API call.
from openai import OpenAI
import hashlib
client = OpenAI()
cache = {}
def cached_chat(prompt):
key = hashlib.md5(prompt.encode()).hexdigest()
if key in cache:
print('[cache hit]')
return cache[key]
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':prompt}]
)
reply = r.choices[0].message.content
cache[key] = reply
return reply
print(cached_chat('What is the speed of light?'))
print(cached_chat('What is the speed of light?')) # served from cache
Discussion