Caching
Reuse results for repeated inputs to cut cost and latency.
Many requests repeat. Caching stores past results and serves them instantly instead of paying for the model again.
Types of caching
- Exact-match cache — same input, return the stored output. Simple and reliable for deterministic (temperature 0) tasks.
- Semantic cache — reuse an answer when a new question is very similar (measured by embeddings) to a previous one.
- Prompt caching — some APIs let you cache a long, shared prefix (like a big system prompt) so you are not billed to re-read it every call.
Example
const key = hash(model + JSON.stringify(messages));
const cached = await cache.get(key);
if (cached) return cached; // instant, free
const res = await chat({ model, messages, temperature: 0 });
await cache.set(key, res, { ttlSeconds: 3600 });
return res;When to use it
- A documentation chatbot caches the embedding of its 10k-token system prompt using prompt caching to avoid re-charging for it on every user query.
- A content pipeline caches the LLM's analysis of a stable reference document so only newly submitted user text incurs generation cost.
- A high-volume classification service caches common one-liner inputs in Redis so repeated labels are returned in under 1ms without an API call.
More examples
Redis-backed response cache
Stores LLM responses in Redis with a TTL so repeated identical prompts are served from cache.
import hashlib
import redis
from openai import OpenAI
client = OpenAI()
r = redis.Redis(host='localhost', port=6379, decode_responses=True)
TTL = 3600 # cache for 1 hour
def cached_completion(prompt):
key = 'llm:' + hashlib.sha256(prompt.encode()).hexdigest()
cached = r.get(key)
if cached:
return cached
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':prompt}]
).choices[0].message.content
r.setex(key, TTL, response)
return response
print(cached_completion('What is the boiling point of water?'))Semantic cache with embeddings
Uses embedding similarity to match near-duplicate prompts to cached responses, not just exact string matches.
import numpy as np
from openai import OpenAI
client = OpenAI()
def embed(text):
return np.array(
client.embeddings.create(model='text-embedding-3-small', input=text).data[0].embedding
)
cosine = lambda a, b: np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
SIM_THRESHOLD = 0.97
cache = [] # list of (prompt_vec, response)
def semantic_cached_chat(prompt):
pv = embed(prompt)
for cached_vec, cached_resp in cache:
if cosine(pv, cached_vec) >= SIM_THRESHOLD:
print('[semantic cache hit]')
return cached_resp
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':prompt}]
).choices[0].message.content
cache.append((pv, r))
return r
print(semantic_cached_chat('What is the speed of light?'))
print(semantic_cached_chat('How fast does light travel?')) # cache hitCache with invalidation by version key
Prefixes cache keys with a version string so bumping PROMPT_VERSION instantly invalidates all cached entries.
import hashlib
from openai import OpenAI
client = OpenAI()
cache = {}
PROMPT_VERSION = 'v2' # bump this to invalidate all cached responses
def versioned_cache_key(prompt):
return PROMPT_VERSION + ':' + hashlib.md5(prompt.encode()).hexdigest()
def get_reply(prompt):
key = versioned_cache_key(prompt)
if key in cache:
return cache[key]
reply = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':prompt}]
).choices[0].message.content
cache[key] = reply
return reply
print(get_reply('Explain caching in one sentence.'))
Discussion