Retrieve, Augment, Generate
The query-time flow that turns a question into a grounded answer.
At query time, RAG runs three steps: retrieve relevant chunks, augment the prompt with them, and generate the answer.
The augmented prompt tells the model: answer the question using only the context below.
Example
{
"messages": [{
"role": "user",
"content": "Answer using ONLY the context. If it is not there, say you don't know.\n\nCONTEXT:\n- Refunds are available within 14 days.\n- Shipping is free over $50.\n\nQUESTION: How long do I have to get a refund?"
}]
}When to use it
- A customer-service bot runs the full RAG flow for every incoming question, retrieving from the product knowledge base before generating any reply.
- An internal wiki assistant retrieves the three most relevant wiki pages for each employee question and passes them as context to the LLM.
- A medical Q&A tool retrieves current guideline passages at query time, augments the prompt with them, then generates a grounded answer with citations.
More examples
Complete RAG pipeline in one script
End-to-end RAG: index documents, retrieve the best match for a query, augment the prompt, and generate a grounded answer.
import numpy as np
from openai import OpenAI
client = OpenAI()
def embed(text):
return np.array(
client.embeddings.create(model='text-embedding-3-small', input=text).data[0].embedding
)
cosine = lambda a, b: np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
# 1. Index
docs = [
'Free tier: 1GB storage, 100 API calls/day.',
'Pro plan: 100GB storage, unlimited API calls, $29/month.',
'Enterprise: custom storage, SLA, contact sales.',
]
doc_vecs = [embed(d) for d in docs]
# 2. Retrieve
query = 'What do I get on the Pro plan?'
q_vec = embed(query)
best = max(range(len(docs)), key=lambda i: cosine(q_vec, doc_vecs[i]))
context = docs[best]
# 3. Generate
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': f'Context: {context}\n\nQuestion: {query}'}]
)
print(r.choices[0].message.content)Top-k retrieval for richer context
Retrieves the top-3 matching documents and concatenates them as context for a richer augmented prompt.
import numpy as np
from openai import OpenAI
client = OpenAI()
def embed(text):
return np.array(
client.embeddings.create(model='text-embedding-3-small', input=text).data[0].embedding
)
cosine = lambda a, b: np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
docs = [
'Refunds are available within 30 days.',
'Contact support at [email protected].',
'Items must be in original packaging for returns.',
'Shipping takes 5-7 business days.',
]
doc_vecs = [embed(d) for d in docs]
query = 'How do I return a product?'
q_vec = embed(query)
scores = [(i, cosine(q_vec, v)) for i, v in enumerate(doc_vecs)]
top3 = sorted(scores, key=lambda x: x[1], reverse=True)[:3]
context = '\n'.join(docs[i] for i, _ in top3)
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': f'Context:\n{context}\n\nQuestion: {query}'}]
)
print(r.choices[0].message.content)System prompt RAG pattern
Puts the retrieved context in the system prompt and instructs the model to answer only from it.
from openai import OpenAI
client = OpenAI()
retrieved_context = (
'Policy excerpt: Orders over $50 ship free. '
'Express shipping is available for $9.99. '
'International shipping starts at $14.99.'
)
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[
{'role': 'system', 'content':
'You are a helpful assistant. Answer ONLY using the provided context. '
'If the answer is not in the context, say so.\n\n'
f'CONTEXT:\n{retrieved_context}'},
{'role': 'user', 'content': 'How much is express shipping?'}
]
)
print(r.choices[0].message.content)
Discussion