Why RAG?
Give the model your own data at query time so answers are grounded and current.
LLMs only know their training data and have a knowledge cutoff. They do not know your company's docs, your product, or anything private or recent. Retrieval-Augmented Generation (RAG) fixes this by fetching relevant text and putting it into the prompt.
The core idea
Instead of hoping the model memorised the answer, you retrieve the right source material and ask the model to answer using it. The model becomes a reasoning engine over your data.
Benefits
- Up-to-date and private knowledge without retraining.
- Far fewer hallucinations.
- Answers can cite their sources.
Example
{
"without_rag": {"q": "What is our refund window?", "a": "Typically 30 days (guess)."},
"with_rag": {
"retrieved": "Our policy: refunds within 14 days of purchase.",
"a": "Your refund window is 14 days, per the policy document."
}
}When to use it
- A law firm uses RAG to let the model answer questions about internal case notes without retraining or leaking confidential data.
- A software company builds a RAG-powered docs assistant so the model always answers about the current API version, not the training-data snapshot.
- A hospital uses RAG to ground the model's answers in up-to-date clinical guidelines fetched at query time, preventing stale medical advice.
More examples
RAG vs direct LLM for current data
Demonstrates how injecting a retrieved context block grounds the answer compared to relying on training data alone.
from openai import OpenAI
client = OpenAI()
# Without RAG — model may hallucinate or use stale data
r1 = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'What is the current price of our Pro plan?'}]
)
print('Without RAG:', r1.choices[0].message.content)
# With RAG — inject live context at query time
context = 'Current pricing: Starter $9/mo, Pro $29/mo, Enterprise $99/mo.'
r2 = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': f'Context: {context}\n\nWhat is the current price of the Pro plan?'}]
)
print('With RAG: ', r2.choices[0].message.content)Fetch context from an API then query LLM
Fetches live data from an external source and injects it as context so the model can answer about today's state.
import requests
from openai import OpenAI
client = OpenAI()
def get_weather(city):
# Real apps call a weather API here
return f'Current weather in {city}: 22°C, partly cloudy, humidity 60%.'
city = 'Berlin'
weather_context = get_weather(city)
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content':
f'Weather context: {weather_context}\n\n'
f'Should I bring an umbrella to {city} today?'
}]
)
print(r.choices[0].message.content)Simple in-memory RAG with embeddings
Full minimal RAG: embed a small corpus, retrieve the best match, inject it, then answer the question.
import numpy as np
from openai import OpenAI
client = OpenAI()
def embed(text):
return np.array(
client.embeddings.create(model='text-embedding-3-small', input=text).data[0].embedding
)
cosine = lambda a, b: np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
docs = [
'Refund policy: full refund within 30 days of purchase.',
'Shipping: standard 5-7 days, express 1-2 days.',
'Returns: items must be unworn and in original packaging.',
]
query = 'Can I get my money back?'
q_vec = embed(query)
best_doc = max(docs, key=lambda d: cosine(q_vec, embed(d)))
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': f'Context: {best_doc}\n\n{query}'}]
)
print(r.choices[0].message.content)
Discussion