Evaluating RAG
Measure retrieval quality and answer quality separately.
RAG has two parts that can each fail, so evaluate them separately.
Retrieval quality
- Did the right chunk make it into the top results? (recall)
- Were the results actually relevant? (precision)
Answer quality
- Faithfulness — does the answer stick to the retrieved context, without invention?
- Relevance — does it actually address the question?
How to test
Build a set of real questions with known correct answers and expected sources. Run them after every change to catch regressions. An LLM can even help grade answers against a reference.
Example
{
"question": "What is the refund window?",
"expected_source": "Refund Policy",
"retrieved_top_source": "Refund Policy",
"retrieval_ok": true,
"answer": "14 days",
"faithful": true
}When to use it
- A team measures retrieval recall by checking whether the correct document appears in the top-5 retrieved chunks for each golden question.
- A QA engineer scores answer faithfulness by asking a second LLM whether the generated answer is fully supported by the retrieved context.
- A product team tracks context relevance separately from answer quality to pinpoint whether failures come from bad retrieval or bad generation.
More examples
Measure retrieval recall at K
Checks whether the correct document appears within the top-3 retrieved results across a golden test set.
golden_set = [
{'query': 'refund policy', 'relevant_id': 'doc-refunds'},
{'query': 'shipping time', 'relevant_id': 'doc-shipping'},
]
def retrieve_top_k(query, k=3):
# Stub: replace with real vector search
return ['doc-refunds', 'doc-faq', 'doc-billing'] if 'refund' in query else ['doc-shipping']
hits = sum(
1 for item in golden_set
if item['relevant_id'] in retrieve_top_k(item['query'], k=3)
)
recall_at_3 = hits / len(golden_set)
print(f'Recall@3: {recall_at_3:.0%}')LLM-as-judge faithfulness check
Uses a second LLM call to judge whether the generated answer is faithful to the retrieved context.
from openai import OpenAI
client = OpenAI()
context = 'Refunds are available within 30 days of purchase.'
question = 'Can I get a refund after 2 weeks?'
answer = 'Yes, refunds are available within 30 days so 2 weeks is fine.'
judge_prompt = (
f'Context: {context}\n'
f'Question: {question}\n'
f'Answer: {answer}\n\n'
'Is the answer fully supported by the context? '
'Reply with YES or NO and a one-sentence reason.'
)
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': judge_prompt}]
)
print(r.choices[0].message.content)Score answer relevance with numeric rating
Asks the LLM to rate how well the answer addresses the question and return a structured numeric score.
import json
from openai import OpenAI
client = OpenAI()
def score_relevance(question, answer):
prompt = (
f'Question: {question}\nAnswer: {answer}\n\n'
'Rate how well the answer addresses the question on a scale of 1-5. '
'Return JSON: {"score": <int>, "reason": "<one sentence>"}'
)
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': prompt}],
response_format={'type': 'json_object'}
)
return json.loads(r.choices[0].message.content)
result = score_relevance('What is the refund window?', 'You can return items within 30 days.')
print(f"Score: {result['score']}/5 — {result['reason']}")
Discussion