What is RAG?

Retrieval-Augmented Generation (RAG) combines information retrieval with LLM generation. Instead of relying solely on the model's training data, RAG systems retrieve relevant documents from a knowledge base and use them as context for generating responses.

Why RAG Evaluation is Challenging

RAG systems have multiple failure points:

• Retrieval: Did we find the right documents?
• Relevance: Is the retrieved context useful?
• Generation: Did the LLM use the context correctly?
• Grounding: Is the answer supported by retrieved docs?

Each component must be evaluated separately and holistically.

Evaluation Framework

1. Retrieval Quality

Are you retrieving the right documents?

Metrics:

• Precision@K: Of the top K retrieved docs, how munknown are relevant?
• Recall@K: Of all relevant docs, how munknown are in top K?
• MRR (Mean Reciprocal Rank): Position of first relevant document
• NDCG (Normalized Discounted Cumulative Gain): Quality of ranking

Evaluation Method:

2. Context Relevance

Is the retrieved context actually useful for answering the query?

Metrics:

• Context Relevance Score: LLM judges if context helps answer the query (1-5)
• Context Precision: % of retrieved chunks that are relevant

LLM Judge Prompt:

3. Answer Faithfulness

Is the generated answer grounded in the retrieved context, or is it hallucinating?

Metrics:

• Faithfulness Score: % of claims in answer supported by context
• Citation Coverage: Are all facts attributed to sources?

Evaluation Approach:

4. Answer Relevance

Does the answer actually address the user's query?

Example:

5. Answer Correctness

Is the answer factually accurate compared to ground truth?

Methods:

• Exact match: For factual queries (dates, numbers)
• Semantic similarity: Compare to reference answer
• LLM-as-judge: Evaluate correctness holistically

Building a RAG Test Suite

Step 1: Create Query-Answer Pairs

Collect 100-200 representative queries with gold-standard answers:

Step 2: Test Retrieval Separately

Before evaluating end-to-end, isolate retrieval:

Step 3: Evaluate End-to-End

Run full RAG pipeline and check all quality dimensions:

Common Failure Modes

1. Retrieval Failures

2. Context Window Overflow

Retrieved too munknown docs, exceeded LLM context limit.

Solution: Rerank and truncate to most relevant chunks.

3. Answer Hallucination

Solution: Add instruction: "Only use information from the provided context. If unsure, say so."

4. Incomplete Answers

Relevant information was retrieved but not included in answer.

Solution: Improve generation prompt to cover all relevant points from context.

Advanced Techniques

1. Hybrid Search

Combine semantic search with keyword matching:

2. Reranking

After initial retrieval, use a reranker to improve ordering:

• Cross-encoder models (BERT-based)
• LLM-as-reranker
• Learned-to-rank models

3. Query Rewriting

Transform user queries into better search queries:

4. Multi-Hop Retrieval

For complex queries, retrieve iteratively:

1. Retrieve documents answering first part of query
2. Use those results to refine query for second retrieval
3. Combine information from both retrievals

Monitoring Production RAG

Track these metrics continuously:

• Answer rate: % of queries answered vs. "I don't know"
• User feedback: Thumbs up/down on answers
• Retrieval latency: Time to fetch documents
• Generation latency: Time to generate answer
• Context usage: Are retrieved docs actually being used?

Optimization Strategies

Improve Retrieval

• Fine-tune embedding models on your domain
• Increase chunk overlap to avoid splitting related information
• Add metadata filters (date, category, author)
• Use better chunking strategies (semantic, sentence-based)

Improve Generation

• Provide clear instructions about using context
• Include examples of good citations
• Use chain-of-thought to explain reasoning
• Add explicit "Don't hallucinate" instructions

Real-World Example