Evaluating AI systems concept

How to evaluate a RAG system

Reference

How to evaluate a RAG system: test what it retrieves and the answer it produces. Retrieval-augmented generation (RAG) combines a search step with a language model that uses retrieved text to answer a question. Assess retrieval quality, answer faithfulness to the retrieved context, factual correctness, citation support, and permissions as separate checks. Keep representative test cases so you can compare results after changes.

Reviewed 2026-10-09

Treat it as a whole application

A retrieval-augmented generation (RAG) system searches a collection for useful text, then gives retrieved text to a language model to help answer a question. Evaluate the retrieval step and the response it produces: a fluent answer can still rely on poor retrieval, and relevant passages do not guarantee a correct answer. Check retrieval quality, answer faithfulness to the retrieved context, factual correctness, and relevance separately. Check whether each citation supports the claim beside it. For systems handling restricted material, also check that retrieved passages match the user's permissions. Microsoft's RAG evaluation guidance describes distinct response measures, and its secure RAG guide addresses authorized data access.

Build the evaluation in stages

Start with representative user questions, including common requests, difficult cases, and questions the documents cannot answer. Record the expected answer or relevant source passages, and define what counts as a useful result for your system. Next, check retrieval separately: compare the returned passages with the passages that should answer each question. Precision at K measures how many of the top results are relevant; recall at K measures how many relevant results appear among them. Then assess the complete response against its question and retrieved context for relevance, completeness, faithfulness, and factual correctness. Finally, compare results with your acceptance criteria and retain the cases for later runs. Microsoft's retrieval evaluation guide describes retrieval measures and recommends testing positive and negative examples.

Watch for scores that hide failures

A combined average can hide where failures cluster, so inspect individual cases and keep answerable and unanswerable questions distinct. Retrieval and response scores answer different questions: a faithful response can still be wrong if the retrieved text is wrong, and a correct response may not be supported by its cited passage. Automated language-model graders can help review open-ended answers, but treat a grader's score as one measure alongside factual correctness and citation support. Language-model responses can vary between runs, so consider a target range instead of relying on one result. For permission-sensitive systems, include requests from users with different access rights and inspect what evidence each request retrieves. Microsoft's evaluation guidance discusses separate response measures and response variation; its secure RAG guide says grounding data should be limited to what a user is authorized to access.

Check the result before relying on it

Review sample cases from start to finish: confirm that the needed evidence exists, retrieval returns it, and the response answers the question using supported claims. Follow each citation back to its passage and check that the passage supports the specific claim; citation correctness and answer correctness are distinct checks. Include questions with no answer in the collection and check how the system handles missing evidence. For restricted collections, verify that a user cannot retrieve material outside their access. When you change retrieval or response behavior, rerun the same cases and compare failures as well as aggregate scores. Separating these checks helps show where a failure occurred. The RAGAS paper describes evaluation dimensions for retrieval, faithful use of context, and generation quality.

Questions people ask

How do I measure RAG accuracy?

Label which passages should answer each test question, then compare those labels with the retrieved results. Precision at K measures the share of top results that are relevant; recall at K measures how many relevant passages appear in those results. Mean reciprocal rank focuses on where the first relevant result appears.

How can I test RAG answer quality?

Check whether the response addresses the question, covers its important parts, stays supported by retrieved context, and is factually correct. Review citations against the passages they point to: a relevant citation or a grounded-sounding answer alone does not establish that the cited passage supports the specific claim.

Can an LLM judge evaluate a RAG system?

Yes. A language-model grader can assess qualities such as groundedness or relevance using the response and, where applicable, the question and retrieved context. Treat its grade as one measurement, and separately check factual correctness and citation support.

How do I test questions the documents cannot answer?

Include questions for which the collection has no supporting material. Check that retrieval does not present unrelated passages as evidence and that the response handles missing information as expected. Review these cases separately from questions the collection can answer.

How do I check permissions in a RAG system?

Test requests under different user access rights, then inspect the retrieved passages for each one. Confirm that grounding data shown to a user is limited to material that user is authorized to access.

Learn more

Develop a RAG Solution on Azure: Large Language Model End-to-End Evaluation Phase
Explains response measures such as groundedness, completeness, relevance, and correctness, and how to interpret them together.
Develop a RAG Solution on Azure: Information-Retrieval Phase
Introduces retrieval checks including precision at K, recall at K, and mean reciprocal rank.
RAGAS: Automated Evaluation of Retrieval Augmented Generation
Presents evaluation dimensions for retrieval, faithful use of retrieved context, and generation quality.

What changed

Sources

Part of Evaluating AI systems: Define tests and choose graders Evaluate each part of a RAG application Track regressions and user outcomes