Demystifying evals for AI agents
Start with a task and success criteria
Read the overview of agent evaluations, then write one test case with explicit inputs and success criteria.
Start with: Demystifying evals for AI agents
RedLectern topic
Evaluating AI systems
Evaluate complete RAG and agent applications · tool actions · regressions · user outcomes.
For: Application builders
After this page you can: Test application behavior beyond the underlying model's score · record the model, tools, scaffold, and environment for agent trials.
Page reviewed
Part of: Artificial Intelligence › AI evaluation
About Evaluating AI systems
Evaluating an AI application means checking whether the whole system does the job its users need. Start with representative tasks, spell out the inputs and success criteria, and inspect examples as well as summary metrics. For agent trials, record the model, tools, scaffold, and environment; these define the setup being evaluated. A trial record can include outputs, tool calls, and intermediate steps that help explain a pass or failure.
For a RAG application, distinguish retrieval quality from how the answer uses the retrieved context. Check factual correctness, whether citations support the claims, and whether the requesting user is permitted to access the retrieved material. Test tool choices and actions as part of the application behavior. Keep regression cases and compare changes against them. Use direct checks where possible, model-based judging for criteria that need interpretation, and human review when a judgment needs validation.
Start here
Retrieval-Augmented Generation (RAG) Evaluators
Separate the RAG failure points
Use the RAG evaluation guide to distinguish retrieval checks from checks on the final answer.
Start with: Retrieval-Augmented Generation (RAG) Evaluators
Learn the foundations
1.Define tests and choose gradersStart with representative user tasks and write down the input and completion criteria before choosing a score.
Start with representative user tasks and write down the input and completion criteria before choosing a score. Task success rate is the share of trials that meet those criteria. Repeated trials can show whether variable behavior is consistent. Combine direct checks, such as required fields or tool-call parameters, with model-based judging for qualities that need interpretation. Treat an LLM-as-judge score as one evaluator signal and compare it with human ratings on examples from your use case. For an agent evaluation, identify the model, tools, scaffold, and environment behind the trial so readers can interpret the result.
- Demystifying evals for AI agents
Defines a task by its inputs and success criteria, and a trial as an attempt at a task.
- Evaluate a judge model
Describes comparing model-based evaluation scores with human ratings for a use case.
LLM-as-judge
Using a language model to assess outputs against evaluation criteria. For agent trials, report the judged system's model, tools, scaffold, and environment alongside the result.
Task success rate
The share of trials meeting explicitly defined completion criteria.
Try it
Design a test for an appointment-booking agent. State its input, a completion criterion, and what model, tools, scaffold, and environment you would record for the trial.
Example: Input: request a 30-minute appointment with Lee next Tuesday afternoon. Completion: a calendar event exists at an agreed time with the right attendees and duration. Record the exact model identifier, calendar tools, orchestration and instructions, and an isolated calendar environment.
2.Evaluate each part of a RAG applicationA RAG evaluation should distinguish the retrieval step from the response the application produces.
A RAG evaluation should distinguish the retrieval step from the response the application produces. Check whether retrieved passages are relevant to the query, then check whether the answer is grounded in those passages, correct, and complete for the task. Check citations against the passages they point to, and test that retrieval respects the requesting user's permissions. These checks expose different failure points: retrieval quality, answer faithfulness, factual correctness, citation support, and permissions. Keep them visible as separate signals so a good answer score cannot conceal an access-control or retrieval problem.
- RAGAS: Automated Evaluation of Retrieval Augmented Generation
Frames RAG evaluation around relevant context retrieval, faithful use of passages, and generation quality.
- Retrieval augmented generation and indexes
Covers citation metadata and document-level access control in retrieval.
Try it
A RAG answer is factually correct, but its cited passage does not support one claim. Name two separate checks that would catch this and what each checks.
Check factual correctness against a trusted reference to assess whether the claim is true. Check citation support by verifying that the cited passage actually entails or supports that claim.
3.Test agent decisions and tool actionsFor an agent evaluation, inspect the path to the result as well as the final response.
For an agent evaluation, inspect the path to the result as well as the final response. A trial record can include outputs, tool calls, and intermediate results; graders can check distinct parts of the performance. Test whether the agent selected an appropriate tool, supplied correct parameters, respected constraints, and left the environment in the intended state. Include cases where a tool call fails or returns unexpected information, since recovery is part of the behavior users experience. When reporting a result, identify the model, tools, scaffold, and environment that produced it; these define the system and conditions being evaluated.
- Built-in Evaluators Reference
Includes tool-input evaluation for parameters such as grounding, format, and completeness.
- Evaluation types
Shows an agent evaluation using expected tool calls and a heuristic evaluator.
Try it
Design a test for an agent that must cancel a meeting only after the user confirms which meeting. State the input, required behavior, failure to flag, and the model, tools, scaffold, and environment to record.
Input: “Cancel my meeting with Lee.” Required behavior: ask which meeting, then wait for confirmation. Flag a calendar cancellation call made before confirmation. Record the exact model identifier, calendar tools, orchestration and instructions, and an isolated calendar environment.
4.Track regressions and user outcomesKeep a curated set of representative cases and rerun it when a change could affect application behavior.
Keep a curated set of representative cases and rerun it when a change could affect application behavior. Compare versions case by case as well as across aggregate metrics; a regression can appear in a small group even when the overall score barely moves. Historical production examples can add realistic inputs that a hand-built dataset missed, while production monitoring can surface new failures for later testing. Tie the evaluation to user outcomes: define what a completed task means in the product, and review examples where the score and the user's result disagree. For agent comparisons, record the model, tools, scaffold, and environment for each version so the results have interpretable conditions.
- Develop a RAG Solution: End-to-End Evaluation Phase
Discusses multiple response metrics and notes that responses can vary for the same prompt.
Try it
A new release improves the average evaluation score, but two previously successful cases now fail. What should you inspect before deciding whether to ship, and what setup details should accompany any agent comparison?
Inspect both failed cases and compare their traces and outputs with the prior version. Check which criteria regressed and whether the failures affect a user task or a required safety, correctness, or permission constraint. For agent comparisons, record the model, tools, scaffold, and environment for both versions.
Latest
No recent updates are available.
Books
- AI Engineering: Building Applications with Foundation Models
AI Engineering: Building Applications with Foundation Models
Chip Huyen · 2025
Get a framework for adapting foundation models to applications, with guidance on evaluating applications and building and evaluating RAG and agents.