Observability primer
Start with an observable request
Read the OpenTelemetry primer, then sketch the signals and request path you need to investigate one AI application failure.
Start with: Observability primer
RedLectern topic
AI observability
Monitor quality · failures · usage · costs · changing behavior · operational evidence.
For: Builders · operators.
After this page you can: Detect and investigate an AI application's regression.
Page reviewed
Part of: Artificial Intelligence › AI deployment
About AI observability
AI observability is the practice of collecting enough evidence about an AI application to notice when it changes and investigate why. A useful view connects the request and response to the steps that produced them, while tracking service behavior such as latency and errors alongside application quality. Traces, metrics and logs provide different views of system activity; together, they help operators follow a request and examine what happened at each step.
For language-model applications, inspect inputs and outputs where appropriate, retrieval or tool steps, evaluation results, user feedback, usage and cost. Keep those signals tied to the application version and request context you need for diagnosis, while handling sensitive data deliberately. A rising error rate may point to a service failure; a quality change with steady service metrics may call for examining outputs, evaluation results or user feedback. Observability helps build a case for a regression and its likely cause; it does not by itself prove a cause. Start with the evidence you can collect reliably, then add checks that answer a real operational question.
Start here
LLM Observability for AI Agents and Applications
Investigate a language-model trace
See how traces and evaluations can help locate application problems across a request.
Start with: LLM Observability for AI Agents and Applications
Learn the foundations
1.Operations across the model lifecycleMLOps covers practices for developing, deploying and maintaining machine-learning systems.
MLOps covers practices for developing, deploying and maintaining machine-learning systems. LLMOps applies operational practices to the specific challenges of language-model applications. The terms overlap: an application may use familiar deployment and monitoring practices while needing additional evidence about generated outputs, prompt or retrieval steps, and user-facing quality. The useful distinction is the system being operated and the signals needed to understand it, not a strict boundary between two separate disciplines.
A model's observed behavior or performance can change over time. Treat that change as a cue to investigate: compare relevant quality measures and operational signals with an appropriate baseline, then inspect examples and the request path. Drift alone does not identify its cause or say whether the change matters to users.
- Practitioners Guide to Machine Learning Operations (MLOps)
Introduces the MLOps lifecycle, deployment and predictive-performance monitoring.
- Monitoring
Discusses production monitoring, logging and data changes as evidence for investigation.
MLOps
Practices for developing, deploying and maintaining machine-learning systems.
LLMOps
Operational practices specific to language-model applications.
Model drift
A change in a model's observed behavior or performance over time.
Try it
A language-model application has steady request volume but user reports say answers have become less relevant. Name one operational signal and one quality signal to inspect, and say how you would compare them.
Inspect request traces or logs for changes in the request path, and compare a relevance evaluation or user feedback with an earlier baseline. Then review affected examples to see whether the change aligns with a particular application step.
2.Follow requests through signals and tracesObservability depends on signals emitted by an instrumented application.
Observability depends on signals emitted by an instrumented application. Traces show a request's path through the application; metrics capture measurements at runtime; logs record events. These views answer different questions: a metric may reveal a rising latency pattern, while a trace can help locate the slow step and a log can provide event detail. Link the evidence with consistent request context so an operator can move from an observed symptom to the relevant execution.
For an AI application, include the meaningful stages in its request path, such as model calls or retrieval when present. Choose what to record with care: operational evidence should help explain behavior, and captured inputs or outputs may contain sensitive information. A trace is useful only to the extent that instrumentation captures the steps needed for the investigation.
- LLM Tracing and Observability With Arize Phoenix
Shows how tracing can expose execution steps and help investigate latency or application problems.
Try it
A user reports one unusually slow AI response. Write down which signal you would use first to find the slow step and what other signal could help explain the event.
Start with the request trace to find which step took the time. Use a metric to check whether the latency is part of a wider pattern, and consult logs for event details around the affected request.
3.Connect quality checks to operational evidenceOperational health and answer quality are related but distinct.
Operational health and answer quality are related but distinct. Request counts, latency and errors describe service behavior; evaluation results and user feedback can help assess whether outputs meet the application's expectations. A healthy service can still produce a quality regression, and a quality score by itself may not explain which step changed. Connect quality checks to traces or other request-level evidence so you can examine examples associated with a change.
When investigating, compare like with like: use a relevant baseline, identify the affected application version or time period, and inspect representative failures. For a retrieval-based workflow, an evaluation can distinguish problems with retrieved material from problems in the final answer. Treat evaluator results as evidence to inspect alongside examples and feedback, not as a complete account of user impact or cause.
- Agent Evaluation
Explains evaluating individual agent components and responses.
Try it
An evaluation score drops after an application change, but latency and error rates remain steady. Name two pieces of evidence to inspect before deciding what caused the drop.
Compare the affected examples and evaluation results before and after the change, then inspect their traces for a changed request step. User feedback or a relevant reference answer can help check whether the measured drop reflects a real quality problem.
Latest
No recent updates are available.
Books
AI Engineering: Building Applications with Foundation Models
Chip Huyen · 2025
Learn how to evaluate foundation-model applications and reason about production latency and cost. Apply those ideas to a real application while investigating a regression.