Books

RedLectern topic

Choosing and comparing models

Select models against task quality · cost · latency · privacy · deployment constraints.

For: Builders · buyers · operators.

After this page you can: Run a task-specific comparison and justify a model choice.

Page reviewed

Part of: Artificial Intelligence › Foundation models › AI model landscape

About Choosing and comparing models

Choosing a model starts with the work it must do and the conditions it must meet. Write down the expected inputs and outputs, what counts as a useful result, what errors matter, and any limits on latency, cost, data handling, or deployment. A public benchmark can help you find candidates, but it does not replace testing the task and workflow you intend to run. NIST’s AI Risk Management Framework recommends defining system context and requirements, documenting test data and methods, and assessing performance under conditions similar to deployment.

Build a small, representative set of examples and run each candidate with the same task instructions and conditions. Score quality against explicit criteria, then record latency, usage, errors, and operational constraints alongside the quality results. For variable outputs, repeated trials can reveal whether results are consistent. The final choice should explain which requirements mattered most, what tradeoffs remain, and what evidence would trigger another comparison. Treat the result as a dated decision: model offerings and provider terms change, so revisit it when the task, model, or deployment conditions change.

Start here

route

Start with an evaluation plan

Define a task, a few success criteria, and a representative set of examples before comparing candidates.

Start with: Evaluation best practices

route

Compare candidates on your task

Use the same examples and grading criteria for each candidate, then record quality and operational results.

Start with: Demystifying evals for AI agents

route

Check the deployment conditions

Review provider data controls, current pricing, and deployment requirements before making a decision.

Start with: AI Risk Management Framework Core

Learn the foundations

1.Define the task and constraintsBegin with the actual task and the setting in which the system will be used.

Begin with the actual task and the setting in which the system will be used. Describe the inputs, expected outputs, users, and what a successful result must contain. Separate essential requirements from preferences, then note unacceptable errors and any human review or fallback process. Include operational limits such as response-time targets, data-handling requirements, and where the system is allowed to run.

This definition makes the comparison answer a concrete question: which candidate meets the task’s requirements under the intended conditions? NIST recommends documenting the system context, requirements, test design, data selection, benefits, and costs. Record constraints before looking at candidate scores so that a high score on one dimension does not silently outweigh a requirement that is mandatory for deployment.

Try it

Choose a task you might automate. Write one required output, one unacceptable error, one response-time limit, and one data-handling constraint.

Example: classify incoming support requests into an approved category; an incorrect urgent-case label is unacceptable; results must arrive before the agent opens the ticket; customer text must not be sent to a service whose data terms do not meet the organization’s policy.

2.Build a comparable testCreate examples that resemble the inputs the system will encounter, including ordinary cases and difficult or unusual cases.

Create examples that resemble the inputs the system will encounter, including ordinary cases and difficult or unusual cases. Define what a good answer looks like before scoring candidates. Where possible, use checks with clear outcomes, such as whether a required field is present or a classification matches a verified label. For qualities that need judgment, write specific criteria and review a sample of outputs yourself.

Run candidates on the same examples with the same task instructions and comparable settings. If outputs vary between runs, repeat trials and keep the results rather than relying on one response. For a multi-step system, evaluate the workflow that will actually be deployed, including its tools and environment. OpenAI recommends testing on expected production inputs; Anthropic describes trials, graders, transcripts, and the evaluation harness as parts of a rigorous evaluation.

Try it

Draft three test examples for a model that extracts invoice fields. Include one ordinary invoice, one with a missing field, and one with an ambiguous value. Name one check for each.

Ordinary: verify the extracted invoice number against the source. Missing field: verify the model returns an empty value or explicit missing marker. Ambiguous value: verify it flags the ambiguity instead of choosing between the two candidate dates.

3.Measure quality and operational resultsKeep task quality and operating behavior visible in the same comparison record.

Keep task quality and operating behavior visible in the same comparison record. Quality measures should reflect the task’s success criteria; operational measures can include latency, usage, and errors for each completed task. For variable outputs, record results across repeated runs and inspect examples where candidates disagree. A single aggregate score can hide a failure that matters for a particular input type, so retain slices for important cases and review borderline outputs.

Report how the test was run, which candidates and settings were used, and what the test does not cover. Do not treat a public benchmark as a universal ranking: your examples, criteria, and deployment workflow determine what the results mean for your use. NIST advises documenting test sets, metrics, tools, performance conditions, and limitations, and evaluating systems in conditions similar to deployment.

Try it

Two candidates each answer the same ten test cases. One has higher average quality, but fails two cases your team labels critical. What should your comparison report say?

Report the average quality and the two critical failures separately. State that the higher average does not meet the critical-case requirement, show the examples and test conditions, and record whether a fallback or another candidate is needed.

4.Compare cost and latencyMeasure operating cost and response time using the workload you expect to serve.

Measure operating cost and response time using the workload you expect to serve. Record input and output sizes, any additional model calls or tools, and the time users actually wait; these affect the comparison. Use each provider’s current official pricing page to estimate the workload, then validate the estimate against observed usage where available. This section intentionally gives no price figures because provider pricing changes.

Treat cost and latency as constraints and tradeoffs alongside quality. A candidate that meets the quality bar but exceeds the allowed response time may not fit an interactive task; a lower-cost candidate that fails a critical quality criterion may not be viable either. Anthropic’s evaluation guidance describes tracking latency, token usage, and cost per task across a fixed task set. Check official pricing pages again when the model, request pattern, or provider terms change.

Try it

List three measures you would record for a task with an interactive response-time limit and a monthly usage budget.

Record task quality against the acceptance criteria, response latency for each request, and estimated or observed cost per completed task. Include the tested input and output sizes so the cost estimate matches the workload.

5.Check privacy and deployment fitA model choice also depends on the service and system around it.

A model choice also depends on the service and system around it. Check where inference runs, what data is sent, whether prompts or responses are retained, how they may be used, and what controls or regional options are available. Review the provider terms that apply to the exact product, account, and endpoint; policies can distinguish unpaid from paid services and may set different data-use or retention conditions.

Then check deployment requirements such as supported interfaces, access controls, monitoring, and fallback behavior. If data must remain within a specified boundary, confirm that the full data path meets that requirement, including logs, application state, and any tools or external services. Record the source and date of each constraint check. A decision memo should identify any requirements that remain unverified instead of assuming that a model or deployment label settles them.

Try it

A team says a candidate is suitable because it runs through a locally installed application. Name two checks needed before accepting that privacy claim.

Trace the data path to verify where inference runs and whether requests may go to cloud models inside the application. Check whether telemetry, logs, remote embeddings, or application state retain or transmit prompt data.

Latest

No recent updates are available.

See the sources list

Books

Questions and comments