RedLectern topic
Inference services · batching · caching · throughput · latency · scaling · serving engines.
For: Developers · operators.
After this page you can: Measure and improve an inference service against workload requirements.
Page reviewed
Part of: Artificial Intelligence › AI deployment
Model serving is the operational layer that accepts inference requests and returns model outputs. In a typical server architecture, requests arrive through an API, go to a per-model scheduler, may be batched, and then reach a model backend. The right performance target depends on the workload: a service that streams text needs a clear first-output measure, while a service that returns one result per request may focus on whole-request latency and request throughput.
Start by measuring a representative workload before changing the serving configuration. State where each timer starts and stops, and record request latency separately from time to first token for streamed output. Also record the workload and load conditions, since request throughput and output-token throughput describe different amounts of work. Then test one change at a time—such as batching delay, concurrency, model instances, or caching—and compare the resulting latency distribution and throughput against the service requirements.
Follow a deployment and measurement workflow: check compatibility, run a baseline benchmark, then tune configuration against your use case.
Start with: Deploying your trained model using Triton
An inference service receives requests, routes them to a model scheduler, and returns results from a model backend. For streamed language-model output, the first visible output and the complete request have different timing boundaries, so name the event measured by each latency metric. As checked on 2026-10-09, vLLM documentation lists time-to-first-token and inter-token latency as separate metrics. Treat these as measurements of a particular service and workload, not as interchangeable measures of speed.
Lists request, queue, token, and cache metrics exposed by the serving engine.
Model serving
Operating a model behind an interface that accepts inference requests and returns results.
vLLM
As checked on 2026-10-09, vLLM is an inference and serving engine for language models.
Continuous batching
Scheduling that lets new inference requests join work already being processed, rather than waiting for an entire fixed batch to finish.
Latency
Elapsed time between a request and a specified response event.
Time to first token
Elapsed time from sending a request until the first streamed output is received; state the measurement boundaries.
Tokens per second
A rate of token processing or generation, interpreted under the stated workload, measurement interval, and token-counting method.
Throughput
The amount of work completed per unit of time, such as final responses or output tokens during a benchmark.
For a streamed response, name two different latency measurements and state the event that ends each timer.
Time to first token ends when the client receives the first streamed output. End-to-end request latency ends when it receives the final response.
Choose metrics that match what the service delivers. For streamed generation, time to first token measures the interval from sending a request until its first response arrives; request latency runs until the final response. Output-token throughput counts generated tokens over the benchmark duration, while request throughput counts final responses over that duration. Record input and output sequence lengths as well, so a change in the workload is not mistaken for a serving improvement. Compare measurements under representative request patterns and load, and examine latency percentiles alongside throughput rather than relying on a single average.
Shows how to establish a baseline and assess configuration changes for a use case.
A streaming benchmark reports first-output time, final-response time, generated-token count, and total benchmark duration. Which two rates can you calculate, and what does each count?
Request throughput is the number of final responses divided by benchmark duration. Output-token throughput is the number of generated tokens divided by benchmark duration.
Dynamic batching combines inference requests into batches, and a scheduler may wait briefly so more requests can join. That wait can trade additional latency for throughput, so test it against the service's latency budget. Caching is useful when requests repeat: a response cache can reuse a result for a matching request rather than executing the model again. Measure cache hits and misses as well as request latency, because cache lookup and management add work, and workloads with mostly unique requests may not benefit. Compare one configuration change at a time with the same workload and load conditions.
Describes cache hits, misses, and when repeated requests may benefit from reusing results.
Documents request, execution, batch, and cache statistics for interpreting service behavior.
Your workload has many repeated, identical requests. Name one batching setting to test, one cache measure to track, and the latency measure you would protect.
Test a small scheduler batch delay, track cache hit and miss counts, and protect the request-latency percentile required by the service. Compare the result against a baseline using the same request workload.
No recent updates are available.
Chip Huyen · 2022
A systems perspective on production ML, including deployment, prediction services, monitoring, and an ML platform that serves across use cases. The 2022 edition predates today’s LLM-serving stack, so use it for fundamentals rather than current engine instructions.