Generation
Start with generation settings
Read the generation parameter descriptions, then compare how temperature, top-k, and top-p alter token selection.
Start with: Generation
RedLectern topic
LLM inference and reasoning
Generation controls · inference stages · reasoning methods · compute and speed tradeoffs.
For: Builders · technical learners.
After this page you can: Choose generation settings and explain inference tradeoffs.
Page reviewed
Part of: Artificial Intelligence › Foundation models › Large language models
About LLM inference and reasoning
LLM inference is the process of using a trained language model to produce an output. A request typically starts by processing the supplied prompt, then generates output token by token. Temperature, top-p, and top-k are controls used during token selection; they affect how a token is selected, not whether an answer is factually correct. Parameter names, accepted values, and defaults can vary by system, so check the documentation for the system you use. Hugging Face's generation documentation describes these controls and cache options for that documentation version.
Reasoning methods add a compute choice: a system may spend more inference-time work on a difficult problem, such as generating intermediate text or considering multiple candidate answers. More generated tokens do not by themselves establish better reasoning. Prompt processing and token generation have different performance characteristics, and a KV cache can reuse attention state during generation. This page connects those ideas so you can choose settings for a task and explain the tradeoffs in output variation, speed, and compute. (OpenAI on test-time compute; vLLM on prefill and decode; PMLR study of reasoning effort)
Start here
Context Parallel Deployment
Understand inference stages
Learn what prompt processing and token generation do, then trace where cached attention state is reused.
Start with: Context Parallel Deployment
Learning to reason with LLMs
Explore reasoning compute
Read about test-time compute and practice separating a compute budget from a claim about answer quality.
Start with: Learning to reason with LLMs
Learn the foundations
1.Choose how the next token is selectedA language model produces a probability distribution over possible next tokens.
A language model produces a probability distribution over possible next tokens. Sampling draws from that distribution; temperature changes the concentration of the probabilities, while top-k and top-p restrict which candidates remain eligible. These controls can change how varied or constrained outputs are, but they do not verify whether an answer is true. The meaning of a setting, its accepted values, and its default can vary by API or runtime, so check the documentation for the system you are using. (Hugging Face generation parameters)
Reasoning is also an inference choice. Chain-of-thought prompting asks for intermediate reasoning text, but a visible trace should not be treated as proof of how a model reached its answer. Some systems use reasoning tokens that are not shown as answer text, and their accounting and visibility depend on the system. Test-time compute describes resources used during problem solving. More inference work can change performance, but generated length alone is not a reliable measure of reasoning quality. (OpenAI on reasoning tokens; PMLR study of reasoning effort)
- Reasoning models
Explains reasoning tokens, their visibility, and their relation to output limits.
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
A research study examining why generated length alone can be an unreliable proxy for reasoning effort.
Temperature
A setting that changes how concentrated the probabilities are when selecting a token.
Top-p
Sampling from the smallest set of likely candidates whose probabilities reach a chosen cumulative share.
Top-k
Restricting token sampling to the k highest-probability candidates.
Sampling
Selecting an output token according to a probability distribution.
Chain of thought
Generated intermediate reasoning text that need not reveal the model's actual causes.
Test-time compute
Computation spent while solving a problem rather than training the model.
Reasoning tokens
Tokens used for intermediate reasoning; their accounting and visibility vary by system.
Speculative decoding
A draft process proposes tokens that a target model verifies.
KV cache
Saved attention calculations reused when generating later tokens.
Prefill
The inference stage that processes the supplied prompt.
Decode
The inference stage that generates subsequent output tokens.
Try it
For a response that should be varied but still limited to plausible next tokens, explain what changing temperature does and how top-p or top-k could further restrict candidates.
Temperature reshapes the next-token probabilities. Top-p retains the smallest high-probability set reaching a cumulative probability threshold; top-k keeps only a fixed number of highest-probability candidates.
2.Trace prompt processing and token generationInference has two useful stages to distinguish.
Inference has two useful stages to distinguish. During prefill, the system processes the supplied prompt and computes attention state for its tokens. During decode, it generates output one token at a time, reusing cached key and value state. These stages have different performance characteristics: prompt processing affects time to the first token, while token generation and available cache capacity affect how output is served. The exact serving design is implementation-dependent, so treat this as a mental model and consult the runtime documentation for operational details. (vLLM context parallel deployment; speculative decoding paper)
A KV cache avoids recomputing stored attention state for earlier tokens, but it occupies memory and can constrain serving capacity. Speculative decoding tries a different speed tradeoff: a draft proposes multiple tokens, then the target verifies them. It can improve throughput in suitable settings, but the draft's latency and how well its proposals match the target matter. Compare output and latency under the same workload before deciding whether the optimization helps. (speculative decoding study)
- Decoding Speculative Decoding
Examines draft-model latency and target verification in speculative decoding.
Try it
A request has a long prompt and then produces a long answer. Identify which stage processes the prompt, which generates the answer, and what the KV cache reuses.
Prefill processes the supplied prompt. Decode generates subsequent tokens. The KV cache stores attention key and value state from earlier tokens for reuse during generation.
3.Spend inference compute deliberatelyReasoning methods allocate additional work to solving a prompt.
Reasoning methods allocate additional work to solving a prompt. One approach generates an intermediate reasoning trace; other approaches may use multiple candidate generations or a longer inference budget. These choices can increase latency and token use, so choose them according to task difficulty and the value of additional work. Do not infer quality from answer length alone: a 2026 study reports that raw generation length does not consistently correlate with accuracy in its evaluated settings. (OpenAI on test-time compute; PMLR study)
Treat reasoning text and answer quality as separate things to evaluate. Chain-of-thought text may be intermediate output, but it is not necessarily a faithful record of the causes behind an answer. Reasoning tokens may be hidden or counted differently across systems. Test a method on representative tasks, record latency and token use, and check correctness independently. (OpenAI reasoning documentation; OpenAI chain-of-thought monitorability study)
- Evaluating chain-of-thought monitorability
Discusses limits and evaluation of visible reasoning traces.
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
Provides evidence that output length alone is not a dependable proxy for reasoning effort.
Try it
A team proposes increasing the output-token budget because longer answers seem more reasoned. Give one reason to test this change and two measurements to compare.
A larger budget may allow additional inference work, but length alone does not establish reasoning quality. Compare correctness on representative tasks and latency; also track token use to understand the compute tradeoff.
Latest
No recent updates are available.
Books
Build a Large Language Model (From Scratch)
Sebastian Raschka · 2024
Build a small GPT-style model and follow how its parts produce text. The author’s chapter map includes text generation and sampling.
- Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models
Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models
Daniel Jurafsky and James H. Martin · 2026
A current, author-posted third-edition draft with LLM material. The authors invite readers to use its draft chapters and slides freely.
- AI Engineering: Building Applications with Foundation Models
AI Engineering: Building Applications with Foundation Models
Chip Huyen · 2025
A systems-focused companion for reasoning about inference performance and serving tradeoffs. The author’s book resources include questions about making models faster and cheaper.
- AI Systems Performance Engineering: Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch
AI Systems Performance Engineering: Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch
Chris Fregly · 2025
A performance-focused reference for GPU-based inference, including prefill/decode, KV cache, and speculative decoding.