Temperature top-p and top-k

Three controls, three jobs

A language model assigns scores, often called logits, to possible next tokens. A probability distribution turns those scores into chances of selecting each token. Sampling means choosing a token using those chances. Temperature changes the distribution: increasing it makes the probabilities more even; decreasing it makes them more concentrated around the higher-probability tokens.

Top-k and top-p limit the candidate set. Top-k retains a fixed number of the most likely tokens. Top-p, also called nucleus sampling, retains the smallest set whose probabilities add up to at least its chosen threshold. Because the probabilities can differ from one generation step to another, the number of tokens that meet a top-p threshold can also differ.

How the controls shape a generation step

First, the model produces scores for possible next tokens. Temperature scales those scores as they are turned into probabilities: a higher temperature spreads probability more evenly, and a lower one concentrates it. The probability distribution is the starting point for the selection controls.

Next, top-k can retain a fixed number of leading candidates, or top-p can retain enough leading candidates to reach a cumulative probability threshold. A generator can combine nucleus sampling with top-k and temperature sampling. In the described sampling procedure, a token is then chosen at random from the retained set.

The selected token becomes part of the sequence, and the model determines the next token from the updated sequence. The same general selection process can then be applied again. The key distinction is that temperature changes probabilities, while top-k and top-p restrict which candidates remain available.

Common mix-ups

Top-p does not mean “keep this percentage of the tokens.” Its threshold refers to cumulative probability: a top-p value describes how much probability the retained candidates must cover. Top-k, by contrast, specifies how many of the highest-probability candidates to retain.

Temperature is not a candidate count or a cumulative probability threshold. It changes the distribution used to assess candidate probabilities. Since top-p uses those probabilities to decide how many candidates meet its threshold, changing temperature can also change the top-p set.

A setting such as top-k 20 means a candidate count; a top-p threshold such as 0.9 means a cumulative probability target. These examples describe what the controls specify, not the final token or the quality of the generated answer.

Check your understanding

Test the distinction with two descriptions: “keep the 20 most likely tokens” names top-k, because it fixes the number of candidates. “Keep the smallest group whose probabilities add up to at least 0.9” names top-p, because it sets a cumulative probability threshold. In the second case, the number of retained tokens depends on the distribution.

Now explain temperature in one sentence without using “number of tokens” or “probability threshold.” A good answer says that temperature changes how concentrated or even the candidate probabilities are. If you can also explain why a top-p set may contain different numbers of tokens at different steps, you have the central distinction.