LLM inference and reasoning concept

What is a KV cache?

Reference

What is a KV cache? It is short-term storage for the key and value vectors that an attention-based language model has calculated for earlier tokens. During generation, the model can reuse these stored values instead of recalculating them each step. This saves repeated computation, while the cache itself takes memory and usually grows as more tokens are processed.

Reviewed 2026-10-09

A record of attention calculations

KV means key-value. In an attention-based language model, keys and values are vectors calculated from tokens and used when the model determines how tokens relate. The cache keeps these calculated vectors for tokens the model has already processed.

Think of it as working notes for the current generation. The cache is not the model’s learned parameters, nor is it the conversation text itself. It stores intermediate attention information that helps the model continue producing tokens without repeating some earlier calculations.

Reuse during generation

Generation commonly has a prompt-processing stage, called prefill, followed by a token-by-token stage, called decode. During prefill, the model processes the prompt and builds attention state for its tokens. During decode, it calculates key and value vectors for the new token, adds them to the cache, and uses cached vectors from earlier tokens when calculating attention.

This is why a KV cache can reduce repeated work as a response is generated. It does not remove the work of processing the growing context: the model still uses prior tokens’ cached values to calculate attention for each new token. The cache is an inference aid, not a new way of choosing the next token.

A working part of inference

A KV cache belongs to inference: the process of using a trained model to produce an output. It holds intermediate attention information for a sequence, while learned parameters are part of the model itself. In the parent topic’s terms, it connects prompt processing, or prefill, with token-by-token generation, or decode.

The cache also sits alongside the context window, which is the limit on how much input and generated text the model can handle in one sequence. The cache can help avoid repeating some calculations within that sequence, but it does not itself set or enlarge that limit. For more on that neighbour, see context windows explained.

Why longer context takes memory

A longer context means more processed tokens whose key and value vectors may need to be kept. So the KV cache can grow as the context grows, and it can take a substantial share of memory during long-context generation. The amount depends on the model and how its attention and cache are implemented; there is no single memory figure that applies to every model.

A context window is the limit on how much input and generated text a model can handle in a sequence. The KV cache is memory used to help calculate attention over processed tokens in that sequence. Caching does not, by itself, make the context window larger. Some attention designs keep only a sliding window of cached tokens, so the cache behavior can differ by model.

Questions people ask

What does KV stand for?

KV stands for key-value: the key and value vectors used in attention calculations. The cache stores those vectors for tokens the model has processed.

Why does context use more VRAM?

As more tokens are processed, the cache may need to hold more key and value vectors. That can make it a substantial part of memory use during long-context generation. The amount varies with the model and cache design.

Does a KV cache make the context window bigger?

No. It stores attention information to reduce repeated computation. A model’s context window is a separate limit; caching alone does not increase it.

Is the KV cache the same as saving a conversation?

No. It holds intermediate attention information for generation. A system may reuse a cache for a shared prompt prefix, but that is a cache reuse feature, not the same thing as saving the conversation text.

Learn more

KV cache strategies
Explains cache strategies, memory tradeoffs, offloading, and prefix-cache reuse.
KV Cache: How It Works
Walks through why key and value vectors can be reused during token-by-token generation.

What changed

Sources

Part of LLM inference and reasoning: Trace prompt processing and token generation