AI hardware concept

How much memory does a local LLM need?

Reference

How much RAM for a local LLM? There is no single amount: it depends on the model, how its weights are stored, the context it handles, and where inference runs. Check the full working set, including the KV cache and temporary memory, rather than judging by a model file alone. A model-specific estimate is a more useful starting point than a device label.

Reviewed 2026-10-09

Estimate the whole working set

The memory question is whether the computer can hold the model while it handles the intended input and generates a response. Model weights are only one part of that working set. Context is the text, measured in tokens, the model can consider; a KV cache stores key and value calculations from earlier tokens so they can be reused. Temporary activations and the software running the model also need room, so a model file's size does not by itself tell you whether generation will fit.

RAM is general computer memory; VRAM is memory on a graphics card. A model can use both, depending on how much is placed on the graphics processor and where the KV cache is kept. For an estimate, specify the model, weight quantization, context length, and GPU split. The LM Studio load documentation, checked 2026-10-09, exposes context length and KV-cache placement as load settings.

What changes the estimate

The model's weight count and weight quantization affect how much memory its weights occupy. Quantization represents weights with fewer bits; it can reduce their storage, while higher quantization levels can require more memory. That does not remove the space needed for the context, KV cache, activations, or runtime. A memory estimate should state the weight quantization it assumes, instead of treating every model with the same parameter count as identical.

Context length matters because the KV cache can grow as tokens are processed, and the cache can become a memory bottleneck during long-context generation. GPU offload changes which memory holds model layers; some settings can also place the KV cache in GPU memory or computer RAM. Concurrent requests and other running programs may use additional memory too, so the available headroom matters. The Hugging Face cache guide and LM Studio load documentation, checked 2026-10-09, describe these cache and placement considerations.

A simple rule of thumb

Start with an estimate for the exact model and weight quantization, at the context length you plan to use and with the intended GPU split. Then allow room for the context, KV cache, activations, runtime, and other programs. For a GPU setup, check VRAM and RAM separately: offloading more model layers to the GPU shifts some of the working set, but does not make the rest of the memory requirement disappear.

Treat a model loader's estimate as a starting point tied to its settings, not a guarantee for every workload. If the estimate is close to the available memory, try checking a shorter context or a different GPU split in the estimator before choosing. The LM Studio load documentation, checked 2026-10-09, documents context length and KV-cache placement settings; the Hugging Face guide describes cache memory use and options for moving cache data to the CPU.

Worked examples

For a concrete reference, Ollama's Llama 2 library page, checked 2026-10-09, gives general starting RAM guidance of 8 GB for 7B models, 16 GB for 13B models, and 64 GB for 70B models. The same page says Ollama uses 4-bit weight quantization by default. Treat these as that page's examples for Llama 2, not universal requirements: the context length, KV cache, activations, runtime, and other programs affect the working set. The page states a default context length of 4096 tokens.

Suppose Ollama's Llama 2 example and its 4-bit default are what you want to estimate. Use the page's corresponding starting RAM figure, then check the intended context and the KV cache; the figure alone does not establish that a machine has enough working memory. For a GPU setup, check the chosen layer split and KV-cache placement as well. A model-specific estimate can reflect context and placement settings, while these dated Llama 2 figures provide a separate, clearly scoped reference.

Questions people ask

How much VRAM does an LLM need?

There is no single VRAM amount for every model. It depends on the model, quantization, context length, how many layers are placed on the GPU, and whether the KV cache is placed there. Estimate those settings together.

Can my computer run an LLM?

Check the model's estimated memory for the intended context and GPU split, then compare it with the available RAM and VRAM. A local model needs to fit the hardware resources it will use; changing context or cache placement changes the estimate.

Why can a model need more memory than its weight file?

Generation also uses memory for the context and KV cache. The cache stores calculations from earlier tokens for reuse, and it can take a significant part of memory during long-context generation.

Does running a model locally mean every request stays on my computer?

Not necessarily. Check the data path and where inference runs: a local model runs on-device, while a cloud model is hosted and a remote model runs on another device. An app's model choice determines which case applies.

Learn more

Llama 2 library: memory requirements
It gives dated, model-specific RAM examples and states the default weight quantization and context length.
Load a model
It shows how context length and KV-cache placement are configured for a model load.
Cache strategies
It explains what the KV cache stores, how it grows, and why it can use substantial memory.

What changed

Sources

Part of AI hardware: Processors, memory, and accelerator support Estimate inference memory