Offline Operation
I'm new to local LLMs
Start with the offline guide: learn what needs to be downloaded, then confirm which features run locally.
Start with: Offline Operation
RedLectern topic
Local LLMs
Obtain and run language models locally · verify operation · troubleshoot common failures.
For: Practical beginners · developers
After this page you can: Run a suitable local model and verify where inference happens.
Page reviewed
Part of: Artificial Intelligence › Local and on-device AI
About Local LLMs
A local large language model (LLM) is one whose inference runs on a device you control. To get started, choose a model file that fits your machine, load it with a compatible runtime, and try a small task. A model card can help you check the model’s intended uses and limitations before downloading it.
Then verify the route your request took. Check that the selected model is marked local and that it appears in the runtime’s loaded-model list; a reply by itself does not show where inference happened. Trace the data path for a test request: the prompt goes to the selected model, inference runs on the selected host, and a response returns. A cloud model choice sends inference to a hosted service; a remote model can run on another device. Check the model choice for each request, and remember that model search, downloads, and app updates can make separate network requests. “Local” describes where inference runs for the selected request; it does not, by itself, establish how every connected feature handles data. The LM Studio offline guide and model selection guide explain these distinctions in documentation checked 2026-10-09.
Start here
Choose a Cloud, Local, or Remote Model
Check where a request runs
Compare local, remote, and cloud model choices, then check the selected model for a test request.
Start with: Choose a Cloud, Local, or Remote Model
Learn the foundations
1.Choose a model that fitsStart with the model card: check the intended use, limitations, evaluation information, and license, then confirm that your chosen runtime supports the model f…
Start with the model card: check the intended use, limitations, evaluation information, and license, then confirm that your chosen runtime supports the model files. For LM Studio documentation checked 2026-10-09, a local model must be downloaded and fit the available hardware resources. If you compare quantized files, record the quantization named for each file; quantization reduces weight precision and can introduce accuracy loss.
Plan for working memory, not just the model file on disk. Set the assumed weight quantization and context length when estimating capacity, and account for the KV cache and runtime settings. The estimate is a planning aid; actual performance depends on your machine and configuration.
- Load a model
Describes setting context length and GPU offload, and estimating memory use before loading.
- Quantize
Describes how quantization changes weight precision, file size, and potential accuracy.
Try it
Open a model card and a model file listing. What intended use, license, and quantization would you record before trying the model?
Record the model file and its named quantization, the license shown in the model card metadata, and the card’s stated intended use.
2.Run a small local testUse an interface that makes the selected model and its location visible.
Use an interface that makes the selected model and its location visible. In LM Studio documentation checked 2026-10-09, a local model is downloaded, selected as Local, and run on-device if it fits the available hardware. Start with a short, ordinary text task so you can distinguish a successful load from a useful answer without introducing extra setup.
If you develop against a local server, confirm that your client is talking to the local endpoint and that the selected model is the one you loaded. A local server can accept requests over localhost; serving on a wider network changes who can reach the server. Keep the first test simple: select the local model, ask one question, and note the chosen model and endpoint.
- LM Studio as a Local LLM API Server
Shows the local server option and its API interfaces.
- Choose a Cloud, Local, or Remote Model
Identifies which model choice runs on-device and which choices handle inference elsewhere.
Try it
Run one short test with a selected local model. Record the model choice, whether it appears in the loaded-model list, and the server address if you used an API.
A useful check records a Local selection, a loaded model, and a localhost address for a local-server request. If the selection says remote or cloud, inference is elsewhere.
3.Verify the data path and troubleshootTreat a generated reply as one clue, not proof of local execution.
Treat a generated reply as one clue, not proof of local execution. For LM Studio documentation checked 2026-10-09, the app distinguishes local models from remote models that run on another device and cloud models that use hosted inference. Confirm the selected model’s location and, if using an API, check that your request reached your own local server. This verifies the inference route for that test, not every feature in the app.
If a model will not load, check that it was downloaded and that it fits available hardware resources. If memory is tight or loading is slow, review the context length, GPU offload, and memory estimate using the same quantization assumption as the model file. If you get no response, check that the model is loaded and that the client is reaching the intended server. LM Studio’s documentation also lists which models are loaded and provides server status controls.
- lms CLI
Explains how to inspect models loaded into memory.
- lms server start
Explains local server binding and how to check server status.
- Load a model
Covers load settings and resource estimates that can help investigate memory problems.
Try it
A reply appeared, but you are unsure where inference ran. Which two settings or status indicators would you inspect to verify the route?
Check whether the selected model is Local, Remote, or Cloud, then confirm the expected local model is loaded and the request reached the local server if you used one.
Latest
No recent updates are available.
Books
Hands-On Large Language Models: Language Understanding and Generation
Jay Alammar and Maarten Grootendorst · 2024
Its illustrated explanations and companion code offer a practical introduction to LLM concepts and workflows. The examples were mainly tested in Colab, so check current runtime documentation for local setup.
Build a Large Language Model (From Scratch)
Sebastian Raschka · 2024
Implement a small GPT-style model and load pretrained weights to understand what happens inside an LLM. The book teaches model-building methods, not current runtime setup.
AI Engineering: Building Applications with Foundation Models
Chip Huyen · 2024
A broader guide to foundation-model application design, including inference optimization. It provides engineering context rather than a local-runtime walkthrough.