Supported models and formats
Start with model and runtime compatibility
Read the runtime’s model-format requirements, then check the model’s architecture and chat format before selecting an application.
Start with: Supported models and formats
RedLectern topic
Local AI runtimes
Runtime and application choices · model formats · templates · compatibility · local servers.
For: Local users · developers.
After this page you can: Match a model format and architecture to a compatible runtime.
Page reviewed
Part of: Artificial Intelligence › Local and on-device AI
About Local AI runtimes
A local AI runtime is software that executes a model’s computations. To choose one, start with the model files you have and the task you need to do. Check the model architecture, file format, quantization, and expected chat format against the runtime’s documentation. A file extension alone does not establish compatibility. A chat template, for example, formats conversational messages into input the model expects.
Separate the model from the application and the server interface. An application may make model selection and serving easier, while a lower-level runtime may offer more control over integration. Then check where inference runs. “Local” describes a particular data path, not every feature in an application: cloud models inside local apps may use remote inference, and other features may involve remote embeddings, telemetry, or logs. Check the selected model and enabled features before sending sensitive inputs.
Start here
LM Studio as a Local LLM API Server
Explore local server interfaces
Compare local API options and decide which server interface fits your application.
Start with: LM Studio as a Local LLM API Server
Learn the foundations
1.Choose a runtime and applicationAn inference runtime executes a model; an application may package a runtime with model management, a chat interface, and a server.
An inference runtime executes a model; an application may package a runtime with model management, a chat interface, and a server. As checked on 2026-10-09, llama.cpp describes its purpose as LLM inference in C/C++, Ollama documents both local and cloud model options, LM Studio documents an API server on localhost, and MLX describes itself as an array framework for machine learning on Apple silicon. These projects occupy different parts of a local AI setup, so compare the model support and integration features you need.
Before choosing, verify the exact architecture and format against current runtime documentation, then check whether the application exposes the interface your code needs. The data path matters too: a cloud model inside a local app may use remote inference. Check the selected model and enabled features before using sensitive inputs.
- LM Studio as a Local LLM API Server
Read the documentation for serving local models and available API options.
- Cloud models
Check Ollama’s description of where its cloud models run.
Inference runtime
Software that executes a model's computations.
GGUF
As described in the format documentation checked 2026-10-09, a file format for storing models for inference with GGML and executors based on GGML.
Safetensors
A tensor file format designed to avoid executable object deserialization.
llama.cpp
As described by the project on 2026-10-09, an inference engine implemented primarily in C and C++.
Ollama
As documented on 2026-10-09, software for managing and using models through local and cloud options.
LM Studio
As documented on 2026-10-09, an application for serving local models through a server interface.
MLX
As described by the project on 2026-10-09, an array framework for machine learning on Apple silicon.
Chat template
As documented on 2026-10-09, rules that convert conversations into a tokenizable string in the format a model expects.
Model hub
A service hosting and organizing model files and documentation.
Try it
Choose one runtime or application from the resources. Find its official documentation and name one file format or server interface you would check before using it.
For example, I would check whether the model is GGUF before using llama.cpp, then confirm that the model architecture is supported in the project’s model documentation.
2.Match the model files and chat formatA model download is not just a name: inspect its architecture, file format, quantization, and any tokenizer or template files, then confirm that the intended r…
A model download is not just a name: inspect its architecture, file format, quantization, and any tokenizer or template files, then confirm that the intended runtime supports that combination. The llama.cpp model documentation says its required model format is GGUF, and the GGUF specification describes metadata alongside tensor data. Safetensors serves a different purpose: its project describes tensor storage designed to avoid pickle-style deserialization. One runtime’s support for a format does not establish that another runtime accepts it.
For chat, check the model’s expected message format and the template the runtime will use. Hugging Face’s chat-template documentation says different models expect different chat input formats. If output is malformed or the model ignores roles, inspect the template before assuming the problem lies elsewhere.
- Supported models and formats
Consult the project’s model-format requirements.
- Templates for Chat Models
Read how chat templates convert messages to a model’s expected input format.
- Safetensors
Read the format maintainers’ description of tensor storage.
Try it
Choose a model repository and a runtime. Write down the model architecture, file format, and chat-template source you would check before loading it.
A model card might identify a transformer architecture and provide Safetensors files; a llama.cpp workflow requires checking for GGUF, while the tokenizer or model documentation can identify the expected chat template.
3.Connect through a local serverA local server lets another application send requests to a runtime through an interface such as a REST API.
A local server lets another application send requests to a runtime through an interface such as a REST API. As checked on 2026-10-09, LM Studio documents an API server on localhost and lists REST and compatibility endpoints. The llama.cpp server documentation describes a server with REST APIs and a web UI. Treat those interfaces as integration choices: consult endpoint documentation for supported request fields, and check where the server is reachable.
Where inference runs is one part of the data path. Cloud models inside local apps can route requests to remote inference; Ollama’s documentation says its cloud models use inference compute on ollama.com. Before sending real inputs, identify the selected model, where inference runs, and whether enabled features use remote embeddings, telemetry, or logs. A local API address alone does not establish that the model or every connected feature runs locally.
- llama-server documentation
Review the README’s description of the server’s REST APIs and web UI.
- Cloud models
Check Ollama’s description of cloud inference.
- Offline Operation
Read the documentation about offline operation for downloaded local models.
Try it
Trace one request from a client to a selected model. Record the server endpoint, where inference runs, and one setting or model choice that could route data elsewhere.
For example, LM Studio documents an LLM API server on localhost; an Ollama cloud model uses inference compute on ollama.com, so the selected model changes where inference runs.
Latest
No recent updates are available.
Books
Hands-On Large Language Models
Jay Alammar and Maarten Grootendorst · 2024
Use it to build a working mental model of tokens, embeddings, transformers, and model use. Its examples date quickly, so check current runtime and model compatibility separately.
Build a Large Language Model (from Scratch)
Sebastian Raschka · 2024
Follow a code-based path through tokenization, GPT-style architecture, pretraining, and instruction formatting. Use it for durable model concepts, not as a guide to current runtime support.
Designing Machine Learning Systems
Chip Huyen · 2022
Read it for systems-level thinking about deployment, prediction services, monitoring, and infrastructure. Published in 2022, it offers general serving concepts rather than current local-runtime guidance.