Choose a Cloud, Local, or Remote Model
Start with the local, remote, and cloud choices
Compare where inference runs, what needs a connection, and which device handles the work.
Start with: Choose a Cloud, Local, or Remote Model
RedLectern topic
Local and on-device AI
Local versus remote computation · offline operation · device constraints · privacy boundaries.
For: Everyone · builders
After this page you can: Decide which parts of an AI workflow should run locally.
Page reviewed
Part of: Artificial Intelligence
About Local and on-device AI
Local AI describes workflows where some or all model computation happens on a device or infrastructure you control. It can be useful when a task must keep working without a network connection, when sending particular inputs to a remote service is undesirable, or when a nearby device can respond in time. A local model still has limits: it must fit the available hardware, support the task, and be maintained along with the surrounding application.
Choose where each step runs, rather than labeling an entire app “local.” A workflow might handle sensitive input on-device, use a remote machine on a private network for a larger model, and send other requests to a hosted service. The data path matters: cloud models inside local apps, remote embeddings, telemetry, and logs can still move information beyond the device. Offline operation also depends on having the model and required application components already available, and on designing for interrupted synchronization. Decide what must work offline, what data may leave each boundary, and what latency, memory, and maintenance limits the task can tolerate.
Start here
Offline Operation
Check what still works offline
Separate model use from discovery, downloads, and update checks that may need the internet.
Start with: Offline Operation
Learn the foundations
1.Where computation runsLocal inference means the model computation runs on a local machine.
Local inference means the model computation runs on a local machine. On-device AI places the computation on the device handling the task; self-hosting means operating a service on infrastructure under your control. Those boundaries can differ: a model on another computer in your home or workplace is remote from your phone but may still be self-hosted. A local interface also does not establish where every step runs. Trace inputs, model calls, embeddings, telemetry, and logs to find the data path, including cloud models inside local apps and remote embeddings.
Use the boundary that matches the requirement. If a task must continue without internet, test the workflow with connectivity disabled. If information must stay on a particular device or network, verify every service the workflow calls and what it records. Local execution can reduce network dependence and exposure to a hosted inference service, but it does not by itself settle access control, device security, backups, or logging.
- Offline Operation
Describes offline use of downloaded models and network-dependent features.
- Privacy Policy
Describes what the service says it handles for local use and cloud-hosted models.
Local inference
Running model computations on a local machine.
On-device AI
AI computation performed on the device handling the user's task.
Self-hosting
Operating a service on infrastructure under your control.
Try it
A phone app sends text to a model running on a computer in the same office. Which part is on-device, which is remote, and what would you check before calling the workflow local or private?
The phone handles the app, but inference runs on the office computer, so it is remote from the phone. Check the network path, whether any step calls a hosted model or remote embedding service, and what the app and server log.
2.Design for offline operationOffline operation means the needed workflow can complete without relying on an internet connection.
Offline operation means the needed workflow can complete without relying on an internet connection. Having a local model is only one part of that: the model must already be available, and the application must not depend on a remote service for an essential step. Some model discovery, downloads, and update checks require network access even when chatting with a downloaded model can work offline.
For a workflow that must survive disconnection, identify which steps need a network, decide what to store locally, and define what happens to queued work and synchronization when the connection returns. Edge systems also need a plan for local storage during outages and for recovering when connectivity is restored. Test by disconnecting the device and exercising the full task, including its inputs, outputs, and any supporting services.
- Edge computing—challenges and opportunities for enterprise cloud architects
Discusses intermittent connectivity, independent operation, and managing data captured during outages.
Try it
A field app must summarize stored notes when its network is unavailable. Name two dependencies to check and one recovery behavior to specify.
Check that the model and application components are already on the device, and that summarization does not call a remote service. Specify whether queued results synchronize after reconnection and how long unsynchronized data is retained.
3.Match the workload to the deviceA local model must fit the available device resources and support the task.
A local model must fit the available device resources and support the task. Model weights are only part of the resource picture: the chosen context length and other runtime settings also affect memory use. Local hardware can make a workflow less dependent on a network, but it may not meet the required speed, memory, or input capabilities. A remote machine you control can be a middle option when a phone or laptop cannot handle the workload and connectivity to that machine is available.
Start by listing the task’s latency target, input types, offline requirement, and data boundary. Then test the candidate workflow on the actual device and with realistic inputs. Measure load time, response time, and memory under the intended settings; check that the model supports required tools or image input. Revisit the choice when hardware, models, or application dependencies change.
- lms load
Documents memory estimation and settings such as context length and GPU offload.
- Analyzing a Core ML model’s performance in Xcode
Shows how to measure prediction timing and compute-unit use on the hardware being tested.
Try it
A model loads on a laptop but becomes slow when a long conversation is used. Name two measurements or settings to inspect before moving the workflow to a remote machine.
Check memory use and response time on the target laptop, and inspect the configured context length because it affects resource use. Also confirm the model supports the task’s inputs and compare results with realistic conversations.
Explore subtopics
Local LLMs
Obtain and run language models locally; verify operation; troubleshoot common failures.
0 subtopicsModel quantizationReduced numerical precision; memory savings; quality tradeoffs; compatible formats.
0 subtopicsAI hardwareProcessors; memory; accelerators; capacity planning; inference and training requirements
0 subtopicsLocal AI runtimesRuntime and application choices; model formats; templates; compatibility; local servers.
0 subtopicsLatest
No recent updates are available.
Books
TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers
Pete Warden and Daniel Situnayake · 2019
A hands-on path from training a model to deploying inference on microcontrollers, with guidance on latency, energy, and model size. A reviewer notes that the toolchains change rapidly, so check current runtime documentation before following its implementation examples.