Artificial intelligence can run without an internet connection when the model files are stored on the device and the device has enough memory and processing power to run them. Phone features built on Google's Gemini Nano, open-source tools such as llama.cpp, and speech models such as Whisper all work this way once the files are downloaded. The limits are memory, device support, and the fact that an offline model cannot look anything up.
Key takeaways
- Offline AI means the model weights sit on the device, so each request is processed there and no server is contacted.
- Model files must still be downloaded once, and phone features depend on a device list and on the models installed.
- Memory for weights is roughly parameters times bits per weight divided by eight, which is why compression to 4 bits or fewer matters.
- Apple describes an on-device model of about 3 billion parameters compressed to 2 bits per weight, and Meta released 1 billion and 3 billion parameter Llama 3.2 models for edge devices.
- An offline model cannot retrieve current information, and mobile systems add quotas and foreground-only rules.
What "offline" means for an AI model
A cloud chatbot sends each prompt to a data center and returns the reply. An offline model reverses that arrangement: the trained weights are copied to the device, and the processor on that device does the calculation. Three kinds of products work this way today.
The first is operating-system features. Google's documentation for the ML Kit GenAI APIs states that they run entirely on-device and that functionality remains the same without a reliable internet connection. They use Gemini Nano through AICore, an Android system service that shares one copy of the model among apps. The second is open-source runtimes. The llama.cpp project describes itself as LLM inference in C/C++, released under the MIT license, with a plain C/C++ implementation that has no dependencies. The third is single-task models such as OpenAI's Whisper speech recognition, whose code and weights are also released under the MIT license and installed with a command-line package.
In every case an internet connection is needed once, to get the software and the model files. After that, the connection is optional.
Why memory decides what fits
A language model is mostly a long list of numbers called weights. The storage needed for the weights alone is the parameter count multiplied by the bits stored per weight, divided by eight to convert bits to bytes. Many models are published at 16 bits per weight, which is too large for most phones. Quantization stores each weight in fewer bits. The llama.cpp page lists support for integer quantization from 1.5-bit to 8-bit, and the authors of the GPTQ paper (arXiv, first submitted October 31, 2022) report that reducing weights to 3 or 4 bits allowed a 175 billion parameter model to run inside a single GPU, with negligible accuracy degradation relative to the uncompressed baseline.
The table applies the arithmetic. These are illustrative figures for weights only, in decimal gigabytes. Real use needs additional memory for the context the model is reading, so a device needs headroom beyond these numbers.
| Parameters | 16 bits per weight | 8 bits per weight | 4 bits per weight |
|---|---|---|---|
| 1 billion | 2 GB | 1 GB | 0.5 GB |
| 3 billion | 6 GB | 3 GB | 1.5 GB |
| 7 billion | 14 GB | 7 GB | 3.5 GB |
| 70 billion | 140 GB | 70 GB | 35 GB |
Whisper shows the same pattern in a smaller domain. Its repository lists five model sizes, from 39 million parameters needing about 1 GB of video memory to 1,550 million parameters needing about 10 GB, and the speed measurements it reports were taken on an A100 data-center GPU.
Three documented on-device models
Apple published a technical update on June 9, 2025 describing a compact model of approximately 3 billion parameters optimized for Apple silicon. Apple states that it compressed the on-device model to 2 bits per weight using quantization-aware training, and that the models are designed to support 15 languages. By the arithmetic above, 3 billion weights at 2 bits is about 0.75 GB before overhead. The page does not say whether the developer framework works without a network connection, so that point is left open here.
Meta announced Llama 3.2 on September 25, 2024, with 1 billion and 3 billion parameter text-only models described as fitting onto edge and mobile devices, with a context length of 128,000 tokens. Meta lists summarization, instruction following, and rewriting as the intended on-device tasks, and says local processing keeps data such as messages and calendar information from being sent to the cloud. That privacy point is Meta's own description, and the actual behavior of an app depends on how the app is built.
Google's Android documentation shows the practical side. The feature-specific APIs for summarization, proofreading, rewriting, and image description have a device list that differs from the list for the Prompt API, which is split by Gemini Nano version. The advanced mode of speech recognition is limited to Pixel 10 and Pixel 11 for now, while a basic mode covers most devices on Android API level 31 or later.
What an offline model cannot do
- It cannot retrieve current facts. Its knowledge comes from training data and from whatever the user supplies in the prompt.
- It may behave differently between versions. Google notes that different versions of Gemini Nano may return different output from the same prompt.
- It faces platform limits. On Android, inference is allowed only while the app is in the foreground, and per-app quotas return error codes when requests are too frequent or use too much battery.
- It is limited by hardware. Support depends on the device, the memory available, and the models downloaded to it.
- It is not automatically private. An app can run a model locally and still send other data over the network, so the app's own privacy policy and network behavior matter.
Questions to ask before relying on an offline tool
- Which model and version is installed, and how large is the download?
- Which tasks does the maker list for the model, and are they the tasks needed?
- How much memory does the device have, and is it on the supported device list?
- Does the app work with the network switched off, and does it send any data when the network is on?
- How are model updates delivered, and can an older version be kept?
FAQ
Can ChatGPT-style assistants work without internet?
Hosted assistants cannot, because the model runs on the provider's servers. Smaller models can run on a device once downloaded, as the Llama 3.2 1B and 3B releases and Android's Gemini Nano features show.
Does offline AI need a powerful computer?
It depends on the model. Per the table, a 3 billion parameter model at 4 bits needs about 1.5 GB for weights, while a 70 billion parameter model at the same precision needs about 35 GB.
Is a locally run model more accurate than a cloud one?
No source reviewed makes that claim. Local models are chosen for control of data and for working without a connection. Quantization research reports small accuracy loss for large models, but results vary by model and method.
Do offline models learn from what a person types?
Not by default. The weights on the device are fixed unless the maker ships an update, and any storage of prompts is a property of the app.
Related reading
- Whether AI can work without data
- Whether AI can operate without coding
- The components an AI system needs
- AI and cryptography
More in AI Basics.
Last verified: October 2026
Sources
- ML Kit GenAI APIs, Google for Developers, checked October 2026
- llama.cpp repository and README, ggml-org, checked October 2026
- Frantar E., Ashkboos S., Hoefler T., Alistarh D., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, arXiv 2210.17323, October 2022, revised March 2023
- Updates to Apple's On-Device and Server Foundation Language Models, Apple Machine Learning Research, June 9, 2025
- Llama 3.2 announcement for edge and mobile models, Meta AI blog, September 25, 2024
- Whisper repository and README, OpenAI, checked October 2026