On-Device AI
What's different about AI that runs on your own device
① On-device AI means the request starts and finishes entirely inside your phone or laptop. It works with no internet connection and sends nothing to a company's servers.
② The obstacle is size. A server-grade model can't fit on a device as-is, so quantization compresses it and a dedicated chip called an NPU keeps the battery from draining.
③ Whenever you see the phrase "on-device AI," check one thing: does everything really run on the device, or does part of it still travel to a server?
① What "on-device AI" actually means
Every AI covered in this series so far runs on a server. Type a question, and it travels over the internet to a company's large-scale servers, gets computed there, and the answer comes back to your screen. Even the "bowl" analogy in tokens and the context window assumed that server round trip.
On-device AI removes that round trip. The model itself sits entirely inside the phone or laptop, and from the moment you type a question to the moment an answer appears, everything happens inside that device. Not one byte leaves for the internet.
| Server AI (covered so far) | On-Device AI | |
|---|---|---|
| Where the model lives | A company's large servers | Inside your device |
| Internet required | Yes | No |
| Request data | Sent to a server | Never leaves the device |
| Model size | Can be hundreds of billions of parameters | Limited to what device memory can hold |
A phone's photo app recognizing faces offline, or a keyboard predicting the next word with no internet, both work this way. As covered in how an LLM works, a model is ultimately a block of numeric parameters — and here that block is held and computed by the device in your hand, not a server.
② Why run it on-device instead of sending it to a server
Servers are far more powerful, so why give up that performance? Four reasons.
| Reason | Detail |
|---|---|
| Latency | A server round trip always carries network time. Finishing inside the device removes that delay for real-time responsiveness. |
| Offline operation | Functionality keeps working on a plane, in a subway, anywhere without a connection. |
| Privacy | Sensitive data — photos, messages, location — never leaves the device to begin with. |
| Cost | Server compute means the company pays for power and GPUs on every request. On-device shifts that cost to the user's battery. |
Privacy matters more the more sensitive the data is. That said, this piece doesn't oversimplify to "on-device means safe." How much stays on the device and how much still goes to a server varies app by app. What to check when using server-based AI is covered separately in AI and your privacy.
③ The obstacle is size — why you can't just cram it in
It sounds simple enough to copy a server-scale model straight onto a phone. You can't. A model is a list of numbers (parameters), and every one of those numbers takes up storage.
Stored the standard way (FP16, 16 bits per number), even a relatively small 7-billion-parameter model runs about 14GB. A modern phone typically carries 8–12GB of RAM — loading one model alone would leave no room for anything else the device needs to do.
Even if the model fits in memory, running that computation on a general CPU drains a battery fast. Servers can lean on dedicated cooling and grid-scale power; a phone has to survive on a single battery in your palm. Which is why both shrinking the model (quantization) and using a chip built for the job (an NPU) are necessary.
④ Quantization — shrinking the model by lowering precision
Quantization reduces the number of bits used to represent each parameter. Think of it like saving a photo at lower quality to shrink the file — you store the information a bit less precisely, and the size drops sharply.
| Precision | Bits per parameter | Size of a 7B model | Note |
|---|---|---|---|
| FP16 (original) | 16 bits | ~14GB | Common default precision on servers |
| INT8 | 8 bits | ~7GB | Halved |
| INT4 | 4 bits | ~3.5–4GB | Fits comfortably in phone RAM |
Drop the same model from 16 bits to 4 bits and its size shrinks to roughly a quarter. A 14GB model comes down to around 4GB — small enough to fit on a device while leaving room for other apps.
Storing numbers more coarsely accumulates small computational error, so a quantized model may answer slightly worse than the original. Recent quantization techniques have narrowed this gap considerably, to the point it's hard to notice in everyday chat or short summaries. The loss shows up more on complex reasoning or tasks that need precise numeric calculation.
⑤ NPUs — why CPUs and GPUs aren't enough
Even after a model fits on the device, running it on an inefficient processor still drains the battery quickly. That's why recent phones and laptops carry a dedicated chip called an NPU (Neural Processing Unit).
A CPU is a general-purpose device designed to handle all kinds of computation sequentially. A GPU is strong at running the same computation in bulk, in parallel — originally for graphics. An NPU goes a step further: its circuitry is designed specifically for the operations an AI model actually performs (matrix multiplication, chiefly), making it far more efficient at exactly that job.
| Processor | Designed for | AI compute efficiency |
|---|---|---|
| CPU | General computation (sequential) | Low — general-purpose, not specialized |
| GPU | Bulk parallel computation (graphics, etc.) | Medium — usable for AI, but power-hungry |
| NPU | AI computation only | High — same work, far less power |
Apple's Neural Engine, Qualcomm's Hexagon, and Google's Tensor all belong to this category. When a new phone launch touts "AI performance up several-fold," it usually refers to NPU compute (commonly measured in TOPS, trillions of operations per second).
⑥ In practice — what works on-device today, and what doesn't
Advances in quantization and NPUs haven't moved all server-grade AI onto devices. What runs well on-device today and what still needs a server remains sharply divided.
| Examples | Why | |
|---|---|---|
| Works well on-device | Face/object recognition in photos, real-time translation, next-word prediction, short summaries | Tasks a small model handles fine, or that can absorb compression loss |
| Still needs a server | Deep analysis of long documents, complex code generation, questions needing up-to-date information | Requires a large model and wide context window, or access to live data |
Many products claim "on-device AI" while in practice running a hybrid setup — some features on-device, complex requests still routed to a server. The reliable test is simpler than the marketing copy: does the feature work with no internet at all? Turn on airplane mode and try it — the answer shows immediately.
⑦ The investment angle — what demand on-device AI creates
More on-device AI means growing demand across two kinds of semiconductors.
- NPUs themselves — dedicated AI circuitry becoming standard in phone and PC SoCs (system-on-chip) feeds semiconductor design and manufacturing demand. That backdrop is covered in the semiconductor sector outlook and the AI chip three-way.
- Memory inside the device — even compressed, a model still needs RAM and storage to hold and run. Part of the reason Micron's mobile-and-client segment made up 27.8% ($11.52B) of revenue last quarter traces to this demand. Details are in the Micron earnings deep dive.
This piece doesn't single out a stock as the winner, though. On-device AI is one demand driver among several for these chips — the size of that share and how fast it's growing are best checked against the data in the deep-research pieces above.
On-device AI is a trade — you remove the server round trip, and you give up model size. Quantization compresses the model, an NPU saves the battery, and in exchange it isn't as large or as accurate as a server-scale model. The surest way not to be fooled by marketing copy is a habit: whenever you see "on-device," check whether everything runs there, or only part of it does.
※ Written as of August 2026, describing general principles. Specific model sizes, quantization ratios, and NPU performance figures vary by manufacturer and model and change quickly. Investment mentions are illustrative only and are not a recommendation to buy or sell any security.
※ This guide is provided for general educational purposes and simplifies technical details for readability.
New guides, when they land
We publish AI literacy guides twice a week. Subscribe and the next one comes to you — free, unsubscribe anytime.