Offline LLMs: Running AI on Your Own Machine

• Privacy, genuinely. Nothing leaves the machine — which matters for client data, medical notes, anything under an NDA, and anything you simply don't want logged.
• No cost, no rate limit, no quota. Use it as much as you like.
• Works with no connection. Airplane mode, a train, a field site, a secure facility.
• Offline-capable tools built on top of it — transcription, translation, summarisation of your own files.
• Predictable latency, in the sense that it never queues behind a busy server.

Model size in parameters, and the quantisation used to shrink it, determine what fits. The number that actually matters is memory — system RAM if you're loading from disk, or VRAM if you have a GPU.

• 1–3B parameters: 8GB RAM is comfortable. Fast on CPU. Fine for classification, extraction, short rewrites, simple questions.
• 7–8B: 16GB RAM, or any modern GPU with 8GB+ VRAM. This is the sweet spot for general quality.
• 13–14B: 32GB RAM, or 12–16GB VRAM. Noticeably better reasoning and longer context.
• 30B+: 64GB RAM or a serious GPU. Diminishing returns per GB, and it will be slow on CPU.
• Quantisation (running the model at reduced precision) roughly halves the memory needed with a modest quality cost — 4-bit is the common default for a reason.

You do not need to install Python, learn a framework, or write any code. Pick a desktop application, download a model, and start chatting.

1. install a local runner (a desktop app, not a library)
2. download a model file in GGUF format, sized to your RAM
3. run it, and use it exactly like any other chat window

For developers, the same models are served over a local
API and anything that speaks that API works unchanged.

Once it's running, treat it like any other model: the prompting advice in the LLMs guide applies identically, and the failure modes are the same ones.

• Weaker than the best cloud models, especially on hard reasoning, long context and anything current. A 2024-era small local model will not match a frontier model.
• No current information, and no browsing. It knows only what it was trained on, frozen at download time.
• Slower. On a CPU, a long answer can take minutes.
• You own the hardware cost, the disk space (tens of gigabytes) and the setup time.
• Model choice is on you — a bad small model is much worse than a good large one, and the download is long.

• Anything confidential: client material, health notes, source code under NDA, unreleased work.
• High-volume, repetitive work where a per-token bill adds up.
• Environments with poor or no connectivity, or where cloud services are not permitted.
• A base layer for your own tools — local transcription, tagging, and search over your own files.
• When you simply want an experiment you fully control, with no account and no telemetry.

For general research, long documents and current events, a cloud model with search is still better. The two are complements: local for the private and repetitive, cloud for the hard and current.