Jump to content

Running AI models locally

From IdeaWazaWiki

Running AI models locally means using a language model on a computer or phone that you control. Open tools now let ordinary hardware run small and medium models. This page is general information. Check each tool and model license before use.

Why run a model yourself

  • Privacy. Prompts stay on your device when every part is local.
  • Cost. After the download, a question costs only electricity.
  • Offline use. It works on a train.
  • Control. You pick the model and settings.
  • Learning. You see how memory and speed trade off.
  • Reproducibility. A saved file and fixed settings let others repeat your run.

What it takes

A model is a large file of numbers called weights. A runner loads them into memory and predicts text one token at a time. A token is a small piece of text.

Memory comes first. The model should fit in RAM, in the VRAM of a graphics card, or in unified memory shared by the CPU and GPU. The llama.cpp project can split work between CPU and GPU when a model is larger than the VRAM.[1] A GPU is usually faster, but GPT4All says no GPU is required.[2] Storage matters too.

Quantization stores each weight with fewer bits, like rounding a long number to fewer digits. The file shrinks and some quality is lost.[3] llama.cpp supports 1.5 to 8 bits per weight.[1]

Context length is how much text the model considers at once. A longer context needs more memory for the cache of earlier tokens.[3] Speed is measured in tokens per second.

The open tools

A runner is a program. A model is a data file. You need both. Checked on official pages in October 2026.

  • llama.cpp. A C and C++ library. It uses the MIT license.[1]
  • Ollama. Its code uses the MIT license.[4] Its site says models run locally or in the cloud.[5]
  • LM Studio. A desktop app for local models built on llama.cpp and MLX.[6] Its source license was not confirmed here.
  • Open WebUI. A self-hosted chat interface for Ollama and similar APIs. It can run offline. Its license is a mix with a branding rule.[7]
  • GPT4All. A desktop app with a LocalDocs feature for chatting with your files.[8] The code uses the MIT license.[2]
  • Jan. A desktop chat app that says it can run offline. Its README names Apache 2.0.[9]

Open weights and open source

Open weights means you can download the numbers. Open source asks for more. The Open Source AI Definition asks for freedom to use, study, modify, and share, plus information about training data.[10] Read the license that comes with each file.

  • Llama. The Llama 4 license calls the models proprietary to Meta and requires a separate license above 700 million monthly active users.[11] The Open Source Initiative said in 2023 that the Llama 2 license is not open source.[12]
  • Qwen. The Qwen3 repository says its open-weight models use Apache 2.0.[13]
  • Mistral. The Mistral 7B Instruct v0.3 card lists Apache 2.0.[14]
  • Gemma. Gemma 4 uses Apache 2.0.[15] Earlier Gemma models use the Gemma Terms of Use.[16]
  • Phi. The Phi-4-mini-instruct card lists the MIT license.[17]

A path for a beginner

  1. Pick a small model. Try one with a few billion weights.
  2. Install a runner. Choose one tool above, from its official site only.
  3. Download a quantized model. Look for a GGUF file from a reputable publisher. GGUF keeps tensors and metadata in one file.[18]
  4. Test it. Summarize a paragraph you know well. Note the tokens per second.
  5. Add a chat interface. Open WebUI gives you a chat page. Jan and GPT4All include one.
  6. Optional: add local retrieval. The tool feeds passages from your files to the model. Open WebUI and GPT4All offer this.[7][8] For a personal wiki, export pages as text.

A rough hardware guide

These are rules of thumb, not promises.

  • Memory. Plan for memory roughly equal to the model file size, plus overhead. Hugging Face offers a memory estimator.[19]
  • Arithmetic. A 16 bit weight takes 2 bytes, so 4 billion weights need about 8 GB. At 4 bits, about 2 GB. Real files vary.
  • Phones. Use smaller models. llama.cpp documents building for Android.[20]

Limits and honest tradeoffs

  • Capability. Smaller models are less capable than the largest cloud models.
  • Hallucination. A model can state false things with confidence.[21]
  • Speed and energy. A home machine is usually slower than a data center. Long runs use power.
  • Security. Download only from trusted sources. Loading a pickle file can run arbitrary code, and pickle is the default format for PyTorch weights.[22] Safetensors was built as a safer format.[23] Prefer safetensors or GGUF from reputable publishers, and compare the SHA256 hash that Hugging Face file pages show.[24]

Privacy and when cloud still makes sense

Local does not mean private if you add cloud parts. Ollama can run models in the cloud, and Open WebUI can connect to remote APIs and web search.[5][7] Cloud still makes sense when a task needs a larger model than your hardware can hold.

Learning, teaching, and research

  • Classrooms. A school or library can run shared local AI so student text stays on the device.
  • Experiments. Record the model name, file hash, runner version, and settings so others can repeat the run.[25]
  • Study helpers. A model can draft flashcards for spaced repetition. Check each card for errors.
  • Wiki helpers. A model can suggest summaries. A human reviews every change before it is saved.

Local models share a goal with local-first wikis and permanent storage: the reader keeps a copy. Agent software that uses a local model still needs spending rules. Users of GrapheneOS may want an assistant that stays on the phone, but check which apps run first.

Discussion questions

  • When is a small local model good enough, and when would you switch to a cloud model?
  • What must you record so others can repeat your experiment?

Essay ideas

  • Argue whether open weights are enough to call a model open.

AI prompt ideas

  • Explain quantization with an example, then quiz me with five questions.
  • Help me choose a small model, and ask me about my memory and goals first.

Readings from Wikipedia

See also

References

  1. ↑ 1.0 1.1 1.2 llama.cpp, GitHub
  2. ↑ 2.0 2.1 GPT4All, GitHub
  3. ↑ 3.0 3.1 Quantization overview, Hugging Face
  4. ↑ Ollama, GitHub
  5. ↑ 5.0 5.1 Ollama
  6. ↑ LM Studio
  7. ↑ 7.0 7.1 7.2 Open WebUI, GitHub
  8. ↑ 8.0 8.1 GPT4All, Nomic
  9. ↑ Jan, GitHub
  10. ↑ The Open Source AI Definition, Open Source Initiative
  11. ↑ Llama 4 Community License Agreement, Meta
  12. ↑ The LLaMa license is not Open Source, Open Source Initiative
  13. ↑ Qwen3, GitHub
  14. ↑ Mistral-7B-Instruct-v0.3, Hugging Face
  15. ↑ Gemma 4: Expanding the Gemmaverse with Apache 2.0, Google Open Source Blog
  16. ↑ Gemma Terms of Use, Google AI for Developers
  17. ↑ Phi-4-mini-instruct, Hugging Face
  18. ↑ GGUF, Hugging Face
  19. ↑ Model memory estimator, Hugging Face
  20. ↑ Android, llama.cpp documentation
  21. ↑ Hallucination (artificial intelligence), Wikipedia
  22. ↑ Pickle Scanning, Hugging Face
  23. ↑ Safetensors, Hugging Face
  24. ↑ model.safetensors, Qwen3-0.6B, Hugging Face
  25. ↑ Reproducibility, Wikipedia
Retrieved from "https://ideawaza.com/index.php?title=Running_AI_models_locally&oldid=92272"