Run AI Locally in 2026 — Ollama + Open WebUI, Free, Private, 30-Minute Setup
Your data never leaves your machine. No subscriptions. No censorship. This step-by-step guide walks you through installing Ollama v0.32 and Open WebUI on any computer, pulling free open-weight models like Llama 4 Scout, Qwen3, and DeepSeek V4 Flash, and getting a ChatGPT-like interface running entirely offline. Total cost: $0. Total time: 30 minutes.
Download Ollama (Free) →In 30 minutes you'll build a fully private AI assistant. Ollama (v0.32, the model runner), Open WebUI (the ChatGPT-like interface with RAG, voice, web search, and multi-model chat), and your choice of free open-weight models. All running on your own hardware. No internet required after setup. No data sent to any server. No censorship, no content filters, no rate limits. The stack: install Ollama → pull a model (Llama 4 Scout 8B for 8GB VRAM, Qwen3 Coder 32B for 24GB) → install Open WebUI via Docker → open localhost:3000 → you have a private ChatGPT. This guide covers macOS, Linux, and Windows, with hardware recommendations for every budget, advanced configuration, and troubleshooting.
What You're Building
A completely private AI assistant that runs on your own computer. Ollama is the engine — it downloads, manages, and runs AI models locally using llama.cpp under the hood. Version 0.32 (July 2026) brings improved multi-GPU scheduling, GGUF model compatibility, and support for NVIDIA Blackwell GPUs (RTX 50 series). Open WebUI is the interface — it gives you a ChatGPT-like chat experience with conversation history, file uploads (PDF, images, code), RAG document querying, voice input/output via local Whisper + TTS, web search integration, model switching mid-conversation, and a plugin system (Functions). Together they form a self-hosted AI platform that requires zero internet connection after the initial model download. Your prompts, conversations, and uploaded files never leave your machine. No account needed. No subscription. No one can shut it down or change the rules. This is the most important AI skill in 2026 — knowing how to run AI on your own terms, not someone else's server.
Quick Reference: Best Models for Your Hardware
| Model | VRAM Needed | Download Size | Quality Tier | Best For | Command |
|---|---|---|---|---|---|
| Llama 4 Scout 8B | 6-8GB | ~5GB | Good | Everyday chat, laptops | ollama pull llama4-scout:8b |
| Qwen3 Coder 32B | 20-24GB | ~18GB | Excellent | Coding, reasoning | ollama pull qwen3:32b |
| DeepSeek V4 Flash | 8-12GB | ~8GB | Very Good | General purpose, fast | ollama pull deepseek-v4-flash |
| Gemma 3 12B | 8-10GB | ~7GB | Good | Lightweight, fast | ollama pull gemma3:12b |
| Qwen3 32B | 20-24GB | ~18GB | Excellent | Best overall quality | ollama pull qwen3:32b |
| Laguna XS 2.1 (33B MoE) | 10-12GB | ~20GB | Very Good | Coding, 3B active params | ollama pull laguna-xs:2.1 |
Step 1: Install Ollama (5 Minutes)
macOS
Download the DMG from ollama.com/download or use Homebrew. Ollama runs natively on Apple Silicon (M1-M4) with Metal acceleration:
After installation, Ollama runs as a background service on localhost:11434. Verify with ollama --version — you should see v0.32.x or newer.
Linux
One command. Works on Ubuntu, Debian, Fedora, and Arch. Requires systemd:
For NVIDIA GPUs: ensure you have CUDA drivers installed (driver 550+ for compute capability 5.0+). Verify with nvidia-smi. For AMD GPUs: ROCm support is available for RX 7000 series and newer.
Windows
Download the installer from ollama.com/download. Ollama runs natively on Windows 10/11 via WSL2 or as a standalone Windows app. The Windows app uses the NVIDIA CUDA driver directly — no Docker needed for GPU acceleration. After installation, Ollama runs as a system tray app on localhost:11434.
Step 2: Pull Your First Model (2-15 Minutes Depending on Size)
Open a terminal. Pull a model with one command. Download sizes range from 4GB to 20GB depending on the model. Ollama automatically selects the best quantization for your hardware:
OLLAMA_NUM_THREADS=8 to match your core count for best CPU performance.ollama list to see all downloaded models. Run ollama ps to see which models are currently loaded in memory. Run ollama rm model-name to free disk space by removing unused models.Step 3: Install Open WebUI (5 Minutes)
Docker (Recommended)
If you have Docker installed, Open WebUI is one command. This pulls the latest image, mounts a persistent volume for your chat history and settings, and auto-restarts on boot:
Open http://localhost:3000 in your browser. Create an admin account — this account is stored locally only, no external authentication. Open WebUI automatically detects Ollama running on your machine. Your models appear in the dropdown menu at the top of the chat interface. Select a model and start chatting.
Without Docker (Python/pip)
If you prefer pip or don't have Docker:
The pip installation includes all dependencies. It runs on Python 3.11+. Open WebUI detects Ollama automatically on localhost:11434.
Step 4: Explore Open WebUI's Full Feature Set
Open WebUI is far more than a chat interface. Here's what you unlock after the 30-minute setup:
Upload PDFs, DOCX, text files, or entire codebases. Open WebUI indexes them locally using its built-in vector database (ChromaDB or Qdrant). Ask questions against your documents — everything stays on your machine.
Built-in Speech-to-Text via local Whisper models and Text-to-Speech via multiple engines. Hands-free conversation with your local AI. Video and voice calls supported.
Enable web search for real-time information beyond the model's knowledge cutoff. Supports SearXNG, Google PSE, Brave Search, and more. The model retrieves current info mid-conversation.
Extend Open WebUI with Python functions — code execution, image generation via ComfyUI, API calls, database queries. Community functions available in the marketplace. Build custom tools that your AI can invoke.
Switch models mid-conversation. Compare responses from different models side by side. Route specific types of queries to specialized models (coding → Qwen3 Coder, general → Llama 4 Scout).
Open WebUI supports multiple user accounts with role-based access control. Share your instance with family or teammates. Admins control which models each user can access.
Step 5: Advanced Configuration
GPU Layer Control
Control how many model layers run on GPU vs CPU. This is critical for models that exceed your VRAM — offload what fits on GPU, run the rest on CPU:
Context Length Extension
Default context is 2048 tokens. For document analysis or long conversations, increase it (at the cost of higher VRAM usage):
Access Your AI From Anywhere
Open WebUI runs on localhost by default. To access it from other devices on your network or remotely:
Hardware Recommendations — Every Budget
| Budget | Hardware | Best Model | Speed (tok/s) | Context |
|---|---|---|---|---|
| $0 (existing laptop) | 16GB RAM, CPU only | Llama 4 Scout 8B | 5-10 | 2K-4K |
| $500-800 | RTX 3060 12GB | Qwen3 Coder 32B (partial GPU) | 20-40 | 8K-16K |
| $1,000-1,500 | RTX 4070 Ti 12GB / RTX 5070 12GB | DeepSeek V4 Flash, Laguna XS2.1 | 50-80 | 16K-32K |
| $1,500-2,500 | RTX 4090 24GB / RTX 5090 32GB | Qwen3 32B, Llama 4 Maverick | 80-120 | 32K-128K |
| $5,000+ | Dual RTX 4090/5090 / A6000 48GB | DeepSeek V4 Pro, Llama 4 Maverick | 100+ | 128K+ |
FAQ
--num-gpu flag to manually control how many layers stay on GPU. For persistent VRAM constraints, use quantized models (Q4_K_M versions use roughly half the VRAM of FP16 with minimal quality loss) or choose a smaller model from the reference table above.ollama update (or re-run the install script on Linux, download the new installer on Windows/macOS). Open WebUI (Docker): docker pull ghcr.io/open-webui/open-webui:main && docker stop open-webui && docker rm open-webui, then re-run the docker run command. Your data persists in the mounted volume. Open WebUI (pip): pip install --upgrade open-webui. Check for updates monthly — both projects release frequently with new features and model support.You Just Built Your Own Private AI — Here's Why It Matters
In 30 minutes, with two free open-source tools and zero dollars, you built a private AI assistant that rivals ChatGPT. Your data stays on your machine. No subscriptions. No censorship. No one can shut it down. The models are free, the tools are free, and the hardware is whatever you already own. More importantly: you now own your AI infrastructure. When cloud services raise prices, change terms, add censorship, or go offline — your AI keeps running. This is the most important AI skill to learn in 2026: knowing how to run AI on your own terms, not someone else's server. The models will keep improving, but the stack — Ollama + Open WebUI — is stable, mature, and will serve you for years.
The Bottom Line
Related Posts
You run one terminal command, pull a model in under two minutes, open localhost:3000 in your browser, and type your first prompt into a ChatGPT-like interface that runs entirely on your own machine — no internet, no account, no subscription, no censorship.
You type ollama run and watch the model spin up, entirely offline, entirely yours.
🔬 Testing Methodology
- Test period
- 7+ days of hands-on use
- Pricing
- Manually verified at time of writing
- Affiliate
- Some links may earn a commission — full disclosure