5 Minutes. That’s All It Takes.

By the end of this lesson, you’ll have an AI model running on your machine — completely offline, completely private, completely free. Not a demo. Not a trial. The real thing.

Step 1: Install Ollama

Ollama is the de facto standard for running local LLMs. One install, automatic GPU detection, and a CLI that feels like Docker for AI models.

# macOS
brew install ollama

# Linux (one-liner)
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download from https://ollama.com/download
# Native ARM64 support added in 2026 — no more x86 emulation penalty

After installation, the Ollama server starts automatically. Verify it’s running:

ollama --version
# ollama version 0.6.x

Step 2: Pull and Run Your First Model

One command. That’s it.

# Pull and start chatting with Llama 3.1 8B (4.7GB download)
ollama run llama3.1

# You're now in an interactive chat. Type anything:
# >>> Explain quantum computing in 3 sentences
# >>> Write a Python function to merge two sorted lists
# >>> /bye  (to exit)

The first run downloads the model (~4.7GB for the 8B variant). Subsequent runs start instantly — models are cached locally.

Choosing Your Quantization Level

Models come in different quantization levels. Lower bits = smaller file, slightly lower quality:

# Default (Q4_K_M — best balance of quality and size)
ollama run llama3.1

# Specific quantization — when you want to control the tradeoff
ollama run llama3.1:8b-q4_K_M    # 4-bit, 4.7GB — recommended for most
ollama run llama3.1:8b-q8_0      # 8-bit, 8.5GB — higher quality, needs more RAM
ollama run llama3.1:8b-fp16       # Full precision, 16GB — reference quality

Step 3: Manage Your Model Library

# List installed models
ollama list

# Pull a model without starting it
ollama pull mistral
ollama pull gemma3:9b
ollama pull qwen3:7b
ollama pull phi4:14b

# Show model details (size, quantization, template)
ollama show llama3.1

# Remove a model to free disk space
ollama rm mistral

# Pull a specific size variant
ollama pull gemma3:2b    # Tiny — runs on 4GB RAM
ollama pull gemma3:27b   # Large — needs 16GB+ RAM

Step 4: Use the REST API

This is where it gets powerful. Ollama exposes an OpenAI-compatible API on localhost:11434. Any application, script, or tool that calls the OpenAI API can be redirected to your local model by changing the base URL.

# Generate a completion (streaming)
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Write a haiku about debugging"
}'

# Chat format (OpenAI-compatible)
curl http://localhost:11434/v1/chat/completions -d '{
  "model": "llama3.1",
  "messages": [{"role": "user", "content": "Explain REST APIs in 50 words"}]
}'

# List running models
curl http://localhost:11434/api/tags

Using Ollama from Python

# pip install ollama
import ollama

response = ollama.chat(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'What is the capital of France?'}]
)
print(response['message']['content'])

# Or use the OpenAI SDK (just change the base URL)
from openai import OpenAI

client = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')
response = client.chat.completions.create(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'Hello!'}]
)
print(response.choices[0].message.content)

Step 5: Add a Chat UI

Talking to a model through the terminal is fine for testing. For daily use, you want a proper chat interface. Open WebUI is the best option — it looks and feels like ChatGPT but runs entirely on your machine.

# One Docker command — connects to your running Ollama automatically
docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

# Open http://localhost:3000 in your browser
# Create an account (stored locally) and start chatting

Open WebUI gives you model switching, conversation history, file uploads, web search integration, and multi-user support. All running on your hardware.

Step 6: Performance Tuning

Ollama’s defaults are conservative. If you have more hardware, tell it:

# Allow 2 models loaded simultaneously (default: 1)
export OLLAMA_MAX_LOADED_MODELS=2

# Allow 4 parallel requests per model (default: 1)
export OLLAMA_NUM_PARALLEL=4

# Increase context window (default varies by model)
export OLLAMA_NUM_CTX=8192

# Set GPU layers (0 = CPU only, -1 = all layers on GPU)
export OLLAMA_GPU_LAYERS=-1

# Restart Ollama to apply
ollama serve

What You Just Built

You now have a local AI stack: Ollama serving models, an OpenAI-compatible API your code can call, and a browser-based chat UI. Total cost: $0. Data privacy: absolute. Uptime: 100% (it’s your machine).

In lesson 4, we’ll take it further — fine-tuning a model so it speaks your language, knows your codebase, and follows your conventions.

Running Models Locally with Ollama

Related Resources