Your Laptop Won’t Cut It

Everything you’ve done so far — Ollama, fine-tuning, benchmarking — works on a single machine for a single user. Production is a different animal. You need to handle concurrent requests, manage GPU memory across sessions, keep latency under 200ms, and not crash at 3am when traffic spikes.

The good news: the tooling is mature. The three engines that power 85% of production open-model inference are all open source. Let’s set them up.

The Three Production Engines

Engine	Best For	Key Innovation	Throughput
vLLM	General production serving	PagedAttention memory management	Up to 24x vs naive
TensorRT-LLM	Maximum perf on NVIDIA GPUs	Custom CUDA kernels	Highest absolute
TGI (HuggingFace)	HuggingFace ecosystem users	Native HF integration	Best for single-user latency

Default recommendation: vLLM. Largest community (14K+ GitHub stars), weekly releases, broadest hardware support, and the most flexible serving options. Unless you have a specific reason to choose TensorRT or TGI, start with vLLM.

Option 1: vLLM (Recommended)

vLLM’s killer feature is PagedAttention — it manages GPU memory for the KV cache the way an OS manages virtual memory with paging. This means no wasted VRAM, no out-of-memory crashes from long conversations, and near-perfect GPU utilization.

Quick Start with Docker

# Serve Llama 3.1 8B with vLLM (OpenAI-compatible API)
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9

# Test it (OpenAI-compatible API)
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 256
  }'

vLLM 0.8 Features (January 2026)

Speculative decoding — Uses a small draft model to predict tokens, then verifies in batch. 1.5-2x faster for long outputs.
Disaggregated prefill — Separates the prefill (processing the input) from generation (producing tokens) across different GPUs. Better resource utilization.
LoRA hot-swapping — Serve multiple fine-tuned variants from a single base model. Load and unload LoRA adapters per request without restarting the server.

# Serve with LoRA adapters (multi-tenant)
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --lora-modules customer-support=./lora/support legal-review=./lora/legal \
  --max-loras 4

# Request a specific LoRA variant
curl http://localhost:8000/v1/chat/completions \
  -d '{"model": "customer-support", "messages": [{"role": "user", "content": "Help me with a refund"}]}'

Option 2: Ollama for Low-Traffic Production

If your use case is internal tools, small teams, or low-traffic APIs (under ~50 concurrent users), Ollama is production-viable. It’s simpler to set up and maintain than vLLM.

# Docker Compose for production Ollama
cat > docker-compose.yml << 'EOF'
version: '3.8'
services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    environment:
      - OLLAMA_NUM_PARALLEL=4
      - OLLAMA_MAX_LOADED_MODELS=2
    restart: unless-stopped

  webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - webui_data:/app/backend/data
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama_data:
  webui_data:
EOF

docker compose up -d

Option 3: TGI for HuggingFace Users

# Serve with Text Generation Inference
docker run --gpus all \
  -v ~/.cache/huggingface:/data \
  -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --max-input-length 4096 \
  --max-total-tokens 8192

TGI has lower tail latencies for single-user interactive scenarios. If your app is one user at a time (like an internal assistant), TGI gives a snappier feel. For high concurrency, vLLM wins.

The Production Checklist

Before you expose any model to real users, verify these items:

Rate limiting — Put nginx or Caddy in front with request rate limits. One bad actor can monopolize your GPU.
Health checks — /health endpoint monitored every 30s. Restart container on failure.
Input validation — Set max-input-length to prevent adversarial prompts that consume all VRAM.
Output limits — Set max-tokens per request. A runaway generation can lock the GPU for minutes.
Monitoring — Track tokens/sec, queue depth, GPU memory, latency p50/p95/p99. Prometheus + Grafana is the standard stack.
Graceful degradation — Return a queue position or retry-after header when the GPU is saturated. Don’t just timeout.
Model versioning — Pin the exact model version and quantization. “latest” in production is a ticking bomb.
Logging — Log input/output for debugging (with PII scrubbing if handling user data). You’ll need this when the model says something unexpected.

Cost Comparison: Self-Hosted vs API

Setup	Monthly Cost	Tokens/Month	Cost per 1M Tokens
OpenAI GPT-4o	Usage-based	Unlimited	$2.50 - $10.00
Claude 3.5 Sonnet	Usage-based	Unlimited	$3.00 - $15.00
1x H100 (cloud rental)	~$2,000	~500M+ tokens	$0.004
1x RTX 4090 (owned)	~$30 electricity	~100M+ tokens	$0.0003
M3 Max MacBook (owned)	~$5 electricity	~30M+ tokens	$0.0002

The math is brutal. If you’re spending more than $500/month on API calls, self-hosting pays for itself within 1-4 months. If you already have GPUs, the cost difference is 100-1000x.

Your Open Source AI Stack Is Complete

Over six lessons, you’ve gone from “what’s an open source model?” to knowing how to pick one, run it, fine-tune it, evaluate it, and deploy it to production. The barrier between you and GPT-level intelligence is now one ollama run command.

The proprietary AI companies don’t want you to know how easy this has become. Now you do.

From Download to Production — Deploying Open Source Models

Related Resources