Hosting Open-Source Models — Day Companion
Hosting Open-Source Models
Workshop 1 · ~2 hours
Backends — Run Models as APIs
Pick a backend, install it, serve a model, and call it with curl. By the end your laptop is an API server.
Every backend serves an OpenAI-compatible API. Change the base URL, done — your app doesn’t know it switched providers.
The starting point. One command to install, one to pull a model, one to serve it. Best for personal use and small teams.
Bare metal control. Context size, GPU layer allocation, quantization knobs that Ollama hides. For when you need to tune.
The fastest local option on Apple Silicon. Uses unified memory directly — no copying between CPU and GPU.
Production serving. Batched inference, tensor parallelism, PagedAttention. For 10+ concurrent users on NVIDIA GPUs.
Apple Silicon → MLX. NVIDIA + many users → vLLM. Simple setup → Ollama. No GPU → rent cloud or CPU mode.
Quick Start
ollama pull qwen3:8b
ollama serve
# In another terminal:
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen3:8b","messages":[{"role":"user","content":"Say hello in three languages."}]}'
Workshop 2 · ~2 hours
Hosting — Put It on the Internet
Take the model you’re serving locally and make it reachable from outside your network — securely, cheaply, and without port forwarding.
Cost Comparison Calculator → Self-hosting electricity costs vs API bills. Find your break-even point. AI Cost Calculator → Model your full AI workload costs across providers and use cases.Free. Your machine, your model, accessible at your domain. No static IP, no port forwarding, no firewall config.
RunPod, Vast.ai, Modal, Lambda. Spot instances from $0.20/hr. Serverless (Modal) means pay only when running.
OpenRouter: one API key, any model. Groq: fastest inference on custom LPU. Together.ai, Fireworks, Replicate.
Put a proxy between users and the model. It holds the real API key, enforces per-user budgets, and logs everything.
What to watch: requests/min, tokens/sec, error rate, cost/day, p95 latency. You need a log and a script, not Grafana.
Quick Reference
Model Recommendations
| Use Case | Model | Size | Why |
|---|---|---|---|
| General assistant | Qwen3 8B | ~6 GB | Best quality-per-GB in 2026 |
| Code generation | Qwen3 Coder 8B | ~6 GB | Trained on code, fast |
| Embeddings | Qwen3-Embedding-0.6B | ~1 GB | Tiny, fast, surprisingly good |
| Long documents | Qwen3 32B | ~20 GB | 128K context, strong reasoning |
| Maximum quality | Qwen 3.8 27B | ~15 GB | Newest, highest benchmarks |
| Best value (MoE) | Qwen 3.5 35B-A3B | ~7 GB | 35B quality in 7 GB via MoE |
| Minimum cost | Qwen3 0.6B | ~1 GB | Runs on anything, including phones |
Cost Comparison
| Setup | Monthly Cost | Control |
|---|---|---|
| Self-hosted (your Mac) | $0 | Full |
| Cloud GPU (spot, 2 hr/day) | $24 | Full |
| OpenRouter (Qwen3 8B) | $5–20 | Some |
| Cloud GPU (24/7) | $290 | Full |
| OpenAI API (GPT-4o) | $60–200+ | Some |
| ChatGPT Plus | $20 | None |
Hardware → Model Fit
| Your RAM / VRAM | Max Practical Model | Good Default |
|---|---|---|
| 8 GB | 4B quantized | Qwen3 4B Q4 |
| 16 GB | 14B quantized | Qwen 3.5 9B Q4 |
| 32 GB | 32B quantized | Qwen 3.8 27B Q4 |
| 48 GB+ | 70B quantized | Llama 3.3 70B Q4 |
For the Nerds
--speculative-model Qwen3-0.6BFull cheat sheet: /x/bleeding-edge-inference
Everything Else
RTX 3060 12GB ($0.06/hr): GPU not detected — CPU fallback at 3 tok/s. Always check nvidia-smi!
Cost per million tokens self-hosted: $0.034 vs $0.10 DeepInfra vs $3-15 frontier.