Hosting Open-Source Models — Day Companion

Workshop 1 · ~2 hours

Backends — Run Models as APIs

Pick a backend, install it, serve a model, and call it with curl. By the end your laptop is an API server.

What Can I Run? → Pick your chip, slide your RAM, see which models fit with benchmark quality scores and copy-pasteable install commands. Model Hosting Calculator → Full hosting plan: hardware match, benchmark quality badges, break-even vs API costs, install commands.

The Drop-In Pattern

Every backend serves an OpenAI-compatible API. Change the base URL, done — your app doesn’t know it switched providers.

Ollama

The starting point. One command to install, one to pull a model, one to serve it. Best for personal use and small teams.

llama.cpp / llama-server

Bare metal control. Context size, GPU layer allocation, quantization knobs that Ollama hides. For when you need to tune.

MLX

The fastest local option on Apple Silicon. Uses unified memory directly — no copying between CPU and GPU.

vLLM

Production serving. Batched inference, tensor parallelism, PagedAttention. For 10+ concurrent users on NVIDIA GPUs.

The Decision Tree

Apple Silicon → MLX. NVIDIA + many users → vLLM. Simple setup → Ollama. No GPU → rent cloud or CPU mode.

Quick Start

ollama pull qwen3:8b
ollama serve

# In another terminal:
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3:8b","messages":[{"role":"user","content":"Say hello in three languages."}]}'

Workshop 2 · ~2 hours

Hosting — Put It on the Internet

Take the model you’re serving locally and make it reachable from outside your network — securely, cheaply, and without port forwarding.

Cost Comparison Calculator → Self-hosting electricity costs vs API bills. Find your break-even point. AI Cost Calculator → Model your full AI workload costs across providers and use cases.

Enterprise Cost Calculator

Production hosting: interactive vs API service vs batch processing. Break-even across GPU rental, per-token API, spot, and self-hosted.

Omni Model Benchmarks → Compare models by coding, reasoning, and agentic benchmarks with live OpenRouter pricing.

Cloudflare Tunnel

Free. Your machine, your model, accessible at your domain. No static IP, no port forwarding, no firewall config.

Cloud GPU Providers

RunPod, Vast.ai, Modal, Lambda. Spot instances from $0.20/hr. Serverless (Modal) means pay only when running.

Managed Inference

OpenRouter: one API key, any model. Groq: fastest inference on custom LPU. Together.ai, Fireworks, Replicate.

The Proxy Pattern

Put a proxy between users and the model. It holds the real API key, enforces per-user budgets, and logs everything.

Monitoring

What to watch: requests/min, tokens/sec, error rate, cost/day, p95 latency. You need a log and a script, not Grafana.

Quick Reference

Model Recommendations

Use Case	Model	Size	Why
General assistant	Qwen3 8B	~6 GB	Best quality-per-GB in 2026
Code generation	Qwen3 Coder 8B	~6 GB	Trained on code, fast
Embeddings	Qwen3-Embedding-0.6B	~1 GB	Tiny, fast, surprisingly good
Long documents	Qwen3 32B	~20 GB	128K context, strong reasoning
Maximum quality	Qwen 3.8 27B	~15 GB	Newest, highest benchmarks
Best value (MoE)	Qwen 3.5 35B-A3B	~7 GB	35B quality in 7 GB via MoE
Minimum cost	Qwen3 0.6B	~1 GB	Runs on anything, including phones

Cost Comparison

Setup	Monthly Cost	Control
Self-hosted (your Mac)	$0	Full
Cloud GPU (spot, 2 hr/day)	$24	Full
OpenRouter (Qwen3 8B)	$5–20	Some
Cloud GPU (24/7)	$290	Full
OpenAI API (GPT-4o)	$60–200+	Some
ChatGPT Plus	$20	None

Hardware → Model Fit

Your RAM / VRAM	Max Practical Model	Good Default
8 GB	4B quantized	Qwen3 4B Q4
16 GB	14B quantized	Qwen 3.5 9B Q4
32 GB	32B quantized	Qwen 3.8 27B Q4
48 GB+	70B quantized	Llama 3.3 70B Q4