Bleeding Edge Inference — Cheat Sheet

Production Engines

vLLM vs SGLang vs TensorRT-LLM

Engine	Throughput (50 concurrent)	TTFT	Cold Start	Best For
TensorRT-LLM	2,100 tok/s	340ms	28 min compile	Max speed, stable model
SGLang	1,920 tok/s	360ms	58s	Agents, prefix reuse
vLLM	1,850 tok/s	380ms	62s	Widest support, best docs

H100 SXM5 80GB, Llama 3.3 70B FP8. Source: Spheron benchmarks Sep 2026.

The Techniques

12 Ways to Push Past “It Works”

Production

1. Speculative Decoding

Draft model guesses tokens; big model verifies in one pass. Correct guesses are free throughput. 2-5× speedup, identical output quality.

vllm serve Qwen/Qwen3-8B \
  --speculative-model Qwen/Qwen3-0.6B \
  --num-speculative-tokens 5

Free speed. Same quality. Already in vLLM, SGLang, and MLX (DFlash: 3.34× on Apple Silicon).

Production

2. SGLang RadixAttention

Caches shared prefixes in a radix tree. When 20 agent requests share a 50K-token system prompt, SGLang reuses the KV cache instead of recomputing it. 29% faster on prefix-heavy traffic.

docker run --gpus all -p 30000:30000 \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B \
  --host 0.0.0.0 --port 30000

If you're running coding agents or RAG, SGLang beats vLLM. Same API.

Production

3. TensorRT-LLM (Pre-Compiled)

NVIDIA's compiled inference engine. 15-30% faster than vLLM. Skip the 28-min compilation by using pre-built NGC containers.

docker pull nvcr.io/nvidia/tensorrt-llm/release:latest
docker run --rm -it --ipc=host --gpus=all \
  nvcr.io/nvidia/tensorrt-llm/release:latest

Max throughput when your model is stable. Pre-compiled containers skip the build.

Production

4. Two 3090s > One 4090

The RTX 3090 is the last consumer NVIDIA GPU with NVLink — 112 GB/s true VRAM pooling. 4090/5090 only have PCIe (32 GB/s). Two used 3090s ($1,600) give 48GB pooled. One 4090 ($2,200) gives 24GB.

vllm serve Qwen/Qwen3-32B --tensor-parallel-size 2
# or: llama-server -m model.gguf -sm tensor

For 70B models on consumer hardware, the older card wins because of NVLink.

Production

5. KV Cache Offloading

Spill attention state (KV cache) to system RAM or NVMe when context gets long. At ~300 KB/token, 200K context = 60-70 GB. Offloading lets a 24GB GPU serve 1M context.

vllm serve model --kv-cache-dtype fp8 \
  --enable-prefix-caching
# llama.cpp:
llama-server -m model.gguf \
  --cache-type-k q4_0 --cache-type-v q4_0

Reloading from NVMe is faster than recomputing prefill. Long-context agents need this.

Production

6. The Quantization Ladder

Not all quantization is equal. Q4_K_M is the universal default (92-95% quality, 28% size). Q5_K_M is the sweet spot for code. New: NVFP4 on RTX 5090 gives 2× throughput vs FP8.

Know your ladder. Q4 for serving, Q5 for code, FP8 for datacenter. Don't waste RAM on Q8.

Production

7. QLoRA + Unsloth

Quantize base model to 4-bit, train small adapter layers in FP16. Fine-tune a 7B model on 1,000 examples in 15 minutes on a 4090. 70B fits in 24GB (tight).

pip install unsloth
# 60-70% less VRAM than HuggingFace, 2-5× faster

Replace a $200/mo API subscription with a model that knows your domain. $0 compute cost on hardware you own.

Experimental

8. MoE Expert Offloading

Mixture of Experts models (Qwen3 30B-A3B) activate only 3B of 30B params per token. Inactive experts live in system RAM. A 235B MoE model runs on a 24GB GPU at ~20 tok/s.

llama-server -m qwen3-30b-a3b-Q4_K_M.gguf \
  --n-gpu-layers 99
# Only 3B active params hit GPU per forward pass

Run models 10× their labeled size. Slow but it works. A paper ran 35B on a 6GB GTX 580.

Experimental

9. LLMs on Phones

iPhone 17 Pro runs Qwen3 4B at ~40 tok/s fully offline (Private LLM, Core AI). Android via MLC LLM: ~3 tok/s on Snapdragon 8 Gen 3. Heat is the limit — throttles after 10 min.

Private AI in your pocket, zero network. The NPU is real hardware.

Experimental

10. Disaggregated Serving

Separate prefill (compute-bound) from decode (memory-bandwidth-bound) onto different GPUs. This is how Moonshot AI serves Kimi at scale. vLLM V1 has built-in support.

# Mooncake (open source):
# github.com/kvcache-ai/Mooncake
# NVIDIA Dynamo: RDMA-based KV cache transfer

The architecture after monolithic GPU serving. Needed at >1000 concurrent users.

Experimental

11. ExLlamaV3 + TabbyAPI

CUDA quantization engine optimized for consumer NVIDIA GPUs. EXL3 “Trellis” quantization — better quality than GGUF at the same bitrate. Best for multi-GPU consumer setups.

If you have 2× RTX 4090s and want max speed with best quantization quality, this is the move. NVIDIA-only.

Production

12. Tokens Per Watt

Apple M4 Ultra: highest efficiency for <32B models (120W total, silent). Jetson Orin Nano: 67 TOPS at 15W. RTX 5090 FP4: best consumer. H200: best datacenter.

Free win: switch FP16 → FP8. Same quality, half the cost. Just change a flag.

Reference

Quantization Ladder

Quant	Size vs FP16	Quality Kept	Speed	Verdict
Q2_K	25%	~82%	Fastest	Last resort. Broken for MoE.
Q3_K_S	30%	~88%	130 tok/s	Usable for chat, not code.
Q4_K_M	28%	~92-95%	100 tok/s	The universal default.
Q5_K_M	35%	~96%	85 tok/s	Sweet spot for code/reasoning.
Q6_K	42%	~98%	70 tok/s	Diminishing returns above Q5.
Q8_0	55%	~99%	55 tok/s	Overkill for serving.

Tokens Per Watt

Hardware	Efficiency	Notes
Apple M4 Ultra (MLX)	Highest for <32B	120W total system, silent
Jetson Orin Nano	Best edge	67 TOPS at 15W
RTX 5090 (FP4)	Best consumer	Blackwell FP4 tensor cores
H200	Best datacenter	4.8 TB/s HBM3e bandwidth

Companion to Hosting Open-Source Models at The Multiverse School. Sources: Spheron benchmarks, Jarvis Labs, NVIDIA NGC docs, Unsloth docs, cjtrowbridge.com.