AirLLM 70B inference with single 4GB GPU

	AirLLM 70B inference with single 4GB GPU(github.com)
	224 points by Anon84 1 day ago \| 80 comments
	tl;dr: AirLLM runs massive LLMs on tiny GPUs by loading only one layer (or one MoE expert) into VRAM at a time, streaming the rest from disk—enabling Llama 70B on 4GB, Llama 3.1 405B on 8GB, DeepSeek-V3 (671B) on ~12GB, and Kimi K3 (2.8T) on under 4GB, without quantization or pruning. It works via a one-line `AutoModel.from_pretrained()` call with most popular open models (Llama, Qwen, DeepSeek, Mistral, Gemma, etc.), with optional 4/8-bit block-wise compression for ~3x speedup. The tradeoff is heavy disk I/O and slow inference throughput.
	HN Discussion: ~Extremely slow inference makes it impractical for interactive use, only viable for batch jobs ↓Skepticism that this offers meaningful advantages over existing quantization/streaming solutions like llama.cpp ↓Dismissive that the slow throughput makes the technical achievement essentially useless •Confusion about what the project actually adds and how it works in practice ~Appreciation for the clever layer-streaming approach while questioning speed tradeoffs vs quantization