| Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s(github.com) | |
| 226 points by carloslfu 9 days ago | 107 comments | |
tl;dr: Slotstream is a single Swift binary that runs the 104GB Qwen3.8-Flash-Next MoE model on Apple Silicon Macs with far less RAM than the model size, streaming experts from SSD into a fixed cache pool shared across layers. On a 48GB M5 Pro it hits ~12 tok/s warm decode with ~32GB peak memory, and exposes Ollama/OpenAI-compatible APIs on port 11434. It works because MLX can't partially materialize mmap'd tensors (top-10 expert gather touches all 512), so it uses pread into managed slots instead. | |
HN Discussion:
| |