Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s(github.com)
226 points by carloslfu 9 days ago | 107 comments
tl;dr: Slotstream is a single Swift binary that runs the 104GB Qwen3.8-Flash-Next MoE model on Apple Silicon Macs with far less RAM than the model size, streaming experts from SSD into a fixed cache pool shared across layers. On a 48GB M5 Pro it hits ~12 tok/s warm decode with ~32GB peak memory, and exposes Ollama/OpenAI-compatible APIs on port 11434. It works because MLX can't partially materialize mmap'd tensors (top-10 expert gather touches all 512), so it uses pread into managed slots instead.
HN Discussion:
  • ~README needs cleanup and better introduction for new users with zero context
  • Sharing personal experiences running similar models locally on Mac hardware with different tools
  • Enthusiasm for democratizing AI by running large models on modest hardware
  • Skepticism about performance claims on low-memory hardware configurations
  • Questions about practical aspects like context window, energy use, and model capability comparisons