| My local model setup on an M4 Pro Mac Mini(lws.io) | |
| 323 points by raybb 8 days ago | 192 comments | |
tl;dr: The author runs a local LLM setup on a 48GB M4 Pro Mac mini using Qwen3.6-35B-A3B and Gemma-4-E4B (both 4-bit quantized) via the oMLX inference server, with Tailscale connecting his iPhone and MacBook to access it through clients like Hermes, Apollo, and Raycast. He argues MoE models are key for consumer hardware since only active parameters (3B of 35B) matter for inference speed, achieving ~34 tok/s generation. The motivation: avoiding cloud API pricing volatility, privacy risks, and geopolitical restrictions, while handling ~80% of daily tasks locally at zero marginal cost. | |
HN Discussion:
| |