Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

	Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU(neomindlabs.com)
	297 points by neomindryan 20 hours ago \| 194 comments
	tl;dr: A hobbyist got Google's Gemma 4 26B MoE model running at ~5 tokens/sec on a 13-year-old dual Xeon E5-2690 v2 server with no GPU, by patching ik_llama.cpp to work on pre-AVX2 hardware. The core bug: the graph builder emitted fused MoE ops (MOE_FUSED_UP_GATE) that had no non-AVX2 compute path, silently leaving expert FFN outputs as uninitialized memory and producing fluent multilingual gibberish. The fix splits those fused ops into separate mul_mat_id calls plus a fused SILU-multiply when IQK is disabled. Claude did the C++ diagnosis; the author drove the debugging process.
	HN Discussion: ↑Optimistic that large MoE models will increasingly run on consumer hardware, sharing similar local setups ~Cost analysis suggests local inference is more expensive than cloud providers for this use case ↑Sharing own benchmarks and similar experiences running LLMs on old Xeon hardware ↓Claims better token/sec performance is achievable on similar old hardware ↑Appreciation for the hobbyist achievement and interest in replicating it