Mesh LLM: distributed AI computing on iroh

	Mesh LLM: distributed AI computing on iroh(iroh.computer)
	265 points by tionis 13 hours ago \| 59 comments
	tl;dr: Mesh LLM pools GPUs across multiple machines into a single OpenAI-compatible endpoint (localhost:9337/v1), routing inference locally, to peers, or splitting large models across nodes via layer-range pipelines. It's built on iroh, which handles authenticated QUIC connections and NAT traversal between nodes identified by public keys, with a custom gossip layer on top for peer discovery and model routing. The catalog includes 40+ models ranging from laptop-sized to 235B MoE, enabling private or public meshes without central servers or vendor lock-in.
	HN Discussion: ~Users lack sufficient hardware/VRAM to realistically benefit from the mesh setup as pictured ↓Skeptical about performance since consumer networks are far slower than local RAM for split models ↑Interested in applying distributed inference to smaller purpose-built models beyond coding LLMs ~Installation and compatibility have rough edges that prevent actual use •Curious whether there's a catch or asking for clarification/more info