Breaking the 1.58-bit Barrier for Ternary LLMs

	Breaking the 1.58-bit Barrier for Ternary LLMs(arxiv.org)
	238 points by matt_d 27 days ago \| 38 comments
	tl;dr: Ternary LLMs typically pack 5 weights per byte (1.625 bits/weight), but analysis of 29 models shows zeros make up to 51.5% of weights. BITCOS exploits this with a presence bitmap plus compacted sign vector, achieving 2−z bits/weight (as low as 1.485) and beating five-trit packing in 26 of 29 models. With optimized AVX-512/AVX2/Xe2 unpacking kernels, it delivers up to 1.28× faster matvec and 1.18–1.27× end-to-end decode throughput on CPUs and GPUs.
	HN Discussion: ↑Surprise that this adaptive encoding wasn't already the default approach ↑Excitement about implications for ASIC/hardware efficiency and embedded deployment ↓Skepticism about ternary quantization being worthwhile compared to alternatives like vector quantization •Curiosity or questions about whether variable-length encoding works in-memory vs just storage ~Suggestion that further compression gains are possible via arithmetic coding or entropy methods