GigaToken: ~1000x faster Language model tokenization

	GigaToken: ~1000x faster Language model tokenization(github.com)
	531 points by syrusakbary 18 hours ago \| 110 comments
	tl;dr: Gigatoken is a Rust-based BPE tokenizer that claims ~1000x speedups over HuggingFace tokenizers and tiktoken, hitting 24 GB/s on a 144-core EPYC and ~8 GB/s on an M4 Max, with drop-in compatibility APIs for both libraries. The gains come from hand-optimized SIMD pretokenization (replacing regex engines), aggressive pretoken caching, minimized Python interop, and per-architecture tuning for AVX512/AVX2/NEON. SentencePiece-based tokenizers (Gemma, Mistral, Llama 2) see smaller ~10-20x gains, and WordPiece isn't yet supported.
	HN Discussion: ↑Enthusiastic praise for the engineering achievement and optimization techniques ↓Tokenization is only 0.1% of inference time so the speedup has limited practical impact ↑The speedup is highly valuable for offline training data preparation at scale ↑Demonstrates that skilled human programming still matters and isn't obsolete due to AI •Curious about further improvements and integration possibilities like perfect hash tables or Rust crates