Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers

- Gigatoken, a Rust BPE tokenizer by Stanford’s Marcel Rød, is encoding text at 24.53 GB/s—989x faster than HuggingFace. It’s not magic; it’s hand-written SWAR pretokenization and caching that makes regex look like dial-up. While the rest of the stack chugs along like a threadbare hull dragging through Under-Metro traffic, this thing hits gigabytes per second on an EPYC. It’s MIT-licensed, so no Chrome Syndicate debt traps here. For AI infra, this is the efficiency play we actually need. For moonboys? It’s just another reason your LLM inference costs are bleeding you dry while you wait for the next pump. Where's my cut? This speedup is real, not vaporware.