#tokenizers

Pedro Cuenca: tokenizers v1 (Rust) is 3 to 30 times faster, coming soon to transformers

Pedro Cuenca said tokenizers v1 (Rust) is 3 to 30 times faster, with an opt-out tokenization training crate, additional optimizations, and extensive benchmarks. He said it is coming soon to transformers.

Tokenizers v1 first release candidate is out, claiming up to 30x faster tokenization

Lysandre (@LysandreJik) said Tokenizers v1’s first release candidate is out, citing up to 30x faster tokenization, thread scaling, lower latency, lower memory use, and a very small crate size. He linked details at https://huggingface.co/blog/tokenizers-v1.