Pedro Cuenca: tokenizers v1 (Rust) is 3 to 30 times faster, coming soon to transformers

Pedro Cuenca (@pcuenq) said tokenizers v1 (Rust) is 3 to 30 times faster, with an opt-out tokenization training crate, additional algorithms and optimizations, and extensive benchmarks.

He said it is coming soon to transformers, and linked a Hugging Face blog post: https://huggingface.co/blog/tokenizers-v1.