# Pedro Cuenca: tokenizers v1 (Rust) is 3 to 30 times faster, coming soon to transformers

Pedro Cuenca said tokenizers v1 (Rust) is 3 to 30 times faster, with an opt-out tokenization training crate, additional optimizations, and extensive benchmarks. He said it is coming soon to transformers.

Language: en
Time zone: America/Los_Angeles
Current time (UTC, cached as of): 2026-11-10T07:32:00.000Z
Query date (America/Los_Angeles): 2026-11-09

HTML: https://didcodexreset.com/news/5b7d54cac1d44e81ad0bb534.html

Content language: en
Localization state: sameLanguage

Source: Pedro Cuenca · Published 9/21/2026, 09:23:42

Pedro Cuenca (@pcuenq) said **tokenizers v1 (Rust)** is **3 to 30 times faster**, with an **opt-out tokenization training crate**, additional algorithms and optimizations, and extensive benchmarks.

He said it is **coming soon to transformers**, and linked a Hugging Face blog post: [https://huggingface.co/blog/tokenizers-v1](https://huggingface.co/blog/tokenizers-v1).

Tags: tokenizers, transformers, Hugging Face, Rust

[View original post](https://x.com/pcuenq/status/2102039519566729378)