# Tokenizers v1 first release candidate is out, claiming up to 30x faster tokenization

Lysandre (@LysandreJik) said Tokenizers v1’s first release candidate is out, citing up to 30x faster tokenization, thread scaling, lower latency, lower memory use, and a very small crate size. He linked details at https://huggingface.co/blog/tokenizers-v1.

Language: ja
Time zone: Asia/Tokyo
Current time (UTC, cached as of): 2026-11-10T07:34:00.000Z
Query date (Asia/Tokyo): 2026-11-10

HTML: https://didcodexreset.com/ja/news/840c5ee3c29343179df3320f.html

Content language: en
Localization state: failed

出典：Lysandre · 2026/9/22 00:32:21 公開

Lysandre (@LysandreJik) said **Tokenizers v1’s first release candidate** is out. He described it as a broad release, claiming **up to 30x faster tokenization**, scaling across threads, lower latency, lower memory use, and a microscopic crate size.

He pointed to a write-up at [huggingface.co/blog/tokenizers-v1](https://huggingface.co/blog/tokenizers-v1).

Tags: Tokenizers, Hugging Face, Release, Performance

[元の投稿を見る](https://x.com/LysandreJik/status/2102035784434061735)