# Tokenizers v1 first release candidate is out, claiming up to 30x faster tokenization

Lysandre (@LysandreJik) said Tokenizers v1’s first release candidate is out, citing up to 30x faster tokenization, thread scaling, lower latency, lower memory use, and a very small crate size. He linked details at https://huggingface.co/blog/tokenizers-v1.

Language: ko
Time zone: Asia/Seoul
Current time (UTC, cached as of): 2026-11-10T07:02:00.000Z
Query date (Asia/Seoul): 2026-11-10

HTML: https://didcodexreset.com/ko/news/840c5ee3c29343179df3320f.html

Content language: en
Localization state: failed

출처: Lysandre · 2026. 9. 22. 00:32:21 게시

Lysandre (@LysandreJik) said **Tokenizers v1’s first release candidate** is out. He described it as a broad release, claiming **up to 30x faster tokenization**, scaling across threads, lower latency, lower memory use, and a microscopic crate size.

He pointed to a write-up at [huggingface.co/blog/tokenizers-v1](https://huggingface.co/blog/tokenizers-v1).

Tags: Tokenizers, Hugging Face, Release, Performance

[원문 게시물 보기](https://x.com/LysandreJik/status/2102035784434061735)