# Tokenizers v1 first release candidate is out, claiming up to 30x faster tokenization

Lysandre (@LysandreJik) said Tokenizers v1’s first release candidate is out, citing up to 30x faster tokenization, thread scaling, lower latency, lower memory use, and a very small crate size. He linked details at https://huggingface.co/blog/tokenizers-v1.

Language: zh-CN
Time zone: Asia/Shanghai
Current time (UTC, cached as of): 2026-11-10T06:09:00.000Z
Query date (Asia/Shanghai): 2026-11-10

HTML: https://didcodexreset.com/zh/news/840c5ee3c29343179df3320f.html

Content language: en
Localization state: failed

来源：Lysandre · 发布于 2026/9/21 23:32:21

Lysandre (@LysandreJik) said **Tokenizers v1’s first release candidate** is out. He described it as a broad release, claiming **up to 30x faster tokenization**, scaling across threads, lower latency, lower memory use, and a microscopic crate size.

He pointed to a write-up at [huggingface.co/blog/tokenizers-v1](https://huggingface.co/blog/tokenizers-v1).

Tags: Tokenizers, Hugging Face, Release, Performance

[查看主帖原文](https://x.com/LysandreJik/status/2102035784434061735)