Pedro Cuenca：tokenizers v1（Rust）快 3 到 30 倍，即将登陆 transformers
Pedro Cuenca 表示 tokenizers v1（Rust）快 3 到 30 倍，具备可选退出的 tokenization 训练 crate、额外优化以及详尽的基准测试。他表示即将登陆 transformers。
#tokenizers
查看全部AI快讯Tokenizers v1 first release candidate is out, claiming up to 30x faster tokenization
Lysandre (@LysandreJik) said Tokenizers v1’s first release candidate is out, citing up to 30x faster tokenization, thread scaling, lower latency, lower memory use, and a very small crate size. He linked details at https://huggingface.co/blog/tokenizers-v1.