Pedro Cuenca：tokenizers v1 (Rust) 快 3 至 30 倍，即將登陸 transformers
Pedro Cuenca 表示 tokenizers v1 (Rust) 快 3 至 30 倍，設有可選擇退出的 tokenization 訓練 crate、額外優化，以及廣泛基準測試。他表示即將登陸 transformers。
#tokenizers
查看全部AI快訊Tokenizers v1 first release candidate is out, claiming up to 30x faster tokenization
Lysandre (@LysandreJik) said Tokenizers v1’s first release candidate is out, citing up to 30x faster tokenization, thread scaling, lower latency, lower memory use, and a very small crate size. He linked details at https://huggingface.co/blog/tokenizers-v1.