This Rust crate is a binding for the sentencepiece unsupervised text tokenizer. Installation requires that the sentencepiece C++ library is installed and discoverable through pkg-config.
sentencepiece
pkg-config
The crate documentation is available online.