Vaporetto is a fast and lightweight pointwise prediction based tokenizer.
```rust use std::fs::File; use std::io::Read;
use vaporetto::{Model, Predictor, Sentence};
let mut f = File::open("model.bin").unwrap(); let mut modeldata = vec![]; f.readtoend(&mut modeldata).unwrap(); let (model, ) = Model::readslice(&model_data).unwrap(); let predictor = Predictor::new(model, false).unwrap();
let s = Sentence::from_raw("火星猫の生態").unwrap(); let s = predictor.predict(s);
println!("{:?}", s.totokenizedvec().unwrap()); // ["火星", "猫", "の", "生態"] ```
The following features are disabled by default:
kytea - Enables the reader for models generated by KyTea.train - Enables the trainer.portable-simd - Uses the portable SIMD API instead
of our SIMD-conscious data layout. (Nightly Rust is required.)charwise-daachorse - Uses the Charwise Daachorse instead of the standard version for faster prediction, although it can make to load a model file slower.The following features are enabled by default:
std - Uses the standard library. If disabled, it uses the core library instead.cache-type-score - Enables caching type scores for faster processing. If disabled, type scores are calculated in a straightforward manner.fix-weight-length - Uses fixed-size arrays for storing scores to facilitate optimization. If disabled, vectors are used instead.tag-prediction - Enables tag prediction.Licensed under either of
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.