rs
Lightweight tokenizers for Rust with a CLI.
This is my first real project written in Rust, so bear with me if not everything is idiomatic!
The tokenizer is inspired by Spacy's tokenizer class and therefore has a very similar API.