A powerful yet simple text tokenizer for your everyday needs!
rust
let options = nipah_tokenizer::options::default();
rust
let text = "Hello, World!".to_string();
let tokens = tokenize(text, &options); ```
Output will be like:
["Hello": Id] [Comma] ["World": Id] [Exclamation]
You can use it as your hearth desires, and it is pretty versatile
Happy coding!