This is a small package containing a simple string-tokenizer for the rust programming language. The package also contains a simple sentence-splitting iterator.
(The sentence splitter might be moved, once I find out where I want it).
Documentation
machtan.github.io/token-rs/token
Add the following to your Cargo.toml file
[dependencies.token] git = "https://github.com/machtan/token-rs"
```rust extern crate token;
let separators = vec![' ', '\n', '\t', '\r']; let source: &str = " Hello world \n How do you do\t-Finely I hope";
let mut tokenizer = tokenizer::Tokenizer::new(source.as_bytes(), separators); println!("Tokenizing..."); for token in tokenizer { println!("- Got token: {}", token.unwrap()); } println!("Done!"); ```
MIT (do what you want with it)