token-iter

A Rust library crate that facilitates writing tokenizers for textual languages.

Please read the API documentation here.

To use it, add this to your Cargo.toml:

toml [dependencies] token-iter = "0.10.0"

And put this in your code:

rust use token_iter::*;

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual-licensed as above, without any additional terms or conditions.

See CONTRIBUTING.md.