xtoken

Iterator based no_std XML Tokenizer using memchr.

Design Goals

• Operates on byte slices
• Minimal Validation
• Support for inline DTD declaration
• Partition whole input into non-empty spans

Example

```rust use xtoken::{Token, Tokenizer};

let tokens = Tokenizer::new(b"Hello World!").collect::>(); assert_eq!(&tokens, &[ Token::Element(b""), Token::Span(b"Hello World!"), Token::ElementEnd(b""), ]);

let tokens = Tokenizer::new(b"").collect::>(); assert_eq!(&tokens, &[ Token::Decl(b"") ]); ```