Tokenizer-lib

Tokenization utilities for building parsers in Rust

Examples

Buffered token channel:

rust let mut btq = BufferedTokenQueue::new(); btq.push(Token(12, Span(0, 2))); btq.push(Token(32, Span(2, 4))); btq.push(Token(52, Span(4, 8))); assert_eq!(btq.next().unwrap(), Token(12, Span(0, 2))); assert_eq!(btq.next().unwrap(), Token(32, Span(2, 4))); assert_eq!(btq.next().unwrap(), Token(52, Span(4, 8))); assert_eq!(btq.next(), None);

(Multi-thread safe) Parallel token queue:

```rust let (mut sender, mut reader) = ParallelTokenQueue::new(); std::thread::spawn(move || { sender.push(Token(12, Span(0, 2))); sender.push(Token(32, Span(2, 4))); sender.push(Token(52, Span(4, 8))); });

asserteq!(reader.next().unwrap(), Token(12, Span(0, 2))); asserteq!(reader.next().unwrap(), Token(32, Span(2, 4))); asserteq!(reader.next().unwrap(), Token(52, Span(4, 8))); asserteq!(reader.next(), None); ```

Generator token queue:

```rust fn lexer(state: &mut u8, sender: &mut GeneratorTokenQueueBufferstate += 1; match state { 1 | 2 | 3 => { sender.push(Token(state * 2, ())) } _ => {} } }

let mut reader = GeneratorTokenQueue::new(lexer, 0);

asserteq!(reader.next().unwrap(), Token(2, ())); asserteq!(reader.next().unwrap(), Token(4, ())); asserteq!(reader.next().unwrap(), Token(6, ())); assert!(reader.next().isnone()); ```

Provides utilities such as peek, peek_n and scan for lookahead. Also expect_next for expecting a token value and conditional_next for advancing on a predicate.