jayce

Jayce is a simple tokenizer

Example

```rust use jayce::Tokenizer;

fn main() { let source = "let value = 5000$"; let duos = &[ ("keywords", r"^(let|const)"), ("whitespace", r"^\s+"), ("variable", r"^[a-zA-Z_]+"), ("price", r"^[0-9]+\$"), ("equals", r"^="), ("newline", r"^\n"), ]; let mut tokenizer = Tokenizer::new(source, duos);

while let Some(token) = tokenizer.eat() {
    println!("{:?}", token);
}

} ```

Result

rust,ignore Token { kind: Some("keywords"), value: "let", line: 1, column: 4 } Token { kind: Some("whitespace"), value: " ", line: 1, column: 5 } Token { kind: Some("variable"), value: "value", line: 1, column: 10 } Token { kind: Some("whitespace"), value: " ", line: 1, column: 11 } Token { kind: Some("equals"), value: "=", line: 1, column: 12 } Token { kind: Some("whitespace"), value: " ", line: 1, column: 13 } Token { kind: Some("price"), value: "5000$", line: 1, column: 18 }

Info

Reaching the end of source returns None

Unknown characters returns a Token with kind as None and value of the unknown character