A tool to create programming languages
Please install langen instead of langen_macro for this tool to work correctly!
Most of the work is done during compilation, so it should run quite fast
```rust use langen;
enum Tokens { #[token(r"[ \t\r]", ignore=true)] _Ignore,
#[token(r"let")]
Let,
#[token(r"=")]
Assign,
#[token(r";")]
Semicolon,
#[token(r"[0-9]+")]
#[token(r"0x[0-9A-F]+")]
IntLiteral,
#[token(r"[A-Za-z_]+")]
Identifier,
}
fn main() {
let tokens = Tokens::scan("let variable = 312;").unwrap();
let mut iter = tokens.iter();
asserteq!(iter.next(), Some(&Tokens::Let));
asserteq!(iter.next(), Some(&Tokens::Identifier));
asserteq!(iter.next(), Some(&Tokens::Assign));
asserteq!(iter.next(), Some(&Tokens::IntLiteral));
asserteq!(iter.next(), Some(&Tokens::Semicolon));
asserteq!(iter.next(), None);
}
``
To use langen, derivelangen::Langenon an enum. To define a token, add#[token()]to it. The first argument inside token will always be a regex. The tokens defined first will get priority (for exampleLethas a higher priority thanIdentifier, although they match the same input). You can also define multiple tokens for one enum variant, for them all to produce that variant. You can optionally addignore=true, so that the token doesn't add anything to the output. You can get the parsed tokens by callingscan(input)` on the enum.
Lexer
Parser
This project is licenced under the MIT licence