uscan

A universal source code scanner

/!\ work-in-progress

features

• configurable keywords, symbols and comments
• handles nested multi-line comments
• handles decimal (15), hexadecimal (0xf or 0xF) and binary (0b1111) literal numbers

usage

```rust const LUACONFIG: ScannerConfig = ScannerConfig { keywords: &[ "and", "break", "do", "else", "elseif", "end", "false", "for", "function", "if", "in", "local", "nil", "not", "or", "repeat", "return", "then", "true", "until", "while", ], symbols: &[ "...", "..", "==", "~=", "<=", ">=", "+", "-", "*", "/", "%", "^", "#", "<", ">", "=", "(", ")", "{", "}", "[", "]", ";", ":", ",", ".", ], singlelinecmt: Some("--"), multilinecmtstart: Some("--[["), multilinecmt_end: Some("]]"), };

let mut scannerdata = ScannerData::default(); let mut scanner = Scanner::default(); scanner.run(sourcecode, &LUACONFIG, &mut scannerdata)?; ```

=> you can now use the ScannerData struct in your parser to build your AST :

```rust pub enum TokenType { Symbol(String), Identifier(String), StringLiteral(String), NumberLiteral(String, Number), Keyword(String), Comment(String), // space Ignore, NewLine, Eof, Unknown, }

pub struct ScannerData { /// complete source code pub source: Vec, /// resulting list of tokens pub tokentypes: Vec, /// token start line in the source code pub tokenlines: Vec, /// token start offset from its line beginning pub tokenstart: Vec, /// token length in characters /// not always = token value's length. /// for example for TokenType::StringLiteral("aa") the value length is 2 but the token length including the quotes is 4 pub tokenlen: Vec, } ```