plrs

The multi-tool of lexical analysis and tokenization.

Install

pip install plrs

Build

maturin build

Examples

• zipfs-law

API

Global variables

EOF_TOKEN

Classes

``` Tokens Settings Token - part - token - setpart - settoken - str - repr

Lexer - new - charforward - skipovercharset - next ```

Functions

is_char_symbol is_char_operator is_char_whitespace is_char_numeric is_single_quote is_double_quote ends_token is_part_numeric tokenize