This library provides a finite-automata-based lexing engine that can flexibly tokenize an input stream.