PGN Tokenizer

Goals

• Use zero-copy byte slices all the way through.
• No heap allocations

API

• A token based iterator that takes a byte slices and returns tokens that reference sub-slices

Example

``` extern crate memmap; extern crate rustpgntokenizer;

pub mod parser;

use memmap::{Mmap, Protection};

fn main() { let filemmap = Mmap::openpath("./lichessdbstandardrated2017-01.pgn", Protection::Read).unwrap(); let mut bytes: &[u8] = unsafe { filemmap.asslice() }; if bytes[0..3] == [239u8, 187u8, 191u8] { bytes = &bytes[3..]; } let results = parser::PGNTokenIterator{bytes: bytes}; let mut gamecount = 0; for x in results { if let parser::Token::Result() = x { gamecount += 1; } } println!("Games: {}", gamecount); } ```