Rust and mmap

The library is published on crates.io as mmapjsonfile and can help with counting and filtering json files with symmetric records as the format below.

[{..}, {..}, ..]

The idea of using memory mapped i/o is check the performance while filtering and creating another file etc from rust while putting serde in harmsway :grin:.

Here is the best read on the topic from Linux forums.

Functionality

• Count the number of records in JSON file.
• Count the number of records with filter
• Filter the JSON file with a condition ( provided by the caller ) and save it to a file specified.

Test

The airports JSON has been taken from the location. It needs to be downloaded and put in the data/ directory.

Thanks to the original mmap lib.

General performance seems to be of the order below.

Debug

• counttestsimplenestedjson: 256.37µs seconds for counting 1 records
• counttestsimplewithfilter_json: 322.471µs seconds.
• counttestsimplenestedwithfilterjson: 396.664µs seconds
• counttestsimple_json: 258.594µs seconds for counting 1 records
• filteroutjsonnoresults: 675.752µs seconds.
• filteroutjsonbyvalue: 785.313µs seconds.
• count_airports: 690.802302ms seconds.
• countwithfilter_airports: 3.913697422s seconds for filtering 57265 records
• filteroutairportsincountry 3.91415908s: seconds for filtering 57265 records
• filteroutairportsnoresults: 3.922528546s seconds for filtering 57265 records

Release

• counttestsimplenestedjson: 147.219µs seconds for counting 1 records
• counttestsimplewithfilter_json: 164.697µs seconds.
• counttestsimple_json: 151.941µs seconds for counting 1 records
• counttestsimplenestedwithfilterjson: 339.415µs seconds
• filteroutjsonnoresults: 631.994µs seconds.
• filteroutjsonbyvalue: 888.325µs seconds.
• count_airports: 20.457554ms seconds.
• filteroutairportsnoresults: 347.208947ms seconds seconds for filtering 57265 records
• countwithfilter_airports: 349.70095ms seconds seconds for filtering 57265 records
• filteroutairportsincountry: 354.040548ms seconds for filtering 57265 records