Rust and mmap
The library is published on crates.io as mmapjsonfile and can help with counting and filtering json files with records all of which as symmetric in their structure ( json array of objects ) as the format below.
[{..}, {..}, ..]
The idea of using memory mapped i/o is check the performance while filtering and creating another file etc from rust while putting serde in harmsway :grin:.
Here is the best read on the topic from Linux forums.
Functionality
- Count the number of records in JSON file.
- Count the number of records with filter
- Filter the JSON file with a condition ( provided by the caller ) and save it to a file specified.
- Distinct values of a key.
Test
The airports JSON has been taken from the location. It needs to be downloaded and put in the data/ directory.
Thanks to the original mmap lib.
General performance seems to be of the order below.
( all tests below are run in a macbook )
cargo test --release -- --nocapture --test-threads 1
Debug
- counttestsimplenestedjson: 256.37µs seconds for counting 1 records
- counttestsimplewithfilter_json: 322.471µs seconds.
- counttestsimplenestedwithfilterjson: 396.664µs seconds
- counttestsimple_json: 258.594µs seconds for counting 1 records
- filteroutjsonnoresults: 675.752µs seconds.
- filteroutjsonbyvalue: 785.313µs seconds.
- count_airports: 690.802302ms seconds.
- countwithfilter_airports: 3.913697422s seconds for filtering 57265 records
- filteroutairportsincountry 3.91415908s: seconds for filtering 57265 records
- filteroutairportsnoresults: 3.922528546s seconds for filtering 57265 records
Release (16MB json - 57,265 records)
- counttestsimplenestedjson: 147.219µs seconds for counting 1 records
- counttestsimplewithfilter_json: 164.697µs seconds.
- counttestsimple_json: 151.941µs seconds for counting 1 records
- counttestsimplenestedwithfilterjson: 339.415µs seconds
- filteroutjsonnoresults: 631.994µs seconds.
- filteroutjsonbyvalue: 888.325µs seconds.
- count_airports: 20.457554ms seconds.
- filteroutairportsnoresults: 347.208947ms seconds seconds for filtering 57265 records
- countwithfilter_airports: 349.70095ms seconds seconds for filtering 57265 records
- filteroutairportsincountry: 354.040548ms seconds for filtering 57265 records
Release (1.1 GB - appended 16MB json multiple times - 4,065,815 records)
- countairports ... countairports: 1.079399532s seconds.
- counttestsimplejson ... counttestsimplejson: 108.995µs seconds for counting 1 records
- counttestsimplenestedjson ... counttestsimplenestedjson: 73.032µs seconds for counting 1 records
- counttestsimplenestedwithfilterjson ... counttestsimplenestedwithfilterjson: 96.173µs seconds
- counttestsimplewithfilterjson ... counttestsimplewithfilterjson: 70.349µs seconds.
- countwithfilterairports ... countwithfilterairports: 20.170824135s seconds.
- filteroutairportsincountry ... filteroutairportsincountry 20.252878728s: seconds for filtering 57265 records
- filteroutairportsnoresults ... filteroutairportsnoresults: 20.181562593s seconds.
- filteroutjsonbyvalue ... filteroutjsonbyvalue: 547.087µs seconds.
- filteroutjsonnoresults ... filteroutjsonnoresults: 490.056µs seconds.
- writedistinctfields ... writedistinctfields: 460.137µs seconds.
- writedistinctfieldslargejson ... writedistinctfields: 26.096326613s seconds.