cargo benchcmp

A small utility for comparing micro-benchmarks produced by cargo bench. The utility takes as input two sets of micro-benchmarks (one "old" and the other "new") and shows as output a comparison between each benchmark.

Dual-licensed under MIT or the UNLICENSE.

Installation

cargo benchcmp can be installed with cargo install:

$ cargo install cargo-benchcmp

The resulting binary should then be in $HOME/.cargo/bin.

Example output

Usage

First, run your benchmarks and save them to a file:

$ cargo bench > control

Next, apply the changes you'd like to test out, and then run the benchmarks and save them to a file again.

$ cargo bench > variable

Finally, use cargo benchcmp to compare the benchmark results!

$ cargo benchcmp control variable name dynamic-no-lazy-dfa ns/iter dynamic ns/iter diff ns/iter diff % bench::anchored_literal_long_match 169 (2,307 MB/s) 75 (5,200 MB/s) -94 -55.62% bench::anchored_literal_long_non_match 85 (4,588 MB/s) 61 (6,393 MB/s) -24 -28.24% bench::anchored_literal_short_match 158 (164 MB/s) 75 (346 MB/s) -83 -52.53% bench::anchored_literal_short_non_match 84 (309 MB/s) 61 (426 MB/s) -23 -27.38% bench::easy0_1K 318 (3,220 MB/s) 196 (5,224 MB/s) -122 -38.36% bench::easy0_1MB 257,205 (4,076 MB/s) 255,138 (4,109 MB/s) -2,067 -0.80% bench::easy0_32 82 (390 MB/s) 71 (450 MB/s) -11 -13.41% bench::easy0_32K 8,666 (3,781 MB/s) 5,392 (6,077 MB/s) -3,274 -37.78% bench::easy1_1K 293 (3,494 MB/s) 241 (4,248 MB/s) -52 -17.75% ...

If you want to compare the same benchmark run in multiple ways, reuse the names in different modules. Then your benchmark output will look like:

module1::ac_two_one_prefix_byte_random ... module2::ac_two_one_prefix_byte_random ...

You can then instruct benchcmp to compare the two modules by providing the two prefixes, followed by the file containing the output:

$ cargo benchcmp module1:: module2:: benchmark-output name dense_boxed ns/iter dense ns/iter diff ns/iter diff % ac_two_one_prefix_byte_random 21,041 (475 MB/s) 16,741 (597 MB/s) -4,300 -20.44% ac_two_one_prefix_byte_no_match 354 (28248 MB/s) 349 (28653 MB/s) -5 -1.41% ac_two_one_prefix_byte_every_match 150,678 (66 MB/s) 112,962 (88 MB/s) -37,716 -25.03% ac_two_diff_prefix 3,139 (3185 MB/s) 3,127 (3197 MB/s) -12 -0.38% ac_two_bytes 3,140 (3184 MB/s) 3,125 (3200 MB/s) -15 -0.48% ac_ten_one_prefix_byte_random 23,972 (417 MB/s) 19,495 (513 MB/s) -4,477 -18.68% ac_ten_one_prefix_byte_no_match 354 (28248 MB/s) 356 (28089 MB/s) 2 0.56% ac_ten_one_prefix_byte_every_match 150,636 (66 MB/s) 115,112 (86 MB/s) -35,524 -23.58% ac_ten_diff_prefix 108,137 (92 MB/s) 59,237 (168 MB/s) -48,900 -45.22% ac_ten_bytes 108,109 (92 MB/s) 59,331 (168 MB/s) -48,778 -45.12% ac_one_prefix_byte_random 20,476 (488 MB/s) 16,515 (605 MB/s) -3,961 -19.34% ac_one_prefix_byte_no_match 354 (28248 MB/s) 358 (27932 MB/s) 4 1.13% ac_one_prefix_byte_every_match 150,619 (66 MB/s) 114,608 (87 MB/s) -36,011 -23.91% ac_one_byte 354 (28248 MB/s) 356 (28089 MB/s) 2 0.56%

The tool supports basic filtering. For example, it's easy to see only improvements:

$ cargo benchcmp old new --improvements name dynamic-no-lazy-dfa ns/iter dynamic ns/iter diff ns/iter diff % bench::anchored_literal_long_match 169 (2,307 MB/s) 75 (5,200 MB/s) -94 -55.62% bench::anchored_literal_long_non_match 85 (4,588 MB/s) 61 (6,393 MB/s) -24 -28.24% bench::anchored_literal_short_match 158 (164 MB/s) 75 (346 MB/s) -83 -52.53% bench::anchored_literal_short_non_match 84 (309 MB/s) 61 (426 MB/s) -23 -27.38% bench::easy0_1K 318 (3,220 MB/s) 196 (5,224 MB/s) -122 -38.36% bench::easy0_1MB 257,205 (4,076 MB/s) 255,138 (4,109 MB/s) -2,067 -0.80%

Or only see regressions:

$ cargo benchcmp old new --regressions name dynamic-no-lazy-dfa ns/iter dynamic ns/iter diff ns/iter diff % bench::easy1_1MB 329,774 (3,179 MB/s) 334,872 (3,131 MB/s) 5,098 1.55% bench::match_class 84 (964 MB/s) 85 (952 MB/s) 1 1.19% bench::medium_1MB 2,034,481 (515 MB/s) 2,044,757 (512 MB/s) 10,276 0.51% bench::replace_all 149 153 4 2.68% bench_dynamic_compile::compile_huge 161,349 165,209 3,860 2.39% bench_dynamic_compile::compile_huge_bytes 18,050,519 18,795,770 745,251 4.13%

Many times, the difference in micro-benchmarks is just noise, so you can filter by percent difference:

$ cargo benchcmp old new --regressions --threshold 3 name dynamic-no-lazy-dfa ns/iter dynamic ns/iter diff ns/iter diff % bench_dynamic_compile::compile_huge_bytes 18,050,519 18,795,770 745,251 4.13%