criterion_compare

Generate markdown comparison tables from Cargo Criterion benchmark output.

Currently, the tool is limited to Github Flavored Markdown (GFM), but adding new output types is simple.

Installation

```bash

If you don't have it already

cargo install cargo-criterion

This project

cargo install criterion-table ```

Usage

1. Ensure your benchmarks meet these basic criteria:

• Benchmark IDs are formatted in two to three sections separated by forward slashes (/)
  • The sections are used like this: <table_name>/<column_name>/[row_name]
  • Case is not currently altered, so set appropriately for display
  • Row name is the only optional field, and if left blank, all results will be a single blank row
  • If using a very basic benchmark_function you would only get a column name, which isn't valid
  • If using benchmark groups you will get two sections automatically
  • If using benchmark groups and BenchmarkId you will get all three sections automatically
• Benchmark data is not reordered, so ensure they execute in the order desired
  • Tables are ordered based on the order they are seen in the data (execution order)
  • The first column seen in each row will be the baseline everything else in that row is compared to, so benchmark execution order matters

Benchmark Example 1 - Manual ID

```rust use criterion::{blackbox, criteriongroup, criterion_main, Criterion};

[inline]

fn fibonacci(n: u64) -> u64 { match n { 0 => 1, 1 => 1, n => fibonacci(n-1) + fibonacci(n-2), } }

pub fn criterionbenchmark(c: &mut Criterion) { let id = "Fibonacci/Recursive Fib/20"; c.benchfunction(id, |b| b.iter(|| fibonacci(black_box(20)))); }

criteriongroup!(benches, criterionbenchmark); criterion_main!(benches); ```

Benchmark Example 2 - Benchmark Group with Parameters

```rust use criterion::{blackbox, BenchmarkId, criteriongroup, criterion_main, Criterion};

[inline]

fn fibonacci(n: u64) -> u64 { match n { 0 => 1, 1 => 1, n => fibonacci(n-1) + fibonacci(n-2), } }

pub fn criterionbenchmark(c: &mut Criterion) { let mut group = c.benchmarkgroup("Fibonacci");

for row in vec![10, 20] {
    let id = BenchmarkId::new("Recursive Fib", row);
    group.bench_with_input(id, &row, |b, row| {
        b.iter(|| fibonacci(black_box(*row)))
    });
}

group.finish();

}

criteriongroup!(benches, criterionbenchmark); criterion_main!(benches); ```

1. Run Benchmarks and Generate Markdown

This can be done in a couple of different ways:

Single Step

This method ensures all benchmarks are included in one step

```bash

Run all benchmarks and convert into the markdown all in one step

cargo criterion --message-format=json | criterion-table > BENCHMARKS.md ```

Multiple Steps

This method allows better control of order and which benchmarks are included

```bash

Execute only the desired benchmarks

cargo criterion --bench recursivefib --message-format=json > recursivefib.json cargo criterion --bench iterativefib --message-format=json > iterativefib.json

Reorder before converting into markdown

cat iterativefib.json recursivefib.json | criterion-table > BENCHMARKS.md ```

Adding New Output File Types

Currently, the tool is hardcoded to GFM, but it is easy to add a new output type via the Formatter trait by creating your own new binary project

1. Add this crate to your binary project as a library

toml [dependencies] criterion-table = "0.1"

1. Create a new type and implement criterion_table::Formatter

```rust use criterion_table::{ColumnInfo, Comparison, TimeUnit}; use flexstr::FlexStr;

pub trait Formatter { fn start(&mut self, buffer: &mut String, tables: &[&FlexStr]);

fn end(&mut self, buffer: &mut String);

fn start_table(&mut self, buffer: &mut String, name: &FlexStr, columns: &[ColumnInfo]);

fn end_table(&mut self, buffer: &mut String);

fn start_row(&mut self, buffer: &mut String, name: &FlexStr, max_width: usize);

fn end_row(&mut self, buffer: &mut String);

fn used_column(&mut self, buffer: &mut String, time: TimeUnit, pct: Comparison, max_width: usize);

fn unused_column(&mut self, buffer: &mut String, max_width: usize);

} ```

1. Create a process function that takes cargo-criterion raw JSON as input and outputs your format as a String:

NOTE: Replace GFMFormatter with your new formatter below

```rust use std::error::Error; use std::io::Read;

// This would be replaced with your formatter use criteriontable::formatter::GFMFormatter; use criteriontable::{CriterionTableData, RawCriterionData};

fn process(r: impl Read) -> Result> { let rawdata = RawCriterionData::fromreader(r)?; let data = CriterionTableData::fromraw(&rawdata)?;

// Pass your formatter to `make_tables` below
Ok(data.make_tables(GFMFormatter))

} ```

1. Save the returned String to the file type of your formatter

License

This project is licensed optionally under either:

• Apache License, Version 2.0, (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)