kurobako is a command-line tool to benchmark black-box optimization algorithms.
Features: - It has the capability to support wide range of optimization problems: - various search sapce: - Continuous numerical, discrete numerical and categorical - Uniform distribution and log uniform distribution - Conditional - Constrainted problems - Multi-objective problems - Generating a markdown report and PNG plots from benchmarking results - Easy to add user-defined optimization problems and solvers - Simulating a concurrent environment in which an optimization process is executed by multiple workers simultaneously - Reproducible
Precompiled binaries for Linux are available in the [releases] page.
console
$ curl -L https://github.com/sile/kurobako/releases/download/${VERSION}/kurobako-${VERSION}.linux-amd64 -o kurobako
$ chmod +x kurobako
$ ./kurobako -h
If you have already installed Cargo, you can install kurobako by executing the following command:
console
$ cargo install kurobako
```console // Define solver. $ kurobako solver random | tee solver.json {"random":{}}
// Define problem. $ curl -OL http://ml4aad.org/wp-content/uploads/2019/01/fcnettabularbenchmarks.tar.gz $ tar xf fcnettabularbenchmarks.tar.gz && cd fcnettabularbenchmarks/ $ kurobako problem hpobench fcnetproteinstructuredata.hdf5 | tee problem.json {"hpobench":{"dataset":"fcnetproteinstructuredata.hdf5"}}
// Run benchmark. $ kurobako studies --solvers $(cat solver.json) --problems $(cat problem.json) | kurobako run > result.json (ALL) [00:00:01] [STUDIES 10/10 100%] [ETA 0s] done
// Report the benchmark result. $ cat result.json | kurobako report ...abbrev...
// Plot the benchmark result. $ cat result.json | kurobako plot curve (PLOT) [00:00:01] [1/1 100%] [ETA 0s] done (dir="images/curve/") ```
Solvers: - Random Search - ASHA - Optuna
Problems: - NASBench - HPOBench - sigopt/evalset
"kurobako" is a Japanese translation of "black box".