TPC Tools

Command-line tools for invoking TPC-H and TPC-DS data generators in parallel and re-organizing the output files into directory structures that can be consumed by tools such as Apache Spark or Apache Arrow DataFusion/Ballista.

Also supports converting the output to Parquet.

TPC-DS

Install dependencies.

bash sudo apt install gcc make flex bison byacc git

Download data generator from https://www.tpc.org/tpcdocumentscurrentversions/currentspecifications5.asp

bash cd /path/to/DSGen-software-code-3.2.0rc1/tools make

Generate data.

```bash mkdir /tmp/tpcds/sf1000

cargo run --release -- generate --benchmark tpcds \ --scale 1000 \ --partitions 48 \ --generator-path /path/to/DSGen-software-code-3.2.0rc1/tools \ --output /tmp/tpcds/sf1000/ ```

Example output.

Generated TPC-DS data at scale factor 1000 with 48 partitions in: 6247.155671938s

Convert to Parquet

```bash mkdir /tmp/tpcds/sf1000-parquet

cargo run --release -- convert --benchmark tpcds \ --input /tmp/tpcds/sf1000/ --output /tmp/tpcds/sf1000-parquet/ ```

TPC-H

Install dependencies.

bash git clone git@github.com:databricks/tpch-dbgen.git cd tpch-dbgen make cd ..

Generate data.

```bash mkdir /tmp/tpch

cargo run --release -- generate --benchmark tpch \ --scale 1 \ --partitions 2 \ --generator-path ./tpch-dbgen/ \ --output /tmp/tpch ```

Convert data to Parquet

```bash mkdir /tmp/tpch-parquet

cargo run --release -- convert \ --benchmark tpch \ --input /tmp/tpch/ \ --output /tmp/tpch-parquet/ ```

Legal Stuff

TPC-H is Copyright © 1993-2022 Transaction Processing Performance Council. The full TPC-H specification in PDF format can be found here

TPC-DS is Copyright © 2021 Transaction Processing Performance Council. The full TPC-DS specification in PDF format can be found here

TPC, TPC Benchmark, TPC-H, and TPC-DS are trademarks of the Transaction Processing Performance Council.