Crawler Data Client

This is a cli tool to help automate the market data distribution process.

Setup

First make sure rust is installed on the system with rustup.

Then install the cli using:

cargo install crawler_data_client

In order to decompress the files, you also need to have a working installation of the zstd package.

Usage

``` crawlerdataclient --address

--username --password --crawler-type --exchange --market --symbol --month --output

OPTIONS: -h, --help (display help text) -a, --address

---------- (ftp server address, eg 12.34.56.78:21. Usually the port is 21) -u, --username -------- (your ftp login username) -p, --password -------- (your ftp login password) -c, --crawler-type (options: trades, l2-events, l2-topk, bbo, fundingrate, ticker (raw only)) -e, --exchange -------- (options: binance, ftx, bybit) -m, --market ------------ (options: spot, linearswap, inverse_swap) -s, --symbol ------------ (eg. BTCUSDT) -o, --output ------------ (the local download destination folder, eg ~/Downloads/crawler-data) --month -------------- (the year-month in ISO, eg 2022-08) --day ------------------ (optional. the day of the month, eg. 25. if not provided, will download whole month) --decompress ----------------- (optional. if set, the data will be automatically decompressed into csv) --raw ------------------------ (optional. download the raw data install of parsed) ```