dbcrossbar: Copy tabular data between databases, CSV files and cloud storage

Build Status

dbcrossbar moves large data sets between different databases and storage formats.

Some examples:

```sh

Copy from a CSV file to a PostgreSQL table.

dbcrossbar cp \ --if-exists=overwrite \ --schema=postgres-sql:mytable.sql \ csv:mytable.csv \ 'postgres://postgres@127.0.0.1:5432/postgres#my_table'

Upsert from a PostgreSQL table to BigQuery.

dbcrossbar cp \ --if-exists=upsert-on:id \ --temporary=gs://$GSTEMPBUCKET \ --temporary=bigquery:$GCLOUDPROJECT:tempdataset \ 'postgres://postgres@127.0.0.1:5432/postgres#mytable' \ bigquery:$GCLOUDPROJECT:mydataset.mytable ```

It can also convert between table schema formats, including PostgreSQL CREATE TABLE statements and BigQuery JSON schemas:

```sh

Convert a PostgreSQL CREATE TABLE statement to a BigQuery JSON schema.

dbcrossbar schema conv postgres-sql:mytable.sql bigquery-schema:mytable.json

Extract a schema from a CSV file and convert to Postgres CREATE TABLE.

dbcrossbar schema conv csv:data.csv postgres-sql:schema.sql ```

For more information, see the documentation.

Contributing

For more instructions about building dbcrossbar, running tests, and contributing code, see CONTRIBUTING.md.

We require nightly Rust. We lock a specific version of nightly Rust using the rust-toolchain file. If you want to update this, take a look at Rustup components history and choose the newest version with support for rls, clippy and rustfmt.

Running integration tests

You can run the regular test suite with cargo test, but to run the full integration tests, you'll need to do the following:

```sh

Run a local PostgreSQL on port 5432.

docker run --name postgres -e POSTGRESPASSWORD= -p 5432:5432 -d mdillon/postgis createdb -h localhost -U postgres -w dbcrossbartest export POSTGRESTESTURL=postgres://postgres:@localhost:5432/dbcrossbartest echo "create extension if not exists postgis;" | psql $POSTGRESTESTURL echo "create extension if not exists citext;" | psql $POSTGRESTESTURL echo "create schema if not exists testme1;" | psql $POSTGRESTEST_URL

Point to test databases and test buckets.

export GSTESTURL=gs://$MYGSTESTBUCKET/dbcrossbar/ export BQTESTDATASET=$MYBQROOT:test export S3TESTURL=s3://$MTS3TESTBUCKET/dbcrossbar/

This helps to ensure that we're not depending on our users to have set

a default gcloud project anywhere.

gcloud config unset project

These can be omitted if you don't want to test Redshift.

export REDSHIFTTESTURL=redshift://user:pass@server:port/db export REDSHIFTTESTIAMROLE=$MYIAMROLE export REDSHIFTTESTREGIION=$MYAWS_REGION

Needed for BigML. Does not work with AWSSESSIONTOKEN.

export AWSACCESSKEYID=... AWSSECRETACCESSKEY=... export BIGMLUSERNAME=... BIGMLAPI_KEY=...

Run the integration tests.

env RUSTBACKTRACE=1 RUSTLOG=warn,dbcrossbarlib=debug,dbcrossbar=debug \ cargo test --all -- --ignored --nocapture ```

License

Licensed under either of:

...at your option.

Contributions

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above without any additional terms or conditions.

Third-party libraries

dbcrossbar depends on a variety of third-party libraries, each with their own copyright and license. We have configured a deny.toml file that currently attempts to restrict our dependencies to MIT, Apache-2.0, BSD-3-Clause, BSD-2-Clause, CC0-1.0, ISC, OpenSSL and Zlib licenses, with a single MPL-2.0 dependency. But please verify the deny.toml file (and individual dependencies) to be certain, because details may change in the future. Each of these licenses imposes certain obligations on redistribution.