Diesel Streamer

A tiny diesel add-on for streaming large tables. It currently allows streaming large serial tables using a cursor-based streaming strategy.

Installation

toml [dependencies] diesel-streamer = "0.1.2"

For tokio async runtime:

toml [dependencies] diesel-streamer = { version = "0.1.2", features = ["async"]}

Example Usage

Stream SomeTable that has a serial_field:

```rust use dieselstreamer::streamserial_table;

fn main() { use crate::schema::sometable::dsl::{sometable, serial_field};

let mut conn = pool.get().await.unwrap();

// with default chunk size of 500 streamserialtable!(sometable, serialfield, conn, |streamedtabledata: Vec| { // do work here dbg!(streamedtabledata); });

// specify chunk size, 130 streamserialtable!(sometable, serialfield, conn, 130, |streamedtabledata: Vec| { // do work here dbg!(streamedtabledata); });

// with cursor's beginning, 5. streamserialtable!(sometable, serialfield, conn, 130, 5, |streamedtabledata: Vec| { // do work here dbg!(streamedtabledata); });

// with cursor's end, 50, streamserialtable!(sometable, serialfield, conn, 130, 5, 50, |streamedtabledata: Vec| { // do work here dbg!(streamedtabledata); }); } ```

Defaults:

Chunk size: 500
Cursor's beginning: lowest value of serial_field in the table
Cursor's end: highest value of serial_field in the table

N/B: Generally, streaming should only be considered when there is a possibility of hitting OOM error when processing the table in question.

Contributing

Spin up a test db using docker-compose up or simply specify a DB url in .env as shown in .env.sample.

Run cargo test for tests.

License

^{Licensed under either of Apache License, Version
2.0 or MIT license at your option.}

_{Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in this crate by you, as defined in the Apache-2.0 license, shall
be dual licensed as above, without any additional terms or conditions.}