DataFusion Proto

DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

This crate is a submodule of DataFusion that provides a protocol buffer format for representing query plans and expressions.

Serializing Expressions

Based on examples/expr_serde.rs

```rust use datafusioncommon::Result; use datafusionexpr::{col, lit, Expr}; use datafusion_proto::bytes::Serializeable;

fn main() -> Result<()> { // Create a new Expr a < 32 let expr = col("a").lt(lit(5i32));

// Convert it to an opaque form
let bytes = expr.to_bytes()?;

// Decode bytes from somewhere (over network, etc.)
let decoded_expr = Expr::from_bytes(&bytes)?;
assert_eq!(expr, decoded_expr);
Ok(())

} ```

Serializing Logical Plans

Based on examples/logicalplanserde.rs

```rust use datafusion::prelude::*; use datafusioncommon::Result; use datafusionproto::bytes::{logicalplanfrombytes, logicalplantobytes};

[tokio::main]

async fn main() -> Result<()> { let ctx = SessionContext::new(); ctx.registercsv("t1", "testdata/test.csv", CsvReadOptions::default()) .await ?; let plan = ctx.table("t1").await?.intooptimizedplan()?; let bytes = logicalplantobytes(&plan)?; let logicalroundtrip = logicalplanfrombytes(&bytes, &ctx)?; asserteq!(format!("{:?}", plan), format!("{:?}", logicalroundtrip)); Ok(()) } ```

Serializing Physical Plans

Based on examples/physicalplanserde.rs

```rust use datafusion::prelude::*; use datafusioncommon::Result; use datafusionproto::bytes::{physicalplanfrombytes,physicalplantobytes};

[tokio::main]

async fn main() -> Result<()> { let ctx = SessionContext::new(); ctx.registercsv("t1", "testdata/test.csv", CsvReadOptions::default()) .await ?; let logicalplan = ctx.table("t1").await?.intooptimizedplan()?; let physicalplan = ctx.createphysicalplan(&logicalplan).await?; let bytes = physicalplantobytes(physicalplan.clone())?; let physicalroundtrip = physicalplanfrombytes(&bytes, &ctx)?; asserteq!(format!("{:?}", physicalplan), format!("{:?}", physicalround_trip)); Ok(()) }

```

Generated Code

The prost/tonic code can be generated by running, which in turn invokes the Rust binary located in gen

This is necessary after modifying the protobuf definitions or altering the dependencies of gen, and requires a valid installation of protoc.

bash ./regen.sh