quick-protobuf

A pure Rust library to serialize/deserialize protobuf files.

Documentation

Description

This library intends to provide a simple yet fast (minimal allocations) protobuf parser implementation.

It provides both: - pb-rs, a code generation tool: - each .proto file will generate a minimal rust module (one function to read, one to write, and one to compute the size of the messages) - each message will generate a rust struct where:

| **Proto**                    | **Rust**                |
|------------------------------|-------------------------|
| bytes                        | `Cow<'a, [u8]>`         |
| string                       | `Cow<'a, str>`          |
| other scalars                | rust primitive          |
| repeated                     | `Vec`                   |
| repeated, packed, fixed size | `Cow<'a, [M]>`          |
| optional                     | `Option`                |
| message                      | `struct`                |
| enum                         | `enum`                  |
| map                          | `HashMap`               |
| oneof Name                   | `OneOfName` enum        |
| nested `m1`                  | `mod_m1` module         |
| package `a.b`                | `mod_a::mod_b` modules  |
| import file_a.proto          | `use super::file_a::*` |

• no need to use google protoc tool to generate the modules
  • quick-protobuf, a protobuf file parser:
• this is the crate that you will typically refer to in your library. The generated modules will assume it has been imported.
• it acts like an event parser, the logic to convert it into struct is handle by pb-rs

Example: protobuf_example project

• 1. Use pb-rs binary, located into codegen directory to automatically generate a foobar.rs module from a foobar.proto proto file

sh git clone https://github.com/tafia/quick-protobuf cd quick-protobuf/codegen cargo run ../../protobuf_example/foo_bar.proto cd ../../protobuf_example

• 1. Add a dependency to quick-protobuf

```toml

Cargo.toml

[dependencies] quick-protobuf = "0.4.0" ```

• 1. Have fun

```rust // main.rs or lib.rs extern crate quick_protobuf;

mod foo_bar; // (see 1.)

use quick_protobuf::Reader;

// We will suppose here that Foo and Bar are two messages defined in the .proto file // and converted into rust structs // // FooBar is the root message defined like this: // message FooBar { // repeated Foo foos = 1; // repeated Bar bars = 2; // } use foo_bar::{FooBar};

fn main() { // create a reader, which will parse the protobuf binary file and pop events // this reader will read the entire file into an internal buffer let mut reader = Reader::from_file("/path/to/binary/protobuf.bin") .expect("Cannot read input file");

// Use the generated module fns with the reader to convert your data into rust structs.
//
// Depending on your input file, the message can or not be prefixed with the encoded length
// for instance, a *stream* which contains several messages generally split them using this
// technique (see https://developers.google.com/protocol-buffers/docs/techniques#streaming)
//
// To read a message without a length prefix you can directly call `FooBar::from_reader`:
// let foobar = reader.read(FooBar::from_reader).expect("Cannot read FooBar message");
// 
// Else to read a length then a message, you can use:
let foobar: FooBar = reader.read(|r, b| r.read_message(b))
    .expect("Cannot read FooBar message");
// Reader::read_message uses `FooBar::from_reader` internally through the `MessageRead`
// trait.

println!("Found {} foos and {} bars!", foobar.foos.len(), foobar.bars.len());

} ```

Examples directory

You can find basic examples in the examples directory. - codegen_example: A basic write/read loop on all datatypes

Message <-> struct

The best way to check for all kind of generated code is to look for the codegenexample data: - definition: datatypes.proto - generated code: data_types.rs

Proto definition

``` enum FooEnum { FIRSTVALUE = 1; SECONDVALUE = 2; }

message BarMessage { required int32 brequiredint32 = 1; }

message FooMessage { optional int32 fint32 = 1; optional int64 fint64 = 2; optional uint32 fuint32 = 3; optional uint64 fuint64 = 4; optional sint32 fsint32 = 5; optional sint64 fsint64 = 6; optional bool fbool = 7; optional FooEnum fFooEnum = 8; optional fixed64 ffixed64 = 9; optional sfixed64 fsfixed64 = 10; optional fixed32 ffixed32 = 11; optional sfixed32 fsfixed32 = 12; optional double fdouble = 13; optional float ffloat = 14; optional bytes fbytes = 15; optional string fstring = 16; optional FooMessage fselfmessage = 17; optional BarMessage fbarmessage = 18; repeated int32 frepeatedint32 = 19; repeated int32 frepeatedpacked_int32 = 20 [ packed = true ]; } ```

Generated structs

```rust

[derive(Debug, PartialEq, Eq, Clone, Copy)]

pub enum FooEnum { FIRSTVALUE = 1, SECONDVALUE = 2, }

[derive(Debug, Default, PartialEq, Clone)]

pub struct BarMessage { // all fields are owned: no lifetime parameter pub brequiredint32: i32, }

[derive(Debug, Default, PartialEq, Clone)]

pub struct FooMessage<'a> { // has borrowed fields: lifetime parameter pub fint32: Option, pub fint64: Option, pub fuint32: Option, pub fuint64: Option, pub fsint32: Option, pub fsint64: Option, pub fbool: Option, pub fFooEnum: Option, pub ffixed64: Option, pub fsfixed64: Option, pub ffixed32: Option, pub fsfixed32: Option, pub fdouble: Option, pub ffloat: Option, pub fbytes: Option Cow<[u8]> pub fstring: Option> // string -> Cow pub fselfmessage: Option>>, // reference cycle -> Boxed message pub fbarmessage: Option, pub frepeatedint32: Vec, // repeated: Vec pub frepeatedpacked_int32: Vec, // repeated packed: Vec } ```

Leverage rust module system

Nested Messages

message A { message B { // ... } }

As rust does not allow a struct and a module to share the same name, we use mod_Name for the nested messages. ```rust pub struct A { //... }

pub mod mod_A { pub struct B { // ... } } ```

Package

package a.b;

Here we could have used the same name, but for consistency with nested messages, modules are prefixed with mod_ as well. rust pub mod mod_a { pub mod mod_b { // ... } }

Why not rust-protobuf

This library is an alternative to the widely used rust-protobuf.

Pros / Cons

• Pros

  • Much faster, in particular when working with string, bytes and repeated packed fixed size fields (no extra allocation)
  • No need to install protoc on your machine
  • No trait objects: faster/simpler parser
  • Very simple generated modules (~10x smaller) so you can easily understand what is happening

• Cons

  • Younger library
  • not battle tested, even if most rust-protobuf tests have been migrated here (see v2 and v3)
  • some missing functionalities
  • Not a drop-in replacement of rust-protobuf
  • everything being explicit you have to handle more things yourself (e.g. Option unwrapping, Cow management)

Codegen

Have a look at the different generated modules for the same .proto file: - rust-protobuf: 2371 loc - quick-protobuf: 302 loc

Benchmarks

An adaptation of rust-protobuf perftest is available and show, on these particular examples, that quick-protobuf is much faster than rust-protobuf.

Contribution

Any help is welcomed! (Pull requests of course, bug report, missing functionality etc...)

Licence

MIT