pqrs

protobuf to json deserializer, written in Rust

pqrs is a tool which deserializes compiled protobuf messages given a set of pre-compiled .fdset files.

Now on crates.io

Contents

1. Usage
   1. Files
2. Message guessing
3. Portability with musl
   1. Download
4. Dependencies
5. Tests
   1. Linting
6. Goals
7. Todo

Usage

1. Put your *.fdset files in ~/.pq:

$ protoc -o dog.fdset dog.proto $ protoc -o person.fdset person.proto $ cp *.fdset ~/.pq/

1. Pipe a single compiled protobuf message to pq:

sevag:pqrs $ ./py-test/generate_random_proto.py | pq | jq { "age": 4, "breed": "poodle", "temperament": "excited" } sevag:pqrs $ ./py-test/generate_random_proto.py | pq | jq { "id": 2, "name": "raffi" }

Args

pqrs operates on stdin/stdout by default but also works with files:

• Use infile instead of stdin: pq /path/to/input.bin
• Use outfile instead of stdout: pq --outfile="/path/to/output.json"
• Explicitly specify fdsets dir: pq --fdsets="/fdsets/path/
• Named message (vs guessing): pq --msgtype="com.dog.Dog"

Message guessing

pqrs by default will guess the message type. You can make it use a specific type by passing the fully qualified message name, e.g. pq --type="com.example.dog.Dog".

Guessing strategy:

• For every message type discovered in ~/.pq/*.fdset, try to decode the message with it
• If the decode attempt has an error, skip this type
• If any fields are empty/null (serde_value::Value::Unit in the codebase), discard the decoded result
• If the decode is successful, store the decoded BTreeMap in a vector
• Display the BTreeMap from the vector which has the most fields

Since protobuf treats fields as positional, similar protos (e.g. Dog: , Person: ) are indistinguishable.

Portability with musl

First, clone and compile musl-gcc on your system:

$ git clone git://git.musl-libc.org/musl $ ./configure && make && sudo make install

Then, run make in this repo - this downloads a local ./rust toolchain with the x86_64-unknown-linux-musl target and runs ./rust/bin/cargo --target=x86_64-unknown-linux-musl to build pqrs.

The result is a static binary:

$ ldd ./target/x86_64-unknown-linux-musl/debug/pq not a dynamic executable $ $ file ./target/x86_64-unknown-linux-musl/debug/pq ./target/x86_64-unknown-linux-musl/debug/pq: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=3aa843efe79d0082aacb674a28e8d1ed8105a5e5, not stripped

Download

You can download a static musl-compiled binary from the releases page.

Dependencies

[dependencies] docopt = "0.7" rustc-serialize = "0.3" serde = "0.9.12" serde-value = "0.4.0" serde_json = "0.9.9" serde-protobuf = "0.5" protobuf = "1.2.1"

Tests

The testing tools are ./py-test for a Python random compiled protobuf generator (py-test README), and ./tests for Rust integration tests. The integration tests invoke the pqrs binary using std::process and checks return codes, stdout, etc. - inspired by the xsv test suite.

Linting

There is no linting in the Travis-CI job because it takes too long, but there is a make target (make lint). This is a bit hacky - it switches to rust-stable to run cargo fmt, rust-nightly to run cargo clippy (and then back to rust-stable). Run this before submitting a PR, or alternatively, run cargo fmt and cargo clippy however you prefer.

Goals

The original goal was to make a UNIX-y tool for generalized protobuf pretty-printing. Since jq already exists, I dropped the pretty-printing requirement and just output ugly JSON.

A new goal is handling a stream of protobuf data.

Todo

• Proper testing. CI with py-test/, Rust tests, etc.
• Figure out how to handle streams (delimiters, etc.?)
• Release on crates.io
• Host static binary on github releases for download