pq

protobuf to json deserializer, written in Rust

pq is a tool which deserializes protobuf messages given a set of pre-compiled .fdset files. It can understand varint/leb128-delimited streams, and it can connect to Kafka.

pq will pretty-print when outputting to a tty, but you should pipe it to jq for more fully-featured json handling.

Download

pq is on crates.io: cargo install pq. You can also download a static binary from the releases page.

Usage

To set up, put your *.fdset files in ~/.pq or /etc/pq or an alternate directory specified with the FDSET_PATH= env var:

$ protoc -o dog.fdset dog.proto $ protoc -o person.fdset person.proto $ cp *.fdset ~/.pq/

Pipe a single compiled protobuf message:

$ pq --msgtype com.example.dog.Dog <./tests/samples/dog { "age": 4, "breed": "poodle", "temperament": "excited" }

Pipe a varint or leb128 delimited stream:

$ pq --msgtype com.example.dog.Dog --stream varint <./tests/samples/dog_stream { "age": 10, "breed": "gsd", "temperament": "aggressive" }

Consume from a Kafka stream:

$ pq kafka my_topic --brokers 192.168.0.1:9092 --beginning --count 1 --msgtype com.example.dog.Dog { "age": 10, "breed": "gsd", "temperament": "aggressive" }

Convert a Kafka stream to varint-delimited:

``` $ pq kafka my_topic --brokers=192.168.0.1:9092 --count 1 --convert varint |\

pq --msgtype com.example.dog.Dog --stream varint { "age": 10, "breed": "gsd", "temperament": "aggressive" } ```

Compile without kafka

To compile pq without kafka support, run:

$ cargo build --no-default-features