pq is a tool which deserializes protobuf messages given a set of pre-compiled .fdset files. It can understand varint/leb128-delimited streams, and it can connect to Kafka.
pq will pretty-print when outputting to a tty, but you should pipe it to jq for more fully-featured json handling.
pq is on crates.io: cargo install pq. You can also download a static binary from the releases page.
Included is pq_docker, a convenience wrapper which runs the latest version of pq in a Docker container.
It takes a path containing *.proto files directly as its first arg (so you can avoid the manual protoc command invocations to generate .fdset files):
sevagh:pq $ ./pq_docker ./tests/schemata/ --version
pq 1.0.0
sevagh:pq $ ./pq_docker ./tests/schemata/ --msgtype com.example.dog.Dog <tests/samples/dog
{"age":3,"breed":"gsd","temperament":"excited"}
To set up, put your *.fdset files in ~/.pq or /etc/pq or an alternate directory specified with the FDSET_PATH= env var:
$ protoc -o dog.fdset dog.proto
$ protoc -o person.fdset person.proto
$ cp *.fdset ~/.pq/
New in 1.0 You can now specify additional fdset directories or files via options:
$ pq --msgtype com.example.dog.Dog --fdsetdir ./tests/fdsets <./tests/samples/dog
$ pq --msgtype com.example.dog.Dog --fdsetfile ./tests/fdsets/dog.fdset <./tests/samples/dog
Pipe a single compiled protobuf message:
$ pq --msgtype com.example.dog.Dog <./tests/samples/dog
{
"age": 4,
"breed": "poodle",
"temperament": "excited"
}
Pipe a varint or leb128 delimited stream:
$ pq --msgtype com.example.dog.Dog --stream varint <./tests/samples/dog_stream
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}
Consume from a Kafka stream:
$ pq kafka my_topic --brokers 192.168.0.1:9092 --beginning --count 1 --msgtype com.example.dog.Dog
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}
Convert a Kafka stream to varint-delimited:
``` $ pq kafka my_topic --brokers=192.168.0.1:9092 --count 1 --convert varint |\
pq --msgtype com.example.dog.Dog --stream varint { "age": 10, "breed": "gsd", "temperament": "aggressive" } ```
To compile pq without kafka support, run:
$ cargo build --no-default-features
As pointed out in the issues, there are some Protobuf JSON specs. It would be nice if pq obeys them.
You can enable this with the --canonical flag.
For now, there is the rule of keys => lowerCamelCase which is enforced (in decode.rs). Other rules can be contributed gradually.