pq is a tool which deserializes protobuf messages given a set of pre-compiled .fdset files. It can understand varint-delimited streams, and it can connect to Kafka.
pq is on crates.io: cargo install pq. You can also download a static binary from the releases page.
To set up, put your *.fdset files in ~/.pq:
$ protoc -o dog.fdset dog.proto
$ protoc -o person.fdset person.proto
$ cp *.fdset ~/.pq/
Pipe a single compiled protobuf message:
$ testbench.py "single()" | pq | jq
{
"age": 4,
"breed": "poodle",
"temperament": "excited"
}
Pipe a varint-delimited stream:
$ testbench.py "stream(limit=2)" | pq --stream="varint" | jq
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}
{
"id": 3,
"name": "khosrov"
}
Consume from a Kafka stream:
$ pq kafka my_topic --brokers=192.168.0.1:9092 --from-beginning --count=1 | jq
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}