Lesser-known but useful
buf CLI is a multipurpose tool for working with Protocol Buffers. The core
features include linting, formatting, breaking change detection, code generation, dependency
But the goal of this post is to highlight some of the lesser-known commands I've personally found
useful. Emphasis on personally. This is an evolving post of the lesser-known
buf commands and
notes on how I use them.
There is a large number of developers working on tooling for other developers, and I think it's important to put yourself in the shoes of the end-user to understand the tooling and the gaps. Someone (I think Rich Harris, the svelte guy) put it nicely:
if all you are doing is working on the framework, instead of building things with the framework, you might lose touch with the needs on the ground.
Assuming correct tooling is installed, all commands are expected to be copy/pastable.
This is one of my favorite commands, it allows you to encode/decode a binary serialized message to/from a human-readable format (JSON). Super handy for development and testing.
JSON to binary¶
For the curious,
jo is small utility for creating JSON objects, read
INPUTvariable is a BSR module, but can be another input format such as a directory of .proto files or git.
binary to JSON¶
Note, by default the fields in the JSON output are formatted as camelCase. But I really prefer viewing proto field names in snake_case1.
I wish the
convertcommand would allow modifying the JSON output, effectively what I'd like is this instead of the default wire-encoding format.
For now, I have found a handy
jqfunction to convert the map keys from
snake_case. Don't @ me.
Alright, gather around .
The Buf Schema Registry API is built with Connect, allowing us to send plain http
requests with our favorite tools. Bonus, the request can have the
Content-Type header set to
application/json to match the encoding of the body.
I'll expand on the rpc
GetImage endpoint in another post, but for now all we need to know is this
endpoint returns a Buf image, think
The response here contains an image for acme/petapis:
But what if we wanted to send the serialized binary request message to the API
application/proto), and then decode that binary output to JSON using
DATA=$(jo owner=acme repository=petapis reference=main exclude_source_info=true) INPUT="buf.build/bufbuild/buf" TYPE="buf.alpha.registry.v1alpha1.GetImageRequest" BINARY=$(buf convert $INPUT --type=$TYPE --from -#format=json <<<$DATA) API="https://api.buf.build" RPC="buf.alpha.registry.v1alpha1.ImageService/GetImage" http POST $API/$RPC content-type:application/proto < <(echo -n $BINARY) # HTTP/1.1 200 OK # accept-encoding: gzip # content-encoding: gzip # content-length: 565 # content-type: application/proto # date: Tue, 15 Nov 2022 03:11:28 GMT # +-----------------------------------------+ # | NOTE: binary data not shown in terminal | # +-----------------------------------------+
Sweet, now let's pipe that response back to
buf convert, output it as JSON, jq -c and md5 the
result. It'll be the exact same as the first example where we sent the JSON payload.
This may seem like a silly example, but
buf convert command has been extremely useful for:
- Creating serialized binary messages ad-hoc
- Converting those messages to a human-readable format
Sometimes it's useful to export a module (think raw Protobuf source files) to your local filesystem. Especially useful if the input is a module on the BSR registry.
If you're a Go developer, this is kind of like
go mod vendor
Note the hash of the Buf image is the exact same as the one we retrieved from the BSR API in the
buf convert example above!
Handy for listing all protobuf files. The
--include-imports flag is useful for including files
that are imports.
I wish this command supported
--log-format=json and marked each file as an import or not. But you
can accomplish this with this
buf build and a bit of
buf build buf.build/acme/petapis --exclude-source-info -o -#format=json |\ jq -r '.file | .name + " " + (.bufExtension.isImport|tostring)' |\ column -t # google/protobuf/duration.proto true # google/type/datetime.proto true # google/type/money.proto true # payment/v1alpha1/payment.proto true # pet/v1/pet.proto false