Skip to content

Lesser-known but useful buf commands

The buf CLI is a multipurpose tool for working with Protocol Buffers. The core features include linting, formatting, breaking change detection, code generation, dependency management, etc.

But the goal of this post is to highlight some of the lesser-known commands I've personally found useful. Emphasis on personally. This is an evolving post of the lesser-known buf commands and notes on how I use them.

There is a large number of developers working on tooling for other developers, and I think it's important to put yourself in the shoes of the end-user to understand the tooling and the gaps. Someone (I think Rich Harris, the svelte guy) put it nicely:

if all you are doing is working on the framework, instead of building things with the framework, you might lose touch with the needs on the ground.

Assuming correct tooling is installed, all commands are expected to be copy/pastable.

buf convert

This is one of my favorite commands, it allows you to encode/decode a binary serialized message to/from a human-readable format (JSON). Super handy for development and testing.

JSON to binary

For the curious, jo is small utility for creating JSON objects, read more here.

from file
jo pet_type="PET_TYPE_DOG" pet_id="7a56f3df" name="dante" > dog.json
INPUT="buf.build/acme/petapis" # (1)!
TYPE="pet.v1.Pet"

buf convert $INPUT --type=$TYPE --from=dog.json
  1. The INPUT variable is a BSR module, but can be another input format such as a directory of .proto files or git.
from stdin
DATA=$(jo pet_type="PET_TYPE_DOG" pet_id="7a56f3df" name="dante")
INPUT="buf.build/acme/petapis"
TYPE="pet.v1.Pet"

buf convert $INPUT --type=$TYPE --from -#format=json <<< $DATA

binary to JSON

from file
INPUT="buf.build/acme/petapis"
TYPE="pet.v1.Pet"
buf convert $INPUT --type=$TYPE --from dog.json --to=dog.bin

buf convert $INPUT --type=$TYPE --from dog.bin --to=-#format=json
from stdin
DATA=$(jo pet_type="PET_TYPE_DOG" pet_id="7a56f3df" name="dante")
INPUT="buf.build/acme/petapis"
TYPE="pet.v1.Pet"
buf convert $INPUT --type=$TYPE --from -#format=json <<< $DATA

buf convert $INPUT --type=$TYPE < <(cat dog.bin) |\
    jq '.|map_keys(camel_to_snake)' # (1)!

  1. Note, by default the fields in the JSON output are formatted as camelCase. But I really prefer viewing proto field names in snake_case1.

    I wish the convert command would allow modifying the JSON output, effectively what I'd like is this instead of the default wire-encoding format.

    protojson.MarshalOptions{
    UseProtoNames: true,
}

    For now, I have found a handy jq function to convert the map keys from camelCase to snake_case. Don't @ me.

from network

Alright, gather around .

The Buf Schema Registry API is built with Connect, allowing us to send plain http requests with our favorite tools. Bonus, the request can have the Content-Type header set to either application/proto or application/json to match the encoding of the body.

I'll expand on the rpc GetImage endpoint in another post, but for now all we need to know is this endpoint returns a Buf image, think FileDescriptorSet.

The response here contains an image for acme/petapis:

API="https://api.buf.build"
RPC="buf.alpha.registry.v1alpha1.ImageService/GetImage"

http POST $API/$RPC content-type:application/json \
    owner=acme repository=petapis reference=main exclude_source_info:=true |\
    jq -c '.image' | md5
# da5cf0da7a4eb9634a251d75947dcb75

But what if we wanted to send the serialized binary request message to the API (application/proto), and then decode that binary output to JSON using buf convert.

DATA=$(jo owner=acme repository=petapis reference=main exclude_source_info=true)
INPUT="buf.build/bufbuild/buf"
TYPE="buf.alpha.registry.v1alpha1.GetImageRequest"
BINARY=$(buf convert $INPUT --type=$TYPE --from -#format=json <<<$DATA)

API="https://api.buf.build"
RPC="buf.alpha.registry.v1alpha1.ImageService/GetImage"

http POST $API/$RPC content-type:application/proto < <(echo -n $BINARY)

# HTTP/1.1 200 OK
# accept-encoding: gzip
# content-encoding: gzip
# content-length: 565
# content-type: application/proto
# date: Tue, 15 Nov 2022 03:11:28 GMT
# +-----------------------------------------+
# | NOTE: binary data not shown in terminal |
# +-----------------------------------------+

Sweet, now let's pipe that response back to buf convert, output it as JSON, jq -c and md5 the result. It'll be the exact same as the first example where we sent the JSON payload.

TYPE="buf.alpha.registry.v1alpha1.GetImageResponse"

http POST $API/$RPC content-type:application/proto < <(echo -n $BINARY) |\
    buf convert $INPUT --type=$TYPE --to=-#format=json | jq -c '.image' | md5
# da5cf0da7a4eb9634a251d75947dcb75

This may seem like a silly example, but buf convert command has been extremely useful for:

  1. Creating serialized binary messages ad-hoc
  2. Converting those messages to a human-readable format

buf export

Sometimes it's useful to export a module (think raw Protobuf source files) to your local filesystem. Especially useful if the input is a module on the BSR registry.

If you're a Go developer, this is kind of like go mod vendor

buf export buf.build/acme/petapis -o petapis

tree petapis/
petapis/
├── google
│   └── type
│       ├── datetime.proto
│       └── money.proto
├── payment
│   └── v1alpha1
│       └── payment.proto
└── pet
    └── v1
        └── pet.proto

Note, by default this will include imports (in this example googleapis and paymentapis), so if you only want the module without its dependencies add --exclude-imports.

buf build

The official documentation for this command is really good. I find this command useful for debugging Buf images (again, think FileDescriptorSetss).

buf build buf.build/acme/petapis --exclude-source-info -o -#format=json |\
    jq -c |\
    md5
# da5cf0da7a4eb9634a251d75947dcb75

Note the hash of the Buf image is the exact same as the one we retrieved from the BSR API in the buf convert example above!

buf ls-files

Handy for listing all protobuf files. The --include-imports flag is useful for including files that are imports.

buf ls-files buf.build/acme/petapis

buf ls-files buf.build/acme/petapis --include-imports

I wish this command supported --log-format=json and marked each file as an import or not. But you can accomplish this with this buf build and a bit of jq:

buf build buf.build/acme/petapis --exclude-source-info -o -#format=json |\
    jq -r '.file[] | .name + " " + (.bufExtension.isImport|tostring)' |\
    column -t

# google/protobuf/duration.proto  true
# google/type/datetime.proto      true
# google/type/money.proto         true
# payment/v1alpha1/payment.proto  true
# pet/v1/pet.proto                false

  1. Heck, there is even a default lint rule for this, FIELD_LOWER_SNAKE_CASE. ↩