Skip to content

Lesser-known but useful buf commands

The buf CLI is a multipurpose tool for working with Protocol Buffers. The core features include linting, formatting, breaking change detection, code generation, dependency management, etc.

But the goal of this post is to highlight some of the lesser-known commands I've personally found useful. Emphasis on personally. This is an evolving post of the lesser-known buf commands and notes on how I use them.

There is a large number of developers working on tooling for other developers, and I think it's important to put yourself in the shoes of the end-user to understand the tooling and the gaps. Someone (I think Rich Harris, the sveltejs guy) put it nicely:

[if all you are doing is] working on the framework, instead of building things with the framework, you might lose touch with the needs on the ground.


Assuming correct tooling is installed, all commands are expected to be copy/pastable.

buf beta convert

This is one of my favorite commands, it allows you to encode/decode a binary serialized message to/from a human-readable format (JSON). Super handy for development and testing.

JSON to binary

For the curious, jo is small utility for creating JSON objects, read more here.

from file
jo pet_type="PET_TYPE_DOG" pet_id="7a56f3df" name="dante" > dog.json
INPUT="buf.build/acme/petapis" # (1)!
TYPE="pet.v1.Pet"

buf beta convert $INPUT --type=$TYPE --from=dog.json
  1. The INPUT variable is a BSR module, but can be another input format such as a directory of .proto files or git.
from stdin
DATA=$(jo pet_type="PET_TYPE_DOG" pet_id="7a56f3df" name="dante")
INPUT="buf.build/acme/petapis"
TYPE="pet.v1.Pet"

buf beta convert $INPUT --type=$TYPE --from -#format=json <<< $DATA

binary to JSON

from stdin
DATA=$(jo pet_type="PET_TYPE_DOG" pet_id="7a56f3df" name="dante")
INPUT="buf.build/acme/petapis"
TYPE="pet.v1.Pet"
buf beta convert $INPUT --type=$TYPE --from -#format=json --to dog.bin <<< $DATA

buf beta convert $INPUT --type=$TYPE < <(cat dog.bin) |\
    jq '.|map_keys(camel_to_snake)'

Note, by default the fields in the JSON output are formatted as camelCase. But I really prefer viewing proto field names in snake_case1.

I wish the convert command would allow modifying the JSON output, effectively what I'd like is this instead of the default wire-encoding format.

protojson.MarshalOptions{
    UseProtoNames: true,
}

For now, I have found this handy jq function to convert the map keys from camelCase to snake_case. Don't @ me.

from file
INPUT="buf.build/acme/petapis"
TYPE="pet.v1.Pet"
buf beta convert $INPUT --type=$TYPE --from dog.json --to=dog.bin

buf beta convert $INPUT --type=$TYPE --from dog.bin --to=-#format=json
from network

Alright, gather around . Since the Buf API is written in Connect we can send Content-type header as either application/json or application/proto.

I'll expand on the rpc GetImage endpoint in another post, but the neat thing is we can fetch a Buf image, think FileDescriptorSets, from this endpoint.

API="https://api.buf.build"
RPC="buf.alpha.registry.v1alpha1.ImageService/GetImage"

http POST $API/$RPC content-type:application/json \
    owner=acme repository=petapis reference=main exclude_source_info:=true |\
    jq -c '.image' | md5
# da5cf0da7a4eb9634a251d75947dcb75

But what if we wanted to send the serialized binary request message to the API (application/proto), and then decode that binary output to JSON using buf convert.

DATA=$(jo owner=acme repository=petapis reference=main exclude_source_info=true)
INPUT="buf.build/bufbuild/buf"
TYPE="buf.alpha.registry.v1alpha1.GetImageRequest"
BINARY=$(buf beta convert $INPUT --type=$TYPE --from -#format=json <<<$DATA)

API="https://api.buf.build"
RPC="buf.alpha.registry.v1alpha1.ImageService/GetImage"

http POST $API/$RPC content-type:application/proto < <(echo -n $BINARY)

# HTTP/1.1 200 OK
# accept-encoding: gzip
# content-encoding: gzip
# content-length: 565
# content-type: application/proto
# date: Tue, 15 Nov 2022 03:11:28 GMT
# +-----------------------------------------+
# | NOTE: binary data not shown in terminal |
# +-----------------------------------------+

Sweet, now let's pipe that response back to buf convert, output it as JSON, jq -c and md5 the result. It'll be the exact same as the first example where we sent the JSON payload.

TYPE="buf.alpha.registry.v1alpha1.GetImageResponse"

http POST $API/$RPC content-type:application/proto < <(echo -n $BINARY) |\
    buf beta convert $INPUT --type=$TYPE --to=-#format=json | jq -c '.image' | md5
# da5cf0da7a4eb9634a251d75947dcb75

This may seem like a silly example, but buf convert command has been extremely useful for:

  1. Creating serialized binary messages ad-hoc
  2. Converting those messages to a human-readable format

buf export

Sometimes it's useful to export a module (think raw Protobuf source files) to your local filesystem. Especially useful if the input is a module on the BSR registry.

If you're a Go developer, this is kind of like go mod vendor

buf export buf.build/acme/petapis

Note, by default this will include imports, so if you only want the module without its dependencies add --exclude-imports.

tree petapis/
petapis/
├── google
│   └── type
│       ├── datetime.proto
│       └── money.proto
├── payment
│   └── v1alpha1
│       └── payment.proto
└── pet
    └── v1
        └── pet.proto

buf build

The official documentation for this command is really good. I find this command useful for debugging Buf images (again, think FileDescriptorSetss).

buf build buf.build/acme/petapis --exclude-source-info -o -#format=json |\
    jq -c |\
    md5
# da5cf0da7a4eb9634a251d75947dcb75

Note the hash of the Buf image is the exact same as the one we retrieved from the BSR API in the buf convert example above!

buf ls-files

Handy for listing all protobuf files. The --include-imports flag is useful for including files that are imports.

buf ls-files buf.build/acme/petapis

buf ls-files buf.build/acme/petapis --include-imports

I wish this command supported --log-format=json and marked each file as an import or not. But you can accomplish this with this buf build and a bit of jq:

buf build buf.build/acme/petapis --exclude-source-info -o -#format=json |\
    jq -r '.file[] | .name + " " + (.bufExtension.isImport|tostring)' |\
    column -t

# google/protobuf/duration.proto  true
# google/type/datetime.proto      true
# google/type/money.proto         true
# payment/v1alpha1/payment.proto  true
# pet/v1/pet.proto                false

  1. Heck, there is even a default lint rule for this, FIELD_LOWER_SNAKE_CASE