Protocol Buffers
Describe a message once in a .proto file, then send it as compact bytes that older and newer services can both decode.
message User { string email = 1; repeated string roles = 2; }Protocol Buffers, usually shortened to protobuf, is a way to write down the shape of a message once and then send it between programs as compact bytes. The shape lives in a text file whose name ends in .proto. A compiler reads that file and writes Go code for it, so on the Go side you end up with an ordinary struct.
Messages and field numbers
A message is a named group of fields. Each field has a type, a name and a number, written string email = 1;. The number is the part that matters. Protobuf puts the number on the wire, never the name. That is why a program built from an older .proto file can still read a message containing fields it has never heard of: it skips the numbers it does not recognise.
What the wire format buys
JSON spells out every field name as text in every single message, and every number as digits. Protobuf writes a one-byte tag for field numbers 1 through 15, and packs integers into as few bytes as the value needs. Messages get smaller and decoding stops being text scanning. The price is that the bytes mean nothing on their own, so you cannot read them in a log.
Presence
In proto3 a plain scalar field has implicit presence: an empty string or a zero is simply not sent, and the decoder cannot tell "empty" from "never set". Writing optional in front of the field restores that difference. In Go the field then becomes a pointer, and the generated GetNickname() method is the short form of "dereference it, or hand me the zero value if it is absent".
Example
package main
import (
"encoding/json"
"fmt"
"log"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/wrapperspb"
)
// user is a plain Go struct, encoded as JSON for comparison.
type user struct {
Name string `json:"name"`
}
func main() {
// StringValue is a real generated message from wrappers.proto:
// one string field, numbered 1.
msg := wrapperspb.String("ada")
wire, err := proto.Marshal(msg) // the protobuf binary wire format
if err != nil {
log.Fatal(err)
}
text, err := json.Marshal(user{Name: "ada"})
if err != nil {
log.Fatal(err)
}
// 0a means field 1, length-delimited. 03 means three bytes follow.
fmt.Printf("proto: % x (%d bytes)\n", wire, len(wire))
fmt.Printf("json: %s (%d bytes)\n", text, len(text))
back := &wrapperspb.StringValue{}
if err := proto.Unmarshal(wire, back); err != nil {
log.Fatal(err)
}
fmt.Println("decoded:", back.GetValue())
}
// Output:
// proto: 0a 03 61 64 61 (5 bytes)
// json: {"name":"ada"} (14 bytes)
// decoded: adaWhen to use it
- Define the request and response types for an internal service once, then generate a Go server and a Python client from the same file.
- Cut bandwidth on a high-volume event stream where JSON field names are most of the payload.
- Add a field to a message that already-deployed services can keep reading without being rebuilt.
- Pin a stable contract between two teams who deploy on different days.
More examples
The .proto file
The whole definition of one message. `repeated` becomes a Go slice, `map` becomes a Go map, and `optional` becomes a pointer so that absence can be detected. The `reserved` line makes protoc reject any future attempt to use number 5 again, which is the only protection against a renamed field silently decoding old bytes. You would write this file first, before any Go code, and treat it as the contract between services.
syntax = "proto3";
package user.v1;
// Where the generated Go code goes: import path, then package name.
option go_package = "example.com/api/user/v1;userv1";
message User {
string email = 1; // implicit presence: "" is not sent at all
optional string nickname = 2; // explicit presence: absence is visible
repeated string roles = 3; // zero or more values, becomes []string
map<string, string> labels = 4;
reserved 5; // field 5 was deleted; nobody may reuse it
}What the generated Go looks like
The generated struct is a normal Go struct with struct tags recording the field number. Every field also gets a getter. Prefer the getters: they are the shorthand for a nil check plus, on an optional field, a pointer dereference, and they keep working when a caller passes a nil message. Reach for direct field access when you are building a message rather than reading one.
// Written by protoc-gen-go from the file above (abridged: the real struct
// also carries internal bookkeeping fields).
type User struct {
Email string `protobuf:"bytes,1,opt,name=email,proto3" json:"email,omitempty"`
Nickname *string `protobuf:"bytes,2,opt,name=nickname,proto3,oneof" json:"nickname,omitempty"`
Roles []string `protobuf:"bytes,3,rep,name=roles,proto3" json:"roles,omitempty"`
}
func greeting(u *userv1.User) string {
// Two ways to read a field: the generated getter, or the struct field
// itself. The getter also survives a nil u, and hides the pointer:
// it returns "" when nickname was never set.
name := u.GetNickname()
if name == "" {
name = u.Email // direct field access: panics when u is nil
}
return "hello " + name
}Looking at a message during debugging
Use this when you need to see a message in a log line or a test failure. It is for humans only. The library deliberately varies the whitespace it emits from one build to the next, so the same message can print differently after you rebuild, even with the protobuf module version unchanged; within a single binary the output does not change between calls, but never compare it as a string or store it as a canonical form. For storage and for the network, use proto.Marshal.
import "google.golang.org/protobuf/encoding/protojson"
func dump(u *userv1.User) (string, error) {
// protojson prints the JSON mapping defined by the .proto file,
// which is readable, unlike the binary wire format.
b, err := protojson.Marshal(u)
if err != nil {
return "", err
}
return string(b), nil
}
Discussion