JSON vs Protocol Buffers

Pick a kind of data and a number of records. The bars show the encoded size in each format, and the inspector shows the bytes. The lower panel writes one user record with one version of the schema and reads it with another.

• Compare sensor readings with log events. Numbers shrink a lot in protobuf; free text barely does.
• Look at the gzip bars at 100 records: compression removes most of JSON's repeated field names, and the gap narrows.
• Write with v2 and read with v1: the new fields come through as unknown fields. Read with v3 (renamed field) and nothing changes for protobuf, while the JSON reader loses the value.

The protobuf encoder produces the same bytes as Google's protobuf runtime, the MessagePack encoder the same bytes as the msgpack library, and decoding across schema versions gives the same values and unknown fields as the Python protobuf runtime.

JSON (JavaScript Object Notation) is a text format for objects, arrays, strings, numbers, booleans and null. Every document describes itself: field names travel with the data, and any program can read it without knowing its structure in advance. Protocol Buffers (protobuf) is Google's binary format: the structure is declared in a .proto file, code is generated from it, and the encoded data carries only field numbers and values, so a reader needs the schema to make sense of it.

The choice affects size, speed, how readable the data is to people and tools, and how safely the format can change over time.

Bytes for each dataset, raw and gzipped (gzip level 6, as computed when this page was built):

A message is a sequence of fields. Each starts with a tag, the field number shifted left three bits plus a wire type: 0 for varints (integers, booleans, enums), 1 for 8-byte values (double), 2 for length-delimited data (strings, bytes, nested messages, packed lists) and 5 for 4-byte values (float). A varint stores 7 bits per byte, setting the high bit when more bytes follow, so 300 is 08 ac 02: the tag 0x08 for field 1, then 0xac 0x02. Negative int32 and int64 values take 10 bytes because they are sign-extended to 64 bits; the sint types use zigzag encoding to keep small negative numbers short.

proto3 leaves out fields equal to their default (0, false, empty string), packs repeated numbers into one length-prefixed block, and lets fields appear in any order. Decoding is a loop over tags; a field the reader does not know is skipped by its wire type, which is what makes adding fields safe.

Services are upgraded one at a time, so for a while old code reads data written by new code and the other way round. The simulator's user record goes through four changes. Written with v2, which added two fields, and read back:

• by v1, which lacks the new fields: they are skipped as unknown fields (3 of them, a repeated field counting once per value) and preserved if the message is written out again;
• by v3, which renamed name to display_name: the value arrives as "Grace Hopper", since only the number 2 is on the wire;
• by v4, which narrowed id to int32: 5,000,000,123 is truncated to 705032827, silently;
• by v5, which deleted email and reused number 3 for an int64: the string arrives with the wrong wire type and is set aside as unknown, so login_count reads as 0. Had both been integers it would have been misread without any sign.

The rules that follow, from the protobuf language guide: adding fields is safe; never change a field's number; never reuse a deleted field's number, and list it as reserved so nobody can; int32, uint32, int64, uint64 and bool are compatible but values are truncated; string and bytes are compatible when the bytes are valid UTF-8. Renaming is safe for the binary format but not for protobuf's JSON mapping or for code that uses the old name. JSON has the opposite profile: names are the contract, so renames break readers, but type and number mix-ups cannot happen silently in the same way.

• Speed. Parsing decimal text, escaping strings and building generic maps costs more than reading varints into generated structs, so protobuf usually encodes and decodes faster; how much depends heavily on the library and language, and modern JSON parsers are fast.
• Readability. JSON can be read in a terminal, logged, diffed and edited by hand. Protobuf needs the schema and a tool (protoc --decode, or --decode_raw for numbers only).
• Types. JSON has one number type, and JavaScript reads it as a double, so integers above 2⁵³ lose precision; protobuf's JSON mapping writes int64 as a string for that reason. Protobuf has fixed integer widths, enums and a bytes type.
• Tooling. Protobuf gives typed, generated clients in many languages and a single definition of the API; JSON needs a separate schema (JSON Schema, OpenAPI) to get the same guarantees.

MessagePack and CBOR (RFC 8949) are binary encodings of the JSON data model: no schema, compact numbers, but field names in every record. Apache Avro writes values with no field tags at all and relies on the writer's schema being available to the reader, which makes it very compact for files of many records. Apache Thrift is close to protobuf in design. FlatBuffers and Cap'n Proto lay data out so it can be read in place without a decoding step, trading some size for zero-copy access.

Douglas Crockford specified JSON in the early 2000s as a subset of JavaScript literals; it was published as RFC 4627 in 2006, as ECMA-404 in 2013, and as the current standard RFC 8259 in 2017. Protocol Buffers began at Google in 2001 and were open-sourced in 2008 as proto2; proto3, which simplified the language and dropped required fields, arrived with protobuf 3.0 in 2016. MessagePack was created by Sadayuki Furuhashi.