Protobuf: Feature Request: Field Validation Parity with JSON Schema

[jaybennett89]

In the REST+JSON world you often see protocols defined with JSON schemas as an alternative to Protobufs. These schemas function very similar to protobuf definitions to define your message payloads. They also offer many of the same rules such as oneof, required, repeated (aka arrays), etc.

However, with JSON schema validators you can provide additional business rules on your fields and get the data validation free as part of using the JSON schema. How do developers using gRPC + Protobufs maintain these validation rules across multiple languages in a consistent way, without these type of rules in the proto definitions?

There are a couple main rules that are badly needed, such as pattern matching, checking int values, and checking repeated field item counts. Could look something like this:

message Example {
    string message = 1; `pattern: ^[a-zA-Z].*`
    int32 percentage = 2; `minValue: 0, maxValue: 100`
    repeated int32 list = 3; `minItems: 1, maxItems: 10`
}

And then have the protoc compiled code for reading/writing the message automatically check these simple rules during the serialization and deserialization of this message and throw an error if the data is not valid.

Wouldn't that be awesome?

[jaybennett89]

I put together an example to point out how much boilerplate is involved in doing this kind of data validation: https://github.com/jaybennett89/protox-go-example. It implements the message Example { ... } from my original post, but with the validation code created manually.

The main issue I have is that I've got no way to automate the generation of the boilerplate validation code (https://github.com/jaybennett89/protox-go-example/blob/master/example/validate.go) when using protobufs without this kind of feature.

I wrote ~150 lines of code and tests just to validate my 5 simple rules, and this scales quite badly as you add new messages that need new rules. It scales even worse if you have multiple languages involved.

[hadrielk]

What you're talking about would be a lot of work to create, to support every programming language protobuf supports. For example a language like C++ doesn't even have regex pattern matching built-in (well, unless you use C++11 or newer, but protobuf doesn't require C++11).

For now, for your use-case, maybe you could add your rules to the .proto file as Custom Options, and during .proto file compilation either hook into the protoc compiler as a plugin to get the custom option info and generate your Go language rule verifiers, or parse the .proto files for them in a second stage?

[xfxyjwf]

This has been brought up before (inside Google, multiple times) and it was decided that we are not going to include such features in core protobuf library. It can, however, be supported as a separate library using custom options as suggested by @hadrielk above.

[htuch]

https://github.com/lyft/protoc-gen-validate is an OSS project to do this.