Protobuf: Missing well-known types?

[bufdev]

Is there any reason ptypes does not have all the well-known types? Missing files are:

• api.proto
• field_mask.proto
• source_context.proto
• type.proto

If golang/protobuf is going to handle well-known types, I would argue it should include all of them, even if some of them seem less useful than others - to really address https://github.com/golang/protobuf/issues/50, it should do all of them I would think so there is no more fragmentation.

[dsymonds]

I only added the ones I thought might be used widely. And at least in the docs that I'm going off, only field_mask.proto is actually missing; the other three aren't "well-known types" in that sense.

I don't object to adding them, but I don't want to rush into it when there's not a compelling need. If the whole set gets moved elsewhere then there's just more churn.

[bufdev]

Short: They almost never change, they do seem to all be considered well-known types, if the protos move it won't affect golang/protobuf, and deciding what end users will and will not want to use (when they are well-known types as google/protobuf defines them) will just lead to more fragmentation.

Long:

They've barely ever changed, and the main thing is that all of them seem to be considered well-known types:

• https://github.com/google/protobuf/blob/48ebb29a8ec118bf6b9ee39f6be42b57321c099a/src/google/protobuf/api.proto#L38
• https://github.com/google/protobuf/blob/48ebb29a8ec118bf6b9ee39f6be42b57321c099a/src/google/protobuf/field_mask.proto#L35
• https://github.com/google/protobuf/blob/48ebb29a8ec118bf6b9ee39f6be42b57321c099a/src/google/protobuf/source_context.proto#L35
• https://github.com/google/protobuf/blob/48ebb29a8ec118bf6b9ee39f6be42b57321c099a/src/google/protobuf/type.proto#L38

In terms of golang/protobuf, I don't think these moving other places would affect anything, the regen script would just have to be updated. I'd say that if we're really going to say 50 is fixed (and I don't think anyone of us wants to revisit that issue any time soon), and if the ptypes package is the solution we want everyone to use, that we should stick with it and not have as much fragmentation. Otherwise, for example, if anyone is using those, we're back to splitting out locations for proto files that are in the same place, ie in https://github.com/peter-edge/pb/tree/master/go/google/protobuf (which does cover all the well-known types), it's going to be a mess.

[dsymonds]

Well, that only shows that the C# folk consider them WKTs. ;-)

But I'll yield for now. Stand by.

[zellyn]

I figured everything here was a WKT. https://developers.google.com/protocol-buffers/docs/reference/google.protobuf

Although I've had no luck getting anyone to explain the philosophy/intent/extent of WKT's - apparently there are design docs but they're Google-internal :-/

[bufdev]

Hey, how about it :)

[dsymonds]

Sorry for the silence. I got sidetracked.

I've come around to realising that this needs to be tackled as a whole, for all the protos in https://github.com/googleapis/googleapis (well, that plus the implicitly overlaid hierarchy of the WKTs). I'm thinking they'll all look like google.golang.org/apis/google/protobuf/timestamp and google.golang.org/apis/google/rpc/status and so on. But I've got to win over a few more people.

[bufdev]

This is probably the 15 millionth time I've annoyed you with these links but happy to just transfer some of this, heh:

https://github.com/peter-edge/google-protobuf-go
https://github.com/peter-edge/googleapis-go
https://github.com/peter-edge/pb

[dsymonds]

Thanks, but actually doing it is the easy part. Deciding exactly what to do is the hard part.

[bufdev]

Ya, heh.

On Thursday, March 24, 2016, David Symonds [email protected] wrote:

Thanks, but actually doing it is the easy part. Deciding exactly what to
do is the hard part.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/golang/protobuf/issues/143#issuecomment-201006371

[awalterschulze]

I don't know that everyone wants googleapis. Lots of people use protobufs for non google things.

[dsymonds]

Yes, but the googleapis tree intentionally overlaps with the WKTs and imports them, and they're designed to be one logical tree (though they're in separate trees right now). The impact on people _only_ wanting the WKTs should only be some extra .proto and .pb.go files lying around.

[bufdev]

Ya, just extra proto and pb files, for those who do not want to use them,
they don't need to import them

On Monday, April 11, 2016, David Symonds [email protected] wrote:

Yes, but the googleapis tree intentionally overlaps with the WKTs and
imports them, and they're designed to be one logical tree (though they're
in separate trees right now). The impact on people _only_ wanting the
WKTs should only be some extra .proto and .pb.go files lying around.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/golang/protobuf/issues/143#issuecomment-208543653

[awalterschulze]

I guess so, but its quite sad.
At least this repo is being kept clean of other dependencies.

[bufdev]

Hey @dsymonds how is the winning Googlers over game going? :)

[Globegitter]

Would be great to get the field_mask.proto type included. We are now using quite a bit in our application and then we would not have to manually include it anymore.

[ernestoalejo]

The googleapis tree is needed to generate the RESTful gateway for GRPC services:
https://github.com/gengo/grpc-gateway/tree/master/third_party/googleapis/google/api

[ghost]

See https://godoc.org/google.golang.org/genproto/protobuf

from godoc:

It is generated from these files:

google.golang.org/genproto/protobuf/api.proto
google.golang.org/genproto/protobuf/descriptor.proto
google.golang.org/genproto/protobuf/field_mask.proto
google.golang.org/genproto/protobuf/source_context.proto
google.golang.org/genproto/protobuf/type.proto

[zombiezen]

cc @okdave

Yes, this is now the home for the WKT. Eventually, we'd like to move the existing ptypes package over to this new repository, but we're unsure how to do this in a non-breaking way. (The type aliases proposal currently under discussion would help here.)