Protobuf: reflect/protoregistry: conflicts with same filename

[guyguy333]

Hello,

Sharing a same .proto filename with different proto package names results in these errors:

2020/05/10 15:03:12 WARNING: proto: file "common.proto" is already registered previously from: "git.example.com/proto/common" currently from: "git.example.com/proto/a" A future release will panic on registration conflicts. See: https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

2020/05/10 15:03:12 WARNING: proto: file "common.proto" is already registered previously from: "git.example.com/proto/common" currently from: "git.example.com/proto/b" A future release will panic on registration conflicts. See: https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

Here is my layout:

proto/
  common/
    common.proto (package common)
  a/
    common.proto (package a and importing common/common.proto)
    foo.proto (importing common.proto)
    ...
  b/
    common.proto (package b and importing common/common.proto)
    bar.proto  (importing common.proto)
    ...

I don't think my case is similar to both quoted in documentation https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict. I don't generate multiple times a same proto file (here I generate one time common/common.proto and I import from others files) and I'm using different package names. It would be better if I rename package common to something else, but I don't think it's my issue here.

What's the best practice to solve this issue ? Is it normal to get this warning in my case ? I mean, it doesn't look uncommon to have the same filename in different projects.

Thanks

[neild]

I don't see any good answers here, so we're probably stuck picking from bad ones. There is a naming conflict which will prevent the two different common.proto files from appearing in the same dependency tree, but it is plausible that they will never do so.

The options I see are to continue to mandate that filenames must be globally unique (do any other implementations require this?), or to permit a conflict. If we permit a conflict, we need to figure out what (*protoreflect.Files).FindDescriptorByName returns for a conflicting name. Probably an error.

[dsnet]

Is there any reason why a/common.proto can't simply be compiled as a/common.proto and b/common.proto simply be compiled as b/common.proto? That seems like it would avoid the problem altogether.

[neild]

It seems plausible that someone might want to link packages outside their control that both contain a common.proto, though.

[dsnet]

It seems plausible that someone might want to link packages outside their control that both contain a common.proto, though.

Sure, but that doesn't seem to be problem in this issue though.

In my opinion, the protobuf developer site is lacking guidance in general in this area, and the protoc tool itself is not particularly friendly towards users who do not work in a mono-repo like Google does (where every .proto source file is relative to the root of the mono-repo). The open-source world is fundamentally a collection of disjoint repositories. There needs to be better guidance on what is the appropriate import path for a .proto file and how to reference a .proto file that crosses repo boundaries.

[guyguy333]

Is there any reason why a/common.proto can't simply be compiled as a/common.proto and b/common.proto simply be compiled as b/common.proto? That seems like it would avoid the problem altogether.

We can do that and yes it will avoid the problem. To be more accurate, A and B are two micro services and we can see common as an _util library._ We considered it was ok to include a directory directly in include path when we generate A proto Go files. As we can see common dir as an "external lib", we decided to include only proto dir (to avoid "import" conflicts) to have a prefix (common in our case).

I think we can compare this issue to a project (whatever the language) doing internal import not prefixed with full path and external import mainly using full path.

[trinitum]

Is there any reason why a/common.proto can't simply be compiled as a/common.proto and b/common.proto simply be compiled as b/common.proto? That seems like it would avoid the problem altogether.

@dsnet may I ask how would one compile a/common.proto as a/common.proto? I've run into a similar problem and when I run protoc --go_out=paths=source_relative,plugins=grpc:a a/common.proto it still warns about the conflict with common.proto from another package.

[strowk]

I also had similar problem and what I don't understand is how it does not take "package" into account. In my case I have two events.proto files in two different folders, they generate events.pb.go , also in two different folders AND both of them have different "package" and go_package. Why do I get proto: file "events.proto" is already registered? Is this a bug in protobuf or I don't understand that documentation - https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict ?
Particulary that

Every protobuf declaration (e.g., enums, enum values, or messages) has an absolute name, which is the concatenation of the package name with the relative name of the declaration in the .proto source file

sounds like I should not have this problem, yet warning is printed

[neild]

The problem is a conflict in the source file name.

The protoregistry package permits looking up a .proto source file by name. If there are two files with same name and path, it has no way to pick between the them.

This can also be a problem when running protoc; if a .proto file includes

import "events.proto";

and there are two different "events.proto" files with no additional path prefix to distinguish them, the import is ambiguous.

I suspect that looking up .proto files by name is sufficiently uncommon that we should probably relax this check and return an error from (*protoreflect.Files).FindFileByPath when there is ambiguity.

[luisguillenc]

I have solved the problem including all the path (relative to gopath) in the "import" and compiling with protoc using the absolute path to the proto file.

After this, in the pb files generated, in the grpc.ServiceDesc structure, Metadata field contains the full path (relative to gopath) and works without warnings.

[rcgoodfellow]

So we have the situation @dsnet describes. We have Go program that depends on two external libraries both of which have a spec.proto and we get this warning when the program runs

2020/08/11 11:20:11 WARNING: proto: file "spec.proto" is already registered
A future release will panic on registration conflicts. See:
https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

I agree with @strowk, at the very least there should be some second level logic based on package name to disambiguate source file name collisions. The workaround of changing how the protobuf is compiled does not work when an external library is managing how the protobuf is compiled.

Along the current logical arc, the implication is that all protobufs in all repositories globally, need to have unique filenames - which is not reasonable. I really hope the threat of panic in future releases does not come to fruition.

[johnsiilver]

I receive these warnings now on and older project I just came back to after a year. This project had been having no issues but, since I've come back I've made the following updates:

• using go.mod instead of gopath
• using a new protoc
• using a new grpc protoc lib
• using the old proto package that is now using the new proto package underneath
• added go_package to my protos that I did not use before

This is happening in a layout model I was using that had components setup such as:

componentA/
io/
io.proto
componentB/
io/
io.proto

So now I'm getting the: WARNING: proto: file "io.proto" is already registered, A future release will panic on registration conflicts

I also moved to using go_package as well with dsymonds overloading setup, for example:

package marmot.example.scripts.juniper_password_change.ssh;

option go_package = "github.com/bearlytools/bearmetal/framework/examples/scripts/juniper_password_change/cogs/ssh/io;io";

I'm sure I haven't thought of a million different ways this could break or is undesirable, but could we avoid this conflict by registering using the go_package full path with a flag passed, like --register_using_go_package? I'm sure this has some undesirable behavior, but I'm 100% sure neild is right, that there is probably just no great solution.

[jabersmith]

Has the protobuf team made any further decisions on this issue?

Much like @rcgoodfellow, I have a Go program that depends on two third-party libraries, each of which defines a telemetry.proto file at the top level of their project. With current google.golang.org/protobuf, the program panics unless I set the GOLANG_PROTOBUF_REGISTRATION_CONFLICT environment variable.

This can't be that uncommon a situation. At this point, I'm not really clear how the protobuf team expect the file-by-path registry to work outside of a monorepo setting.