Protobuf: proto: duplicate type registered

[peterjasc]

I have an issue, where two projects need to use the same .pb.go file and this obviously generates the "duplicate type registered" error, if one of the projects imports the other project.

The layout for the projects:

-proto-project
-- service.proto

-service-project
-| cmd
--- service.go
-| generated
--- service.pb.go

-other-project
-| cmd
--- other.go
-| generated
--- service.pb.go

Obviously, this could be solved by generating the .pb.go inside proto-project, but then there would be some manual versioning needed, if different projects need different version.

Any thoughts?

[puellanivis]

Ideally, the service.proto as well as the service.pb.go should reside in a common import path for both projects. You should avoid generating a .pb.go and then copying it whole into another codebase to make use of it there.

[PxyUp]

Protofiles are shared and used by both project, we have problem in git submodule, because we connect protofiles like git submodule. And problem exist when submodule used protofiles too, like main project, in that case we have naming conflict.

[peterjasc]

@puellanivis yes, I realize, this, but we need to have different versions of .pb.go for different projects. So the layout is actually more similar to this:

-proto-project
-| v3
--- servicev3.proto

-service-project
-| cmd
--- service.go
-| generated
--- servicev2.pb.go

-other-project
-| cmd
--- other.go
-| generated
--- servicev1.pb.go
-other-project2
-| cmd
--- other2.go
-| generated
--- servicev2.pb.go
-other-project3
-| cmd
--- other3.go
-| generated
--- servicev3.pb.go

First workaround
One way to solve this is to remove the init functions from the .pb.go to avoid collision, but it's bit ad hoc

sed '0,/init/{s/init/initNotCalled/}' generated/service.pb.go > /tmp/tmp.$$
cat /tmp/tmp.$$ > generated/service.pb.go
rm -f /tmp/tmp.$$

Second workaround
Use different package name during generation

Third workaround
Pass function as a parameter to service-project to call the .pb.go inside other-project, so that service-project has no .pb.go dependencies

[dsnet]

Obviously, this could be solved by generating the .pb.go inside proto-project

This is highly recommended.

we need to have different versions of .pb.go for different projects

I'm not sure what this means. Using semver definitions, are these major versions or minor versions?

• If major, the proto files should use a different proto package namespace to not conflict with any other major versions (e.g., put the major version in the proto package name: "my.company.prefix.service.v2").
• If minor, I don't see how you can reasonably expect to have multiple different minor versions linked in and not run into type variance issues (i.e., registration conflicts are a form of type variance issue).

First workaround
One way to solve this is to remove the init functions from the .pb.go to avoid collision, but it's bit ad hoc

This is going to run into issues later on as the protobuf language requires the use of registration for dynamic usages like Any and extensions. Even worse, the failures will be more obscure. Loud up-front failure is better than quiet and subtle failures later on.

Second workaround
Use different package name during generation

Are you talking about the proto package or the Go package? Depending on which, both could work:

• Altering the proto package is a suitable way for to ensure each generated file is in a different proto namespace (and hence no conflicts), OR
• Specifying a single Go package which results in a single generated .pb.go file, which all project then import.

Third workaround
Pass function as a parameter to service-project to call the .pb.go inside other-project, so that service-project has no .pb.go dependencies

I'm not sure what this means. Are you suggesting a form of dependency injection? I guess that might work, but it's a roundabout way to avoid the problem.

[dsnet]

As an example of .proto and .pb.go file management, there's the approach that the Google Cloud APIs take:

• The .proto files are located in one repository:
  
  
  
  • github.com/googleapis/googleapis/google
  
  
• The .pb.go files are located in another repository:
  
  
  
  • github.com/googleapis/go-genproto/googleapis
  
  

In these you see examples of what I said earlier:

• Major versions are contained within the proto package path.
• Each .proto file specifies exactly what the canonical Go import path is for that file.
• The go-genproto repository versions the set of all generated .pb.go as a large monolithic Go module. If you wanted finer granularity, you could have each sub-directory to contain its own go.mod file. Thus, you could version each directory separately from another.

[dsnet]

I'm going to close this. If you have addition questions, I can try and help, but global uniqueness in the protobuf namespace is absolute requirement for the proper functioning of protobufs.

[rathourarv]

I have also a similar issue. I have some messages and methods which are in common proto.

• proto-project
  -- common.proto

message Some {
some Int
}

service-1 and service-2 are importing the common.proto like import "common.proto";

-service-1-go-package
-| cmd
--- service-1-proto which includes common.proto

-service-2-go-package
-| cmd
--- service-2-proto which includes common.proto

While using generated pb.go files of these services files, I am getting
a message which says - proto: duplicate proto type registered: common.Some

I know it's registering the common.Some two times. But can I ignore registering the same message multiple times? Could you please suggest to me something helpful?