Protobuf: protoc-gen-go: add better support for modules

[dsnet]

The go toolchain is moving towards the deprecation of $GOPATH where modules are the norm. In such a world, the current default behavior of emitting a file relative according to the Go package path is often not the right choice.

Let's suppose protoc is invoked at the root of repository (located at $REPO_ROOT) with a go.mod file that specifies the go import path as "github.com/user/myproject". Suppose a $REPO_ROOT/foo/bar/hello.proto file has option go_package = "github.com/user/myproject/foo/bar" specified. Currently, the default behavior of protoc is to emit a file located at $REPO_ROOT/github.com/user/myproject/foo/bar/hello.pb.go which is almost certainly not the desired behavior. (I know the suggested use case is --go_out=$GOPATH, but the point is $GOPATH is going away in the future).

We could:

• Accept yet another flag that specifies the current go.mod import path. For example: protoc --go_out=mod_root=github.com/user/myproject:. will know to truncate "github.com/user/myproject" from "github.com/user/myproject/foo/bar" and determine the output directory as "foo/bar".
• Teach protoc-gen-go about go.mod files and find any go.mod files at the current working directory and above to derive the go path root. However, I'm not fond of magic of this sort. That said, the behavior of the Go toolchain follows this logic when building code outside of the $GOPATH.
• Should source_relative be the default behavior instead? This follows the observations that .proto file often (but not always) co-live with the generated .pb.go files.

EDIT: This issue used to be about making source_relative the default, but is now more heavily promoting a module-specific solution.

\cc @neild

[dsnet]

Tentatively marking as "breaking change" since this would change the default behavior of protoc-gen-go. However, the compatibility agreement only states that it only covers the library and generated code, and not the API for how protoc-gen-go is invoked.

[smazurov]

i love mod_root solution. I am running into this now where we have a generic grpc project that is shared across languages, the way we generate it and store them, there isn't an easy way currently to generate go module.

[ericrenard]

I encountered the same issue and thought about the mod_root idea too. Seems to be an easy solution to a problem that will become widespread when go 1.13 is released.

[mkarpusiewicz]

I have the same problem when using go modules and repo is not in $GOPATH. Currently only solution is to move the generated files to correct root. Support for go.mod or mod_root would be very helpful.

[millergarym]

Could this be achieved with a plugin to protoc-gen-go? Ie can plugins receive params?

Other issues out of scope of this issue are; repository tags (specifically multi module repos) & cross module imports (again multi module).

[letmefly]

look forward to support golang modules when a.proto(package A) import share.proto(package common)
the generated code is this importing way:
import (
common "common"
)
but with GO111MODULE=on, this importing way is not work!

[wxue]

Like this propose! (It's bit mess for relative path generation on shared pd now. )

Pick up the package path from go.mod sounds good.

[chipdude]

Since this issue has been raised, rather a long time has passed with no code. Over a year.

Do the Powers That Be disagree that this is a problem, or that a sufficient solution has been proposed, or ...?

[dsnet]

We agree it's a problem, and plan to address it shortly after the v2 release. Our focus at the moment is getting v2 released first.

[alexrudd]

For anyone interested, here's the workaround i'm using (as a makefile target):

# package/protos/messages.proto
syntax = "proto3";

package messages;

option go_package = "github.com/my-org/my-project/package/protos";

message SomeMessage {}

# regenerate go protobuf code
protoc-go: protoc-go-clean
    mkdir -p /tmp/protogen/github.com/my-org/
    ln -s $(shell pwd) /tmp/protogen/github.com/my-org/my-project
    find -iname '*.proto' -exec \
        protoc --proto_path=. --go_out=plugins=grpc:/tmp/protogen/ {} \
        \;

it creates a soft link to the current working directory under tmp/protogen/<github repo path> then finds all project .proto files and generates the code out to there. Protoc follows the soft link so generated files end up in their correct directories as defined by option go_package

[yordis]

@dsnet following up your comment

We agree it's a problem, and plan to address it shortly after the v2 release. Our focus at the moment is getting v2 released first.

Is there any timeline for that? I couldn't see any milestone set up so I am not sure where to look up this information.

[dsnet]

This feature is likely to be in the next release. See https://golang.org/cl/219298

[dsnet]

Removing breaking-change label as this feature requires the user to specify the module that we are generating for (protoc-gen-go fundamentally does not have access to that information). Specifying that information alone is sufficient to signal that we want to operate in a module-specific mode.