Go: cmd/go: setting up a multi-module single repo is difficult

[hyangah]

Let's say we have a repository containing three packages and
we decided to organize them in separate modules for some reasons.
The purple package imports blue and red packages.

> cd $WORKSPACE/example.com/purple
> tree
.
├── blue
│   ├── blue.go
│   └── go.mod
├── go.mod
├── purple.go
└── red
    ├── go.mod
    └── red.go

We created go.mod files in each package directory using 'go mod init example.com/purple/red', etc.
Now we try to build the purple package.

$ go build all
go: import "example.com/purple" ->
    import "example.com/purple/blue": cannot find module providing package example.com/purple/blue
go: import "example.com/purple" ->
    import "example.com/purple/red": cannot find module providing package example.com/purple/red

What's happening underneath is, the go build command queries https://example.com/purple/red?go-get=1, ... to get the meta tags. That means, until I check in the packages and go.mod files to the remote repository and make them accessible, I can't build/test.

What's the easiest way to start a multi-module repo with inter-dependency?

[rsc]

A go.mod is like its own little GOPATH. There is no implicit reference to other nearby modules. In particular being in one repo does _not_ mean that they all move in lock step and always refer to the code from the same commit. That's not what users will get either. If there is an inter-dependency, you need to add it, with something like

cd $WORKSPACE/example.com/purple
go mod edit -replace example.com/purple/red=./red
go mod edit -replace example.com/purple/blue=./blue

[hyangah]

This issue is closely related to #26241, #25053 and other issues that involving development of multiple modules concurrently and access to local modules.

In the above example, one caveat is that the 'replace' rule doesn't take effect until there exist the 'require' entry for the replaced module. That's why I ended up using 'go mod edit -require' which is not meant for use by human. (#27060) So, in order to add the local inter-dependency, users need to either

1) commit the updated red, blue modules upstream so they can be go-gettable. Then transform purple, or
2) manually edit go.mod to include red and blue entries both in 'require' and 'replace'.

At the beginning of converting an existing repo to a multi-module repo, currently, users will need hand-edit go.mod file (go mod edit -require and go mod edit -replace) and careful planning of commit ordering.

[andybons]

@bcmills

[bcmills]

This issue is closely related to #26241, #25053 and other issues that involving development of multiple modules concurrently and access to local modules.

@rogpeppe's gohack tool has some nice prototyping for hacking on multiple modules. I wonder whether that supports multi-module repos, and if so what we can learn from it.

[myitcv]

@hyangah I submitted an experience report for a similar setup.

From my perspective, the only "problem" is the requirement on existence of the submodule (and I don't think we can solve that problem). For me, this was step 1. But this was only a "problem" in so far as my CI build broke. Beyond that commit, all was fine.

At the beginning of converting an existing repo to a multi-module repo, currently, users will need hand-edit go.mod file (go mod edit -require and go mod edit -replace) and careful planning of commit ordering.

This is effectively the crux of my open questions too. I'm less worried about the "hand-edit" part, because I think tooling can easily fix that. It's the commit dance that follows that raises the biggest question. Again, tooling could help, but because of the mix of VCS systems that people may be using, it's not immediately obvious. With Gerrit things are certainly much easier...

[zellyn]

As an experience report, our Go code at Square is organized in a similar fashion. We are still using godep because we were waiting for the Packaging Wars to settle out.

We have O(dozens) of apps at $REPO/$APPNAME or $REPO/$TEAM/$APPNAME, as well as components like $REPO/service/logging, $REPO/service/feeds, $REPO/service/rpc, etc.

It doesn't seem as if hosting multiple modules out of a single repo is clean (version tags are per-repo), but we're considering giving each app a separate go.mod file: it would allow each app to choose what version of third-party libraries it uses.

We're also considering some kind of linter/auto-go.mod-writing tool to ensure that _all_ go.mod files include things like replace directives for our protobuf fork. I was intending to have that tool enter replace $REPO/service/foo => ../service/foo directives for each module imported, or perhaps a global replace $REPO => .. to catch all of them. It's awkward, but not unworkable: those replace directives would be out-of-place in the checked-in code if this were an open-source repo, but since it's our monorepo, everything would probably work okay with them checked in.

Anyway, just wanted to log a report. I expect within Google, Go code is still built using blaze, with custom tooling, so I don't expect monorepo experience help from inside :-)

[myitcv]

@zellyn just to check on one point:

It doesn't seem as if hosting multiple modules out of a single repo is clean (version tags are per-repo)

Do you mean _your_ version tags are per-repo? Because in general multi-module versions work as a result of the submodule prefix on a tag. For example the repo that results from the submodules (multi-module repo) guide has the following two tags (i.e. versions):

b/v0.1.1
a/v1.0.0

We're also considering some kind of linter/auto-go.mod-writing tool ...

This would presumably be a wrapper around go mod edit -replace... and friends?

[zellyn]

@myitcv

Because in general multi-module versions work as a result of the submodule prefix on a tag.

Ah, I somehow missed that you could namespace version labels like that. Simple and clear. Thanks!

This would presumably be a wrapper around go mod edit -replace... and friends?

I imagine so, probably. The more important reason for linting is that we need to catch an app accidentally _not_ using our protobuf fork. It could result in not redacting sensitive fields in logs, for example.

[bcmills]

We're also considering some kind of linter/auto-go.mod-writing tool to ensure that _all_ go.mod files include things like replace directives for our protobuf fork.

That seems like a problem you could resolve with a GOPROXY HTTP server, at least: it could act as a straight proxy for most packages (or serve 404s after https://github.com/golang/go/issues/26334 is fixed), but serve 403s for any module you want to exclude globally.

[zellyn]

Interesting idea. I'll have to see if we can get our Artifactory instance to do that, after we get the basic functionality working).

[bcmills]

We've addressed a number of the underlying issues, although there are certainly still some difficulties in setting up and testing multi-module repos.

Let's track the remaining issues individually rather than under this umbrella issue.

I know that at least the following are related:

• #26640
• #27542
• #28835