Go: cmd/go: support simultaneous edits of interdependent modules

[bcmills]

Many module issues and questions seem to center on editing, testing, and deploying multiple (possibly mutually-interdependent, possibly cyclic) modules (examples: #27514, #27056, #26640, #26377).

The main workaround at the moment is to add replace directives among the modules to be edited, but maintaining those directives is tedious and error-prone. @rogpeppe's gohack tool automates away some of the tedium, but doesn't seem to remove the risk of accidentally checking in a go.mod with what were intended to be local, temporary replacements.

The go command should support multi-module edits in some form. It's not yet clear to me what form that should take, but I figured I'd go ahead and file an issue to collect ideas.

(CC: @rsc @myitcv @thepudds @marwan-at-work @oiooj @hyangah)

[bcmills]

One option (based on #26377) might be to set up a sort of pseudo-GOPATH tree containing the modules to be edited, with the local modules located at their module paths within that tree.

Assuming that all of those modules include metadata from some supported VCS tool, the go command would update the require and replace directives in each local module to accurately reflect the pseudo-versions (or locally-tagged versions!) of its dependencies in use, and go get operations would update the local checkouts to reflect the newly-selected versions.

One of the downsides of that approach is that it reintroduces the need to indicate the root of the “local modules” tree.

[rasky]

One option (based on #26377) might be to set up a sort of pseudo-GOPATH tree containing the modules to be edited, with the local modules located at their module paths within that tree.

We already have (had?) this: it's the vendor directory. If you run go mod vendor, you get a very clean representation of your dependencies, without any binary zip file, laid out on disk to be edited, diffed, committed.

[thepudds]

@bcmills regarding your comment in https://github.com/golang/go/issues/27542#issuecomment-419237656:

a) would it need to rely on VCS metadata (vs. maybe it could rely on relative on-disk location), and
b) would it need to actually insert the replace directives (vs. maybe it could avoid updating the on-disk go.mod files)?

The file tree with a go.mod has been described as something like "a little GOPATH", but as has been observed, it can be awkward to wire together multiple "little GOPATHs", especially if there are many, and especially if it is a routine situation (and not something like a one-off quick debugging investigation into a dependency).

One piece of the puzzle might allowing a user to opt-in in some way to having the relative on-disk location between the "little GOPATHs" always be the same in their development/environment/test/build/CI environment (or more precisely, not "always" the same, but have the relative on-disk locations between the "little GOPATHs" be constant until someone starts further modifying things such as by introducing replace directives if they wanted to re-arrange things such as to try a local fork).

Perhaps a rule could be a "parent" go.mod serves as the anchor for how child/descendant modules could find each other (and without needing any replace directives in any of the child modules).

In other words, if you have a "parent" go.mod located in something like /my/project, the entirety of that go.mod might be:

module example.com/me/something

And /my/project/foo/go.mod might have:

module example.com/me/something/foo

require example.com/me/something/bar v1.2.3

And /my/project/bar/go.mod might have:

module example.com/me/something/bar

Then perhaps the rule could be that foo can find bar without needing any explicit replace directives in any go.mod (given foo is able to relate its own module path to its parent's module path, and map that relationship directly to the on-disk relative directory location)

Or maybe that might break compatibility with Go 1.11 behavior? In which case, maybe there is some signal in the "parent" go.mod that the user is opting in to the behavior. Maybe the "parent" go.mod could read:

module example.com/me/something

replace relative

(or maybe rather than replace relative, the "parent" go.mod could have replace auto, or replace children, or have a something added to the module directive, or an alternative filename extension, or whatever other signpost is deemed more aesthetically pleasing).

I've been noodling on something like this for a bit given the repeated questions around the current need for replace directives, but I'll confess it is not fully thought out.

[bcmills]

@rasky

We already have (had?) this: it's the vendor directory. If you run go mod vendor, you get a very clean representation of your dependencies, without any binary zip file, laid out on disk to be edited, diffed, committed.

Reusing the vendor directory is an interesting idea, although I see a few rough edges:

1. The vendor directory itself needs to be rooted somewhere, presumably in some other module. How do we decide which module goes at the top level? (Presumably it's the one you want to commit last, but what if the task you're working on is splitting or merging two modules that are/were logically peers?)
2. If you're making upstream edits, you likely want submodules from the same repository to remain in the same copy of their repo. (Otherwise, working with commits and branches for those submodules could get confusing pretty quickly.) But if you've copied those submodules into vendor/, now you might have two copies of the top-level repo (or a confusing symlink structure in the vendor/ directory).
3. The existing semantics of the vendor folder are “use exactly these dependencies” — otherwise ignoring go.mod files and module dependencies. In particular, with -mod=vendor we'll use those versions of the dependencies even if the go.mod files say otherwise. In contrast, when we're making changes to a set of modules we presumably want to ensure that the go.mod requirements exactly match the source tree. That difference might be resolvable, but it's a pretty big semantic change.

Any idea how we could resolve those? (Or do you suppose that they'll turn out not to be a big deal in practice?)

[bcmills]

a) would it need to rely on VCS metadata (vs. maybe it could rely on relative on-disk location)

I think so, yes: in particular, version tags (and commit hashes, for pseudoversions) are not otherwise present in the source tree.

b) would it need to actually insert the replace directives (vs. maybe it could avoid updating the on-disk go.mod files)?

The replace directives are maybe not that big a deal, but it would definitely need to update the version requirements to reflect new tags. For example, if you're splitting one module into two mutually-dependent modules, you want to ensure that each requires the correct (updated) version of the other.

Perhaps a rule could be a "parent" go.mod serves as the anchor for how child/descendant modules could find each other (and without needing any replace directives in any of the child modules).

That's an interesting idea, but at the moment when we find a go.mod we stop looking upward in the directory tree, and I think that's probably a valuable property to preserve. For example, if we have multiple submodules in the same repo, I don't think we want to force edits to those submodules to always occur in lock-step.

[thepudds]

@bcmills

That's an interesting idea, but at the moment when we find a go.mod we stop looking upward in the directory tree, and I think that's probably a valuable property to preserve. For example, if we have multiple submodules in the same repo, I don't think we want to force edits to those submodules to always occur in lock-step.

Agreed that would be a change in behavior. Part of what I was trying to outline towards the end of my comment above https://github.com/golang/go/issues/27542#issuecomment-419262049 was that it could be a conscious choice to opt-in to these semantics (e.g., perhaps the "parent" go.mod only implies these semantics if it contains a replace relative directive, or some other signal). That would provide for backwards compatibility with 1.11 (including given that would not be a valid 1.11 go.mod), as well as make these semantics only kick in when someone does indeed want these different properties.

Side note is that the number of directories that would need to be walked upward from a given go.mod hunting for a "parent" go.mod would be limited to roughly the number of elements in the module path. (In other words, if you were in an extreme case where you are 1,000 levels deep in your directory structure and your go.mod reads module example.com/some/project/some/child, you don't need to walk up 1,000 directories to check for a possible "parent" go.mod, because you run out of pieces of your module path to be meaningful for this behavior (that is, for the behavior sketched out above of optionally using a "parent" go.mod as a sort of "super root go.mod" that effectively defines the relative directory location of the encompassed go.mod files in order to understand on-disk relationships between the inter-related modules without requiring N actual require directives to spell out the various relative paths between the related modules).

In any event, perhaps another approach is better, but setting aside the particulars of what I had sketched out above, for the purposes of this issue it could be worth thinking more broadly about how to exploit information that might naturally already exist in terms of inter-related modules (which is part of what I was aiming for here in terms of using the location of go.mod files relative to a "parent" go.mod), or perhaps thinking about how picking some convention in terms of how someone sets up inter-related modules might generate information that could then be exploited to automatically understand the relationships...

[chinglinwen]

Hope this issue been solved.

I'm using local module for some reason ( not ready yet to publish to the pubic VCS, so no domain names for the module, and our internal git servicec not gettable for now, because of https ( no insecure setting) ).

Currently, using local module is a bad experience ( need manual replace directive for every package(module) in a project ).

Some experience of add module support

1. go mod init ( need execute this for every package, in the main, and in the sub directory)
2. change go.mod ( add require directive line and replace directive line for every local modules )
3. the go module path is unclear ( currently using relative path ../other-project/pkg/foo, adds an restrict for other person to clone the projects )
   // while the old go get just simply works ( why can't go mod figure out local package path(which relate to gopath) )

[thepudds]

Recording some other suggestions/comments from this thread:
https://groups.google.com/d/msg/golang-nuts/0KQ4ZuSpzy8/tAsI8_vVBAAJ

These two comments were follow-ups to conversation in that thread around go.mod.local and the risk of accidentally checking in a go.mod with a dev only replace.

Another idea is to have a sort of "publish local" semantics, where the go tool has support for something like -devel tags, which override the defined in go.mod. So then you would "publish" the next version to your local mod cache (just creating/updating the module of the special version), and the go tool would then make use of that.

and

Publishing to the local cache is how Maven, a build tool for Java, does it. There is the concept of snapshot versions. For Golang maybe stating master (or any other branch) as version would be fitting. Then your CI could use a master checkout as well.

[theckman]

Based on a conversation in the Go Slack I was asked:

What you personally like about GOPATH, and/or what you hope a future modules-based experience will preserve about what you like about GOPATH

My favorite part about the GOPATH is the consistency it guarantees about where source will live on my workstation, and on the workstations of fellow developers. I know that my code will always be at $GOPATH/src/github.com/theckman/. Go became my favorite language because of the consistency it aimed for, and it's been an amazing experience having a language where I can finally write, or to provide instructions, that will work on any Go developer workstation with very little difference.

While this usage has become less common with tools like glide and dep, I also liked how easy it was to build with my own fork of a dependency if I have an outstanding upstream PR... or if the project is dead and has bugs.

Lastly, I've seen people use it to be able to zip up the entire tree and ship that to another developer to help them troubleshoot a weird issue. They easily had the full working tree of the project.

So in summary, I'd like modules to retain the on-disk consistency I've been able to rely on thus-far.

[thepudds]

General comment: There could potentially be a broader issue opened up with a title something like:
"Try to preserve ~90% of a GOPATH-like experience with modules"

The comment from @theckman above could fall into such an issue.

However, there are already several related issues (e.g., see the initial @bcmills comment here of https://github.com/golang/go/issues/27542#issue-357818109), so not sure if a broader new issue is useful or not.

Even though modules are in many ways "a little GOPATH", my personal opinion is that the biggest way modules today do not provide an overall "GOPATH-like experience" is the increase in complexity that arises once you have multiple modules, and hence the more use-case-based comments above from @theckman about what he values about GOPATH also make sense here in this issue about dealing with multiple modules.

Above, @theckman is placing a very high value on the consistency of GOPATH, including across developers. I have seen others express similar sentiments. At least for me, I don't know if modules will ever provide 100% of a GOPATH-like experience, mainly because modules enable more choice. For example, even if an individual or a team chooses to place 100% of their modules together in some consistent location (and assuming things like this issue here #27542 is resolved in some nice way), their approach might turn into very much of a GOPATH-like experience for that individual or team (including consistency), but the fact that a different team might make a different choice seems to imply that retaining _all_ the consistency that was delivered by GOPATH might be at odds with the flexibility that modules offer (including the flexibility modules offer to the people who state "I just want to clone a repo wherever I want on my disk").

But even if 100% of a GOPATH-like experience might not be possible, it might still be an interesting question as to how many of GOPATH's benefits can be preserved in a modules world...

[thepudds]

A few more specific reactions:

@theckman wrote:

I also liked how easy it was to build with my own fork of a dependency if I have an outstanding upstream PR... or if the project is dead and has bugs.

Right now, it's probably fair to say that use case is fairly awkward with the core go tooling, but reasonably nice for at least one-off fixes with @rogpeppe's gohack (and which again gets to the subject of this issue #27542 in terms of dealing with multiple modules).

@theckman also wrote:

Lastly, I've seen people use it to be able to zip up the entire tree and ship that to another developer to help them troubleshoot a weird issue. They easily had the full working tree of the project.

and:

it's been an amazing experience having a language where I can finally write, or to provide instructions, that will work on any Go developer workstation with very little difference.

Those two comments are things that _could_ be preserved with modules, but in part depends on how modules evolve. Perhaps those could be used as at least part of the criteria for evaluating the solution to this multi-module issue #27542.

[bcmills]

We don't have a plan for this, and there are more urgent modules issues for 1.13. Leaving open to collect ideas, but moving to Unplanned.

[spekary]

Following up from comments made to my post at #28868. My issue is not so much a build issue as described above, but how an IDE is supposed to support all of this, and I want to keep the IDE issue on the radar here. IDE's have bigger needs than the build. They do code completion, syntax checking, etc. The current module implementation which allows a particular module to override other modules depending on what directory you call the build tools out of makes the IDE's job difficult. The example situation I describe in #28868 is the most basic. Imagining a very active project with many modules in different stages of work, and with multiple executables, can make things quite difficult to keep straight.

The use case above seems to be covered by #27824 (comment), specifically the link to #26640

Somehow a developer will need to tell an IDE what go.mod file the developer intends to use for a build, so that the correct replace directives can be used to find the correct versions of all of the sources so that the IDE can then do its syntax checking and code completion magic.

Yes, that's true. (@myitcv calls that the “workspace”, if I recall correctly.) In general that should be the working directory in which the IDE or project was opened.

Both of these answers are making some assumptions about an IDE that are not necessarily true. What is this "working directory" in which the IDE was opened in a multi-project situation? Whether you have multiple modules in one workspace, or multiple workspaces, one for each module, the IDE still needs to know what go.mod file the developer intends to use in a particular build in case there are replace statements there that point to alternate sources.

Yes, and if we used environment substitutions, changing the environment would also change the effective module definitions. Fundamentally you're talking about making the build system rely on the environment (rather than the code) to specify which versions to use. That's what we had with GOPATH, and part of what we're trying to avoid with modules.

I agree with the goals. Its just that the replace statement implementation has recreated that problem. With modules, the build system now relies on the active go.mod file, which changes depending on the current working directory from which the build tools are called. Because of this, IDE's cannot track this in a many-module situation. One or two modules, no problem, so 90% of the time, its probably fine.

[spekary]

Leaving open to collect ideas

I was noodling on this, and not sure it solves all the problems listed, but here is a suggestion. However, as a preface, I think one issue is that go.mod is simply overloaded. I believe it was originally thought of as a better vendoring tool, and then it also became a replacement for GOPATH, and with the replace and exclude statements that only works for the top-level go.mod file, now its also a kind of build configuration tool. Its trying to do too many things at once.

Therefore, I think it makes sense to break it apart, creating an additional file, similar to the idea of the go.mod.local idea mentioned, but with the idea it would just have replace and exclude statements. However, instead of this being picked up by name convention like the go.mod file is, I think the go tool should be modified so that you can specify on the command line what the overriding go.mod file (or whatever it gets called), should be. That way the replace statements are not based on the CWD.

Also, if someone wants to still use a vendor directory, maybe something in there could specify this? Perhaps a replace directive that points to a top-level domain would do the trick?

In addition, since this is essentially a build configuration file at this point, it should have a mechanism to respond to GO's current build configuration mechanism, which are build tags. So, something like this:

// vendoring
replace github.com => vendor/github.com

// +build debug
replace github.com/A/B=> ../B-src

// +build !debug releasetest
replace passwords=> ../release-passwords

Something like that.

Advantages:

• The current go.mod file will work the same
• You can check in the file. Since it responds to build tags and only works if you specify it with the command line, you can architect it so that the command-line controls the build and you don't have to worry about developers accidentally checking in a bad go.mod file.
• IDEs can ask for the location of this file, and then they don't have to worry about the changing view of the world based on the location of the file being edited.
• vendoring tools can still work the same as they do today. Its just that to turn on vendoring, the developer will need to spell that out in this build-config file.

Anyways, its a start of an idea we can poke at.

[Helcaraxan]

Re-activating this issue with some thoughts after some related chat in #26344.

From the existing comments it appears that either there are a few different issues that are being conflated or, more likely, the actual use-case that we want to facilitate is not defined clearly enough. What mostly points me in this direction is that we are discussing to what measure the "GOPATH feeling" can be preserved with modules. That is a good, but also vague, term that covers many different behaviours. Something that is further highlighted by the different opinions already expressed above.

If I try to break down the different use-cases from my own perspective I get the following:

Modifying inter-dependent modules in different repositories

Preserving the "canonical" directory structure of GOPATH.

Or put differently, the _"zip up my dependency tree and send it to a friend for debugging"_ compatibility.

This is still possible in the current state without modifications to your workflow. Either your debugging is limited to a single repo and you simply send over your branch / commit. The go.mod of the project will ensure that your colleague will use the same dependencies as you. Or you have made some changes to two or more repos, in which case you'll have added relative file-path replace statements. Nothing prevents you in this situation from still relying on the GOPATH directory structure (or any other one of your choice) as 1.13 has enabled modules "by default". You can still zip up the GOPATH (or other) directory structure that contains the projects in question, send it over, and the relative replace statements will remain valid once unzipped. Only edge-case is if your colleague is on Go <1.13.

Preparing upstream patches to inter-dependent modules and ensuring that cross-module version references are accurate.

The issue here is actually less tied to modules, or even Go, than to VCS in general. I'll use the example of modifying related modules A and B. If the dependency is only from A to B then after preparing your patches locally you are forced (even in GOPATH-mode and / or with the use of dep, glide, ...) to first push your change to B before you can merge your change to A. This is because you'll need to know the VCS reference, _e.g_ commit hash, of your change in B. In the case of a cyclic dependency between A and B, you will need to make your changes to both separately, push these, then do a second change round where you update the cross-references in both repos. Changes that are truly interdependent and mutually required are not possible in this scenario. This is again unrelated to modules, or Go.

Modifying inter-dependent modules in the same repository

The problem.

This is a more curious situation and it is part of a wider problem that is actually acutely felt by multiple large-scale community projects (k8s, Prometheus, ...). I've done a much more elaborated discussion of this in the introduction of a design document for a tool to help deal with this specific situation.

Although it is possible to keep modules in the same repository relatively independent, we'll assume here that there are two and that they are quite strongly related. Them being in the same repository mainly opens up the possibility for developers to commit changes to both the modules in a single PR. A direct consequences of this is a likely need for some form of strong consistency in the versions that they are referencing (either one-way or in a circular dependency), ideally the same commit. This however would run into the same issue, even if we are in the same VCS, as what I described above for cross-repo inter-dependent modules.

The relative statement.

The relative statement suggested previously by @thepudds seems interesting to me for multi-module repositories, albeit with some changes. It might be worth dropping the idea of a "parent module" as the top-level might not be a module and the two modules might simply be same-level sister directories. We can then write (if both are in repo mydomain.com/foo:

module mydomain.com/foo/A

go 1.15

relative // or other TBD top-level directive

require mydomain.com/foo/B v0.0.0-00010101000000-000000000000

The new directive would not just apply for the top-level go.mod (like replace) but also for transitive go.mod files. Its effect would be to infer the required version for any dependent module that is part of the same VCS root to be at the same commit unless an explicit require statement exists, _i.e_ one that not uses the canonical pseudo-version as in the example above. The commit in question can be be translated back into the appropriate pseudo-version / tags for the dependent module via a module query, as usual.

The advantages are multiple from my perspective:

• We provide full backwards compatibility with any existing multi-module repo approach (which relies on explicit require statements).
• Users can opt-in when they want via the new directive. A requirement would be to set their go directive in their go.mod to at least the first Go version supporting replace, as else resolution would fail for earlier toolchains due to the canonical require version. This _could_ be circumvented in edge-cases by downstream users adding their own appropriate replace statement.
• Users can opt-in at a per-module granularity: in the same repo you can have one module using explicit require statements, whereas another uses the new relative approach. We incidentally also deal with the point raised by @bcmills about not wanting to have changes in one module _necessarily_ impact another one, even if in the same repo.
• We could envision making the relative-resolution opt-out in later Go releases to make it the default way of managing multi-module repos, a switch that would be conditioned on the go directive in a module's go.mod.

The potential downsides I can see:

• Although there are different ways of implementing this, it is likely that we would need to add a field to the data stored and returned by the proxy server to tell the VCS root for a given module. Or if we want to be resistant against hosting changes and provide more anonymity, a UUID corresponding to the VCS root that would be identical across the modules of a multi-module repo.
• We would need to determine the exact semantics around the major version used in import-paths for dependency versions resolved via relative. I have not gone through all edge-cases but I _believe_ that we can simply enforce the existing rules without exception, mostly because the opt-in for relative is local and limited to a given module. Hence, if the path for a relative-resolved module changes because of a major version change the importing module will simply need to update it's own import paths accordingly.

[triphocy]

Just wanted to add a hypothetical example that is very similar to a situation I'm facing right now. We are on go 1.13 and are trying to plan a migration from GOPATH to modules.

Assume that we have fully qualified package names - I'm just shortening them for this example.

Suppose we have a project called "govlc", a video player with some fancy UI. I want to separate out the video encoding/decoding engine into a library, "golibvlc" that others can use. I also want to separate out individual decoders such as "gox264" and "gorawvideo". These modules do not necessarily live on a centralized online repository (Git was built to be able to work in a decentralized way, after all).

For development, I set up a local project workspace directory with the following subdirectories (each an independent local Git project and a module):

• gox264, standalone and contains some proprietary code
• gorawvideo, standalone
• golibvlc, depends on gox264 and gorawvideo
• govlc, depends on golibvlc, and let's say this is proprietary and uses licensed code which should never go public

When working on "govlc", I'll also be working on "golibvlc" at the same time. I need my local changes available without needing to push "golibvlc", "gorawvideo", or "gox264" work in progress commits.

Let's say we hired an intern, and don't want to grant access to the main repository. We want him to work on a fork of "govlc" called "govlcfree". This fork only depends on "gorawvideo", and he'll be working on "gorawvideo" and "govlcfree" at the same time. He occasionally pulls from my personal copy of the repository, and will push acceptable changes directly to me for "gorawvideo".

From what I can tell, the current Go modules implementation makes these kinds of workflows difficult. It forces all modules to go into centralized, online revision control before it can be used by another project. As a result, a developer can't work on multiple modules concurrently.

We could use the "replace" keyword to make local references, but we must use relative paths, which are hacky (personal opinion) because we need to make assumptions of the directory hierarchy.

If we combine the components of the entire project into a single module, then we'd have to export the entire stack including all the proprietary parts.

We'd really like a way to use modules locally, so they can refer to each other without revision control or assumptions about directory hierarchy. I've seen a bunch of ideas - such as a go.mod.local - which may help. One other approach would be to treat go.mod as a template and provide environment variables in the expansion, such as:

replace gox264 v0.0.0 => {{ .Env.GOX264ROOT }}/gox264
replace gorawvideo v0.0.0 => {{ .Env.GORAWVIDEOROOT }}/gorawvideo
replace golibvlc v0.0.0 => {{ .Env.GOLIBVLCROOT }}/golibvlc

That way, we could even emulate the old GOPATH approach in our local project, like:

replace gox264 v0.0.0 => {{ .Env.GOPATH }}/src/gox264
replace gorawvideo v0.0.0 => {{ .Env.GOPATH }}/src/gorawvideo
replace golibvlc v0.0.0 => {{ .Env.GOPATH }}/src/golibvlc

Even better, go.mod could have syntax that allows overriding a requirement with a local map that prevents dependency checks and downloads _IF THE PATH EXISTS_, such as:

local (
    gox264 {{ .Env.GOPATH }}/src/gox264
    gorawvideo {{ .Env.GOPATH }}/src/gorawvideo
    golibvlc {{ .Env.GOPATH }}/src/golibvlc
)

That block marks those three modules as related projects that may exist on the local filesystem. The Go tools could use them instead of the managed version from require if the tools find a go.mod file in the specified local directory.