Go: cmd/go: allow replacement modules to alias other active modules

[mwf]

What version of Go are you using (go version)?

Go tip:
go version devel +f2131f6e0c Wed Aug 8 21:37:36 2018 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes (it is not reproduced with go version go1.11beta2 darwin/amd64)

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/ikorolev/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/var/folders/_b/d1934m9s587_8t_6ngv3hnc00000gp/T/tmp.cqU8g8OM/gopath"
GOPROXY=""
GORACE=""
GOROOT="/Users/ikorolev/.gvm/gos/go1.11beta3"
GOTMPDIR=""
GOTOOLDIR="/Users/ikorolev/.gvm/gos/go1.11beta3/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/var/folders/_b/d1934m9s587_8t_6ngv3hnc00000gp/T/tmp.cqU8g8OM/vgo-a-user/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_b/d1934m9s587_8t_6ngv3hnc00000gp/T/go-build138999780=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Sorry, no standalone reproduction, since issue is connected with repository forking

Assume we have a repository A: https://github.com/mwf/vgo-a with the only feature:

package a

var A = "A"

Than we have a fork1 https://github.com/mwf/vgo-a-fork1, adding a feature B :

package a

var B = "B is a new feature in a-fork1"

Unfortunately fork1 will never be merged to the upstream, just because a author don't like this feature.

It's important to note, that both a and a-fork1 don't have go.mod, they are too conservative for that 😄

Then we got a happy user, using both projects in his repo.
go.mod:

module github.com/mwf/vgo-a-user

require (
    github.com/mwf/vgo-a v0.1.0
    github.com/mwf/vgo-a-fork1 v0.2.0
)

main.go

package main

import (
    "fmt"

    "github.com/mwf/vgo-a"
    a_fork "github.com/mwf/vgo-a-fork1"
)

func main() {
    fmt.Printf("A: %q\n", a.A)
    fmt.Printf("B: %q\n", a_fork.B)
}

All just works fine:

$ go run .
A: "A"
B: "B is a new feature in a-fork1"

Here appears fork2 https://github.com/mwf/vgo-a-fork2, forked from fork1, and fixing some bugs both in the upstream and in fork1.

We use the fork2 with replace in our main repo: https://github.com/mwf/vgo-a-user/blob/master/go.mod

module github.com/mwf/vgo-a-user

require (
    github.com/mwf/vgo-a v0.1.0
    github.com/mwf/vgo-a-fork1 v0.2.0
)

replace github.com/mwf/vgo-a => github.com/mwf/vgo-a-fork2 v0.2.1

replace github.com/mwf/vgo-a-fork1 => github.com/mwf/vgo-a-fork2 v0.2.1

What did you expect to see?

Building this with go1.11beta2 works just fine:

cd `mktemp -d`
git clone [email protected]:mwf/vgo-a-user.git .
go version && go run .

Output:

go version go1.11beta2 darwin/amd64
go: finding github.com/mwf/vgo-a-fork2 v0.2.1
go: downloading github.com/mwf/vgo-a-fork2 v0.2.1
go: finding github.com/mwf/vgo-a v0.1.0
go: finding github.com/mwf/vgo-a-fork1 v0.2.0
A: "A, fixed in a-fork2"
B: "B, fixed in a-fork2"

What did you see instead?

Building with the tip (and beta3) returns an error:

cd `mktemp -d`
git clone [email protected]:mwf/vgo-a-user.git .
go version && go run .

Output:

go version devel +f2131f6e0c Wed Aug 8 21:37:36 2018 +0000 darwin/amd64
go: finding github.com/mwf/vgo-a-fork2 v0.2.1
go: downloading github.com/mwf/vgo-a-fork2 v0.2.1
go: github.com/mwf/[email protected] used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

More comments

I understand that this case is very specific and arguable - this should not ever happen ideally, but we have the real case here:
https://github.com/utrack/clay/blob/master/integration/binding_with_body_and_response/go.mod

There is a little workaround, to define go.mod at fork2 and make a replace upstream -> fork2_with_go.mod, but it's too dirty :)

replace github.com/mwf/vgo-a => github.com/mwf/vgo-a-fork2 v0.3.0 // version with go.mod
replace github.com/mwf/vgo-a-fork1 => github.com/mwf/vgo-a-fork2 v0.2.1 // no go.mod

It works with tip and beta3:

$ go version && go run .
go version devel +f2131f6e0c Wed Aug 8 21:37:36 2018 +0000 darwin/amd64
A: "A, fixed in a-fork2"
B: "B, fixed in a-fork2"

If you decide that the case is too specific and crazy, and you'd like to close as "Won't fix" - then I assume we should change the error string, because it's confusing now:

go: github.com/mwf/[email protected] used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

It should look like this:

go: github.com/mwf/[email protected] used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

because it's github.com/mwf/vgo-a-fork2 who's to blame for the error.

[mwf]

And this crazy stuff could happen only in non-go.mod repositories, because you can't use both upstream and fork at the same time if they have go.mod inside:

module github.com/mwf/vgo-a-user

require (
    github.com/mwf/vgo-a-mod v0.1.0
    github.com/mwf/vgo-a-mod-fork1 v0.2.0
)

$ go run .
go: finding github.com/mwf/vgo-a-mod-fork1 v0.2.0
go: github.com/mwf/[email protected]: parsing go.mod: unexpected module path "github.com/mwf/vgo-a-mod"
go: finding github.com/mwf/vgo-a-mod v0.1.0
go: error loading module requirements

You are forced either to use your fork as replace argument, or to change the module in fork's go.mod and never assume it as a fork again :) And this is for the best 👍

So no one will ever hit such an issue in the brave new world of go modules 😄

[bcmills]

Ideally, I think the long-term solution will be to treat replacements as rewriting the import paths rather than the source code, to allow for precisely this kind of fork-unification behavior.

(For an example of how this can go wrong otherwise, see the code in #26607.)

[mwf]

@bcmills you set "Go1.12" milestone for the issue.

Maybe we could at least fix the error string in Go1.11 to eliminate the confusion?

If you decide that the case is too specific and crazy, and you'd like to close as "Won't fix" - then I assume we should change the error string, because it's confusing now:

go: github.com/mwf/[email protected] used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

It should look like this:

go: github.com/mwf/[email protected] used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

because it's github.com/mwf/vgo-a-fork2 who's to blame for the error.

[bcmills]

Oh, yeah, that's an easy fix.

[gopherbot]

Change https://golang.org/cl/128878 mentions this issue: cmd/go/internal/modload: emit correct module in duplication error

[F21]

I am also seeing this problem when using go mod why. Since testcontainers/testcontainer-go was renamed to testcontainers/testcontainers-go recently, modules break for me since it is being used by one of my dependencies (not sure how deep).

I added:

replace github.com/testcontainers/testcontainer-go => github.com/testcontainers/testcontainers-go v0.0.0-20190108154635-47c0da630f72

to go.mod in order to get go get -u and go mod tidy to work correctly.

However, if I run go mod why github.com/testcontainers/testcontainers-go, I get:

go: finding github.com/testcontainers/testcontainers-go latest
go: github.com/testcontainers/[email protected] used for two different module paths (github.com/testcontainers/testcontainer-go and github.com/testcontainers/testcontainers-go)

[bcmills]

We should consider how to address #30831 in concert with this change. (Replacement aliasing seems clearly “necessary” for #30831, but we should make it as close as possible to “sufficient” too.)

[gopherbot]

Change https://golang.org/cl/174939 mentions this issue: cmd/go: use replacement, not replaced, module paths in binary

[Smityz]

I am confusing the situation,is there any solution can fix my problem(#32462 )?
I am sorry,i am a rookie in programming.

[bcmills]

@Smityz, it's not clear why you need those replace statements to begin with. If you need help setting up a project using modules (and have already checked http://golang.org/wiki/Modules for solutions), the venues on http://golang.org/wiki/Questions are likely to be more effective.

[Smityz]

@Smityz, it's not clear why you need those replace statements to begin with. If you need help setting up a project using modules (and have already checked http://golang.org/wiki/Modules for solutions), the venues on http://golang.org/wiki/Questions are likely to be more effective.

In China.For some reasons,we can't access golang.org or other repository,the only repository we can access is which in github,so I have to do that to get my package.

[agnivade]

Use go mod edit -replace=<src>=<target> to replace golang.org modules by github.com paths. See https://github.com/golang/go/issues/28652 for more examples and context. Use http://golang.org/wiki/Questions if you have further questions.

[bcmills]

replace in general needs some rework, which we plan to investigate in conjunction with broader improvements to the MVS / module loading process.

The right syntax for this sort of replace depends on what we can (or can't) eliminate at that point, so putting on hold until 1.15 so that we can tackle all of these holistically.

[mwf]

Here is one more example for further investigation.

Half of the company still uses dep for vendoring, but we've got migration to modules in progress.

Since dep knows nothing about /vX library notation (https://github.com/golang/dep/issues/1962), we can't switch to versioned import paths, but in order to use modules we have to use replace directives.

Imagine we have library A, library B and service S.
Libraries are used both in dep-based and modules-based projects.

Library A go.mod:

module go.avito.ru/gl/A/v7

...

Library B go.mod looks like:

module go.avito.ru/gl/B

require(
   go.avito.ru/gl/A v7.1.0+incompatible // we need this in order to preserve dep compatibility in Gopkg.toml
)

replace go.avito.ru/gl/A => go.avito.ru/gl/A/v7 v7.1.0

And service S has

module go.avito.ru/av/S

require (
    go.avito.ru/gl/A/v7 v7.1.0 // direct dependency on module-based A lib
)

replace go.avito.ru/gl/A => go.avito.ru/gl/A/v7 v7.1.0

The service S build ends up with error:

$ go build ./cmd/service
go: go.avito.ru/gl/A/[email protected] used for two different module paths (go.avito.ru/gl/A and go.avito.ru/gl/A/v7)

What helps us - go mod vendor somehow bypasses this error (have you made it intentionally?) and vendors go.avito.ru/gl/A under both passes in vendor tree.
Thus go build -mod=vendor ./cmd/service goes without any error.

PS I believe this problem is much more of real-world than the original one in my issue. So if you like, I could edit the original one in favour of an updated case.
Or better just leave it as is?

[bcmills]

@mwf, note that the modules using the major-subdirectory layout should work with versioned import paths with both dep and modules.

A replace directive should not be necessary, especially since that forces all downstream consumers of your module to replicated that directive in their own go.mod file in order to successfully build.

[mwf]

Yeap, thanks!

We use simple flow with latest major in master branch. I'm not sure if we could switch to subdirectory layout for now.

[dmitshur]

This issue is currently labeled as early-in-cycle for Go 1.15. That time is now, so friendly ping. If it no longer needs to be done early in cycle, that label can be removed.

[kolaente]

I'm hitting this with a depdendency of a dependency of a dependeny of a dependency (with the exact dependency from #37911). The interesting thing is that _none of the packages used in my application_ actually use the faulty package. They all actively use the fixed version, but have a reference to the broken one in their go.sum file (the go.mod file is fine everywhere).

Edit: I was able to """fix""" that by forking the module, changing its path and then adding a replace to my go.mod to use the fork with the change package name. While this works, it is very dirty and should not be necessary.

[kevinburkemeter]

Just a heads up that one of the linked answers is the top result on Google when you search for "used for two different module paths" so it might be good to provide some easy guidance here on how to solve that problem. As it stands you have to read a lot to get your head around how this solves the issue.

[dmitshur]

This issue is currently labeled as early-in-cycle for Go 1.16.
That time is now, so this is a friendly ping so the issue is looked at again.

[bcmills]

I'm dropping the early-in-cycle, label on the theory that (empirically) hardly anybody tests the more advanced details of module mode until at least the beta anyway.

[mvdan]

To add another real case where this is causing pain: github.com/gopherjs/vecty has been moved to github.com/hexops/vecty, but dozens of library modules like github.com/marwan-at-work/vecty-router still require the old module. I tried upgrading vecty to master and then using replace github.com/gopherjs/vecty => github.com/hexops/vecty master to force third party libraries to come along for the upgrade, but encountered this issue.

cc @marwan-at-work in case he's open to fixing this in his module directly.

[marwan-at-work]

@mvdan done on the vecty-router side thanks to @NateWilliams2 https://github.com/marwan-at-work/vecty-router/pull/12

[iamkirkbater]

Another case where this is causing pain:

An upstream dependency (A) is pointing to a git repo that has been deleted (B). Another dependency (C) is using a fork (D) of the repo that has been deleted. The fork (D) and the repo that has been deleted (B) are both exactly the same, just B no longer exists.

So a simplified Dep tree would look like this:

Our Project
  | - Dependency
  |   | - Dependency A
  |   |   | - Package B (now deleted)
  | - Dependency C
  |   | - Package D (Fork of B with no changes)

As a stopgap to continue our builds until the upstream dependency (A), we thought we could just replace B with D, but that's throwing the "X used for two different module paths", even though that's exactly what we want to do.

[Dragomir-Ivanov]

I have another issue with 1.15.3. Maybe I need to file another Issue, but will post here for now:
Having these in go.mod

replace gopkg.in/mgo.v2 => ../mgo

replace github.com/globalsign/mgo => ../mgo

Results in: go: ../mgo@ used for two different module paths (github.com/globalsign/mgo and gopkg.in/mgo.v2).
I feel that this should be allowed.

[bcmills]

@Dragomir-Ivanov, no need to file a separate issue. This is next on my list after #36460, although it might not make 1.16 before the freeze.

However, note that github.com/globalsign/mgo imports packages from itself. The replace directives that you describe, if they were allowed today, would result in two distinct copies of each package (one beginning with github.com/globalsign/mgo, and one beginning with gopkg.in/mgo.v2), but with each copy _importing_ only one other copy (presumably github.com/globalsign/mgo?).

That could well produce type errors if types from the imported packages are used in the exported API, since each distinct package also defines its own types.

It seems that what you ideally need is exactly what we haven't implemented yet, which is a way to reinterpret import statements that refer to one of those copies as if they instead referred to the other.

[Dragomir-Ivanov]

@bcmills Thanks for your reply and all the efforts! I workaround my issues so far, by making private fork, and replaced all self import paths in github.com/globalsign/mgo to gopkg.in/mgo.v2.
In any case, I really think that once you replace a package, go toolchain should implicitly redirect replaced package self-imports to the replacer.

There are lots of abandoned packages, then revived by someone else and continued (as this mgo one), and we need a clean way to replace the package in the whole project(thus replacing package in our dependencies as well) without any forks, source modifications, etc.

I am in no rush what so ever, and can wait for 1.16, 1.17, etc.
Also I am thinking in what takes if we want to replace a package only for certain dependency packages. This can quickly become NPM :)