Go: cmd/go: emit an explicit error when packages in GOPATH/src/vendor overlap with GOROOT/src/vendor

[javasgl]

What version of Go are you using (go version)?

$ go version
go version go1.13 darwin/amd64

Does this issue reproduce with the latest release?

only in go1.13 happened, back to go1.12.9 can fix this

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/Users/username/Library/Caches/go-build"

GOENV="/Users/username/Library/Application Support/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GONOPROXY=""

GONOSUMDB=""

GOOS="darwin"

GOPATH=""

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/usr/local/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"

GCCGO="gccgo"

AR="ar"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD=""

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/b1/bhf9j3q96m75223l3hh41ccw0000gn/T/go-build173249759=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

go install -v ./...

What did you expect to see?

no errrors

What did you see instead?

src/vendor/golang.org/x/net/http2/frame.go:17:2: use of vendored package not allowed
src/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:28:2: use of vendored package not allowed
src/vendor/golang.org/x/net/http2/transport.go:35:2: use of vendored package not allowed

[mvdan]

Please provide full instructions to show that this worked on 1.12, and broke in 1.13. For example, using the official Docker images.

[bcmills]

@javasgl, to amplify what @mvdan said: go install -v ./... doesn't help us reproduce your problem unless you also include the contents of the module or repo that you were working in.

In particular, do the import statements in your code include the vendor/ prefix?

This change probably relates to CL 164619, but we did add a regression test in that CL to try to avoid any change in behavior for existing code using the correct import paths.

[psanford]

I think I ran into the same issue. I made an example repo here to demonstrate the issue: https://github.com/psanford/gobug34068

If you checkout that repo and set your GOPATH to the git working directory you can try building package example.com/foo from go 1.12 (works) vs 1.13 (errors). example.com/foo is a grpc client hello world example.

Grpc and its dependencies were vendored in src/vendor via gvt.

[javasgl]

@bcmills same project structure with @psanford

[bcmills]

Found a much smaller reproducer.

-- issue34068/issue34068.go --
package issue34068

import (
    _ "net/http"
    _ "golang.org/x/net/http2/hpack"
)
-- $GOPATH/src/vendor/golang.org/x/net/http2/hpack/hpack.go --
package hpack

[bcmills]

I looked into this today, but I'm not sure that it's feasible to fix.

The trouble is that a number of caches within cmd/go/internal/load assume (and have long assumed) that every package has a unique canonical import path, and import paths whose first component does not contain a dot are in general reserved for the standard library.

The simple failure mode is this:

1. First we load golang.org/x/net/http2/hpack from within GOPATH, and resolve it to the path vendor/golang.org/x/net/http2/hpack with location GOPATH/src/vendor/golang.org/x/net/http2/hpack.
   2 Then we load net/http from within GOROOT, and we _also_ resolve _it_ to vendor/golang.org/x/net/http2/hpack. The cache shows the location of that package to be GOPATH/src/vendor/golang.org/x/net/http2/hpack, but if we loaded it from scratch we would have gotten GOROOT/src/vendor/golang.org/x/net/http2/hpack.

A similar failure mode occurs with the relative ordering of GOPATH and GOROOT swapped.

A long-term fix would be to have all of the caches track packages by directory instead of import path. That change is fairly invasive, and in general we are not pursuing long-term fixes in GOPATH mode (although we do try to avoid regressions).

[bcmills]

Fortunately, there are (at least) two workarounds available:

1. You can move the vendor directory from the top level (GOPATH/src/vendor) to one level in (GOPATH/src/example.com/vendor). Then its path no longer conflicts with the vendor subdirectory in GOROOT.

2. You can convert the project to use module mode instead of GOPATH mode, in which case the vendor prefix is no longer added to the canonical path for the packages contained in the project's vendor directory (because there is only one vendor replacement for any given package path).

[bcmills]

Finally, I will note that this issue does not reproduce for any project compatible with go get. Since the vendor directory for any such project must be within its repository, go get would have placed that vendor directory within the projects subdirectory in GOPATH/src, not at the root of GOPATH/src.

[bronze1man]

Hi @bcmills ,
Here is my real problem that tigger this bug:

I have a git repository that use a submodule at GOPATH/src/vendor to store others' repository code, then I come across this bug.
As I need to compile my project with go1.9.7, so I can not translate my repository to use module mode (go version higher than go1.9 use a little more memory with my application, so it can not work in ios9 with extension process which limit the total memory to about 10MB).
I also try to write a program to symlink others' repository into GOPATH/src/xxx , but it do not work in windows 10 in go1.12 (I do not try go1.13 with this bug).

I think I have two solutions for this problem now for me to continue work with go1.13 and go1.9:

• I can try move GOROOT/src/vendor/golang.org/x/net/http2/hpack into another directory in GOROOT , like GOROOT/src/internal/golang.org/x/net/http2/hpack, and replace all the import in the GOROOT. This solution should not break my code. But I need do that again when I upgrade to go1.14
• Move all my code from GOPATH/src/xxx to GOPATH/src/a/xxx and move GOPATH/src/vendor to GOPATH/src/a/xxx. And replace all the import. But this solution will curse a lot of git diff in the import when I merge other branch of my repository.

[bcmills]

I also try to write a program to symlink others' repository into GOPATH/src/xxx , but it do not work in windows 10 in go1.12 (I do not try go1.13 with this bug).

Generally the go command works reasonably well with file symlinks, but not with directory symlinks.
(As far as I am aware, my solution for symlinking testdata inputs in misc/cgo/test works on Windows, but I'm not sure whether that approach works with Go versions older than 1.13.)

You could also consider copying the files instead of symlinking them.

[bcmills]

I need to compile my project with go1.9.7

Given that Go 1.9 is no longer receiving security updates, I would recommend that you reconsider that approach. (Have you filed an issue for the increase in memory footprint? Can you rework other parts of the application to make it work within the desired footprint with a supported Go version?)

[bcmills]

Finally, is there a reason you need to map in the submodule at GOPATH/src/vendor rather than GOPATH/src/$YOUR_DOMAIN/vendor? That should not change the import paths needed in the packages within $YOUR_DOMAIN.

[bronze1man]

Hi @bcmills ,

Generally the go command works reasonably well with file symlinks, but not with directory symlinks.

I tried to create directory symlinks in windows 10, so it do not work. Copying the files on windows 10 should be a good idea, but the build script and .gitignore file maybe another story.

Given that Go 1.9 is no longer receiving security updates, I would recommend that you reconsider that approach.

Another consideration of not using module mode is I do not know how to copy all the source code of dependencies of my project into one directory and then compile there right now.(looks like that is GOPATH mode.) The only way I can sure that my project can recompile success in a long time when I have all the dependencies in my git repository or in my own storage.(The network may fail, but my local disk will work in a long time.)

is there a reason you need to map in the submodule at GOPATH/src/vendor rather than GOPATH/src/$YOUR_DOMAIN/vendor?

I have a lots of packages in GOPATH/src/app/ep/main.go or GOPATH/src/make/main.go, they all need import same package in vendor, If I add vendor to GOPATH/src/config/vendor and GOPATH/src/app/ep/main.go. The binary size will increase and I will need symlinks or copy or vendor directories.

I have another workaround of this bug:

• I can use 2 GOPATH, first one is my project repository, second one is my vendor repository. It should only change my build script and the idea configure.

[bcmills]

I have a lots of packages in GOPATH/src/app/ep/main.go or GOPATH/src/make/main.go

Per https://golang.org/doc/code.html#ImportPaths:

For your own packages, you must choose a base path that is unlikely to collide with future additions to the standard library or other external libraries.

If you keep your code in a source repository somewhere, then you should use the root of that source repository as your base path. For instance, if you have a GitHub account at github.com/user, that should be your base path.

Package paths that do not start with a domain name are in general reserved for the standard library. That rules out dotless names like app and make, especially in module mode.

You should start your package paths with a domain or prefix that you control, and then you can root your vendor directory at that prefix too.

[bcmills]

Another consideration of not using module mode is I do not know how to copy all the source code of dependencies of my project into one directory and then compile there right now.

See https://golang.org/cmd/go/#hdr-Make_vendored_copy_of_dependencies and https://golang.org/cmd/go/#hdr-Modules_and_vendoring.

[bcmills]

Given the existence of workarounds, I think the way forward on this is to change the go command to explicitly reject collisions between GOPATH/src/vendor and GOROOT/src/vendor with a useful error message.

[bronze1man]

I can use 2 GOPATH, first one is my project repository, second one is my vendor repository. It should only change my build script and the idea configure.

• I have to config my ide idea to support two gopath, as i have 10+ projects, this should be a lot of work for everyone in my team.(the ide config can not git sync to others' computers)
• I have to change all my build script to support 2 GOPATH and add a config to my projects.

So here is my new workaround(which is less hard work for me right now):

• Place my git submodule of golang vendor into GOPATH/src/vendor.
• Modify golang source code, move the GOROOT/src/vendor to another place, and delete any check of not allow write code in GOPATH/src/vendor. And upload it to cdn and ask others to install this modify version of golang. This version of golang should work with all standard golang code, it should not break compatibility.

I think modify golang source code should be the best workaround for me now. It is I only need to modify once every golang release.

Here is the workaround bash script(in case me or someone need it again):

cd /usr/local/go
cp -rf ./src/vendor ./src/crypto/vendor
cp -rf ./src/vendor ./src/net/vendor
cp -rf ./src/vendor ./src/go/vendor
rm -rf ./src/vendor
cd ./src
./make.bash

[bcmills]

@bronze1man, forking the entire Go toolchain seems a lot more invasive than prefixing your import path with a domain name, or copying your vendor contents into GOPATH/src rather than mapping them in to GOPATH/src/vendor.

If the problem is an increase in binary size, have you filed an issue about that?

[bronze1man]

forking the entire Go toolchain seems a lot more invasive than prefixing your import path with a domain name,

I do not agree that, our team have 3 huge golang projects, each one have 100+ golang packages and 4000+ golang files, prefixing the import path into one a domain name means I need to rename almost all of the golang files. I do have a tool to do it automatic, but there will be a lot of path relative bugs to fix manually. That is a lot more invasive than modify and forking the entire Go toolchain.

By the way, I have modified the golang source code 1.13.1, and upload to our cdn , this workaround works great.

or copying your vendor contents into GOPATH/src rather than mapping them in to GOPATH/src/vendor.

• This workaround works and it is easy, but I can not use git submodule to manage all of the vendor packages. It will became difficult to change the vendor packages across our 3 huge golang projects.I may need to develop another tool to auto copy and compare the different of those vendor packages(As there are 3 copys of them in different projects)

If the problem is an increase in binary size, have you filed an issue about that?

• Binary size and memory footprint problem have been solved by use go1.9.7 right now. All golang version after 1.9 is worse.

It looks like the using of GOPATH/src/vendor has calcified within our team. If golang do not forbidden it in the begin (like version go1.0), then there will be a lot of usage of it, make it very hard to change.
I am sorry to tell you, I need to maintain a fork of golang source code from golang 1.13.

[bcmills]

@bronze1man, it occurs to me that there is one more possible workaround in GOPATH mode: namely, you can promote the top-level vendor subdirectory to a second complete GOPATH entry.

So you might do something like:

$ mkdir govendor
$ mv $GOPATH/src/vendor govendor/src
$ export GOPATH=$GOPATH:$(pwd)/govendor

Then you can keep building in GOPATH mode without the top-level src/vendor directory.

(That said, at this point I would strongly encourage you to migrate to module mode, and then the semantics of GOPATH/vendor are moot.)

[bronze1man]

@bcmills thanks for your explanation.

vendor has had a special meaning since Go 1.6 (https://golang.org/doc/go1.6#go_command). The ship sailed on reserving that particular prefix a long time ago.

I do not even notice this document when I first use GOPATH/src/vender since go1.6

I think I am trying to choose to do things with cheap cost i think, but this choose is not the same as golang project.
Do not upgrade golang version is the cheapest solution for my project right now.
Multiple GOPATH is another cheap solution.