Go: proposal: go/build: vendor directories are ignored when outside of $GOPATH

[aspeteRakete]

1. What version of Go are you using (go version)?
   1.6
2. What operating system and processor architecture are you using (go env)?
   ubuntu 16.04 x86_64
3. What did you do?
   I want to build a go project using gitlab-ci. The project gets clone on the build machine in /build/project, which is not inside the go path. The build fails due to import errors.
   I reproduced this manually: moving a project outside the GOPATH leads to ignoring the vendor folder.
4. What did you expect to see?
   go searching for imports in the vendor directory
5. What did you see instead?
   The vendor directory is ignored and go only searches in GOROOT and GOPATH

[ianlancetaylor]

The go tool is based on GOPATH. If you don't want to use GOPATH, it's hard to use the go tool.

Note that GOPATH can have multiple entries; see go help gopath.

I'm going to close this; we're not going to be adding support for things like the vendor directory outside of the existing GOPATH support.

[aspeteRakete]

Sorry, but this doesn't make sense to me. The vendoring feature is supposed to solve the problem of dependency management. I got all my dependencies except the go standard library as submodules of my project. Why am I forced to put this to a specific location in my filesystems? My GOPATH is set and the go tool searches for modules in there. When a project is located outside the GOPATH/src i would expect the go tool to search the vendor folder first and then the GOPATH for modules.

[bradfitz]

The design of the "vendor" mechanism is what you should expect: https://golang.org/s/go15vendor

Any deviations from that document should be filed as a bug, but this is not one.

Let's move this to the mailing list. See https://golang.org/wiki/Questions for links.

[trans]

Honestly I find it odd too. I don't get why the "go way" is so phobic to freely located project directories. This has been one of the big three pains-in-my-ass when working with the language.

[mcluseau]

Since the linked specification doesn't state anywhere that the GOPATH is required, making the go tool handle this case, for instance with an option to specify the package path of the current working directory, is not going to be bug. My understanding is that this issue is then a feature request, and thus doesn't deserve to be closed because it's not a bug.
In fact, the linked specification states «This proposal allows the Go community to move from builds that require custom GOPATH configuration beforehand to builds that just work», and one concerned by this feature request can interpret this failure as a deviation from this specification, since any build made where no GOPATH exists will require a custom GOPATH as a consequence.

[mcluseau]

Here's what works for me in GitLab-CI (building the required custom GOPATH to allow the use of vendoring):

build:
    script:
        - mypkg=x/y/z
        - rm -fr workspace
        - mkdir -p workspace/src/$(dirname $mypkg)
        - ln -s $PWD workspace/src/$mypkg
        - GOPATH=$PWD/workspace go build $mypkg/cmd/my-cmd

[quentinmit]

The design of the "vendor" mechanism is what you should expect: https://golang.org/s/go15vendor

Any deviations from that document should be filed as a bug, but this is not one.

This issue, as specified, is describing a deviation from that document. It simply says that, when compiling code located in "d/", the go command will look for imports in "d/vendor/$path". It makes no mention of GOPATH except in an example.

I'm going to reopen this issue to look at for Go 1.8. I think fixing this bug could be a key part of solving #12488.

[rasky]

While this is useful, it doesn't solve the problem for projects that have local packages (subdirectories) in addition to vendor packages.

If you just fix this (which makes sense), you're solving the GOPATH issue only for projects made of a single package. If they have sub-package, they would still need to use a GOPATH workspace.

[metakeule]

@rasky You _could_ put sub-packages inside vendor/lib/ or something similar. That may be considered misuse of vendor/, but it works and has not the problems of typical relative package paths. But it means that the vendor directory now must be committed to source control (at least the lib subfolder and no external tool may clean vendor).

[RichyHBM]

@rasky You can import packages via "./foo/bar", this seems to work fine with packages under the current directory. Not ideal but does the job

[rasky]

If that already works outside of GOPATH, then it's perfectly fine for me, and I agree it does job. Together with the modification described in this issue, it would allow development outside GOPATH (assuming a vendor tool that allows you to download/update dependencies directly within vendor/)

[RichyHBM]

Works fine for me, though I cant use it to access libraries inside the vendor folder odly enough (Gives an error saying the github.com/foo/bar should be in the GoPath)

Ofcourse If you just have packages directly in your roject folder, via submodules for example, then this should work I imagine

[erjoalgo]

Has there been any update on this? Why is Go so obsessed with forcing GOPATH on everyone and everything? https://github.com/golang/go/issues/3515

[mcluseau]

12488 is not going to focus on solving this, so this is still a request per se ;-)

[ascotan]

fetch:
    glide install
    @for dep in $(shell cd vendor; find * -type d -maxdepth 2 -mindepth 2); do \
      echo "Linking $$dep into $$GOPATH"; \
      rm -rf $$GOPATH/src/$$dep; \
        mkdir -p $$GOPATH/src/$${dep%/*}; \
        ln -fs $(shell pwd)/vendor/$$dep $$GOPATH/src/$$dep; \
    done

build: fetch
    ln -fs $(shell pwd) ${GOPATH}/src/gitlab.com/whois/$(shell basename $(shell pwd))
    go build -v

Because I personally have multiple repos in different languages and I just prefer to not install half of them in a GOPATH. Everytime I jump back into go I'm like "Yes I love programming in go! Now create a new project, add a dependency.... Oh god."

[dragonfax]

vendor is used only for storing source code (the repositories), not the generated binaries (executables and package objects). But these items are created and managed solely by the "go tools " and could IMHO be considered a cache, or rather transient.

So perhaps it wouldn't be out of the question to have the "go tools" default to using a path such as /tmp for the contents of "bin" and "pkg" when it finds that GOPATH is unset. Choosing some arbitrary directory like /tmp/ or ~/.<application> is a common practice for command line applications to stuff their generated files into.

[dragonfax]

With this, you have options:

• vendor/ but no GOPATH - Your project is fully self-contained in its repo using vendoring.
• GOPATH but no vendor/ - Your project shares its dependencies with your other projects.
• GOPATH and vendor/ - Dependencies you've vendored with your favorite tool are tracked in your repo, but other dependencies are shared with other projects using the GOPATH.

But GOPATH is completely optional, and left as just a "good idea".

You still get an error if you have no GOPATH and no created vendor/ directory.

[SamWhited]

So perhaps it wouldn't be out of the question to have the "go tools" default to using a path such as /tmp for the contents of "bin" and "pkg" when it finds that GOPATH is unset.

@dragonfax Note that in 1.8 this is already the case (more or less; see #17262); GOPATH is $HOME/go by default, and it can be overriden by setting $GOPATH.

[dragonfax]

@SamWhited That doesn't really solve the problem of this issue. The tools are still going to require your source to be under GOPATH. Even if its choosing a default GOPATH for you. This is about not requiring a GOPATH at all, let alone forcing the source to be under it.

[RichyHBM]

@dragonfax Agreed, @SamWhited it would be ideal to be able to have a go project in any directory of your computer regardless of where the GOPATH is pointing at, which at the moment is not the case as the go tools only recognise the vendor folder if it lies somewhere bellow GOPATH

[SamWhited]

Sure, sure; I was just pointing out that a default path (where build files can be put) will already exist in 1.8, not suggesting that it somehow solves the problem. Unless the point of using /tmp was to not polute the GOPATH with things built outside the GOPATH, in which case please disregard.

[dragonfax]

I didn't know about the default path addition. Thanks. With that, I guess I'm just suggesting loosening the requirement for source to be under GOPATH when its not really needed. i.e all deps under vendor/.

[go101]

I propose to split the GOPATH env into there envs: GOPATH_SRC, GOPATH_PKG and GOPATH_BIN.
This will make building a project outside of GOPATH easier.
We can just call

GOPATH_SRC=$PWD:$GOPATH_SRC go build

in the project folder to build the project.

Splitting the GOPATH env will make writing custom build shell much simpler.

[bradfitz]

@golang101, we already have $GOBIN. And ideally we'd get rid of the "pkg" directory entirely (#4719).

[go101]

The headache one is the src folder.
To add a path in GOPATH, the path must contain a src sub folder.
This is not always feasible.

[bradfitz]

@golang101, yes, we understand. That's what this proposal and #17271 are about.

[eikenb]

17271 seems to be in limbo as you yourself proposed shelving it to see the impact of $HOME/go (#17262). But I don't see how #17262 addresses the problem brought up by this ticket.

[bradfitz]

@eikenb, if by "shelving it" you mean taking things slowly and seeing the impact of one change before doing three things at once, yes.

[eikenb]

I have no problem with being methodical and taking time to do things. But the proposed timeline would make it about 2 years before considering a fix for this issue, which seemed a bit long to me.

[rsc]

There are many problems with running outside GOPATH. Among others, there's no caching of built object files, so everything is much slower than it should be (go build, go test, and so on). Vendoring is pretty low on the list of problems.

It may be that GOPATH should be reconsidered but if so it should be done as a whole change, not just one piece at a time, like @bradfitz said above. The solution for now is to use GOPATH.

Duplicate of #17271 if anything.

[ascotan]

17271 is a different but similar issue. That issue is that people should be able to have an auto-configured GOPATH. The resolution was proposed to auto resolve GOPATH to ./vendor but instead it would be autoresolved to $HOME/go

This issue is that I have a GOPATH configured, but I want to checkout code outside of that path and the ./vendor directory would override packages in the configured GOPATH.

[rsc]

OK, so not a duplicate of #17271 - updated my comment. The rest of my reply still stands.

[joeblew99]

what about JIri ?
https://fuchsia.googlesource.com/jiri/

[steelbrain]

I personally think that the GoLang contributors should give developers at least some amount of freedom. Closing an issue that is requested by so many because you think it'll be slower, for people who are already aware of it and are still requesting it because of how painful the alternatives are is not the best way to go. I have projects in monorepo that use golang for some components.

All other languages work fine but working with Go is a pain because of having to link individual modules into the GOPATH. We get that it'll be slow, but we want it anyway. IMHO a language should empower it's users instead of dictating them.

In my humble opinion, vendor directories should not be ignored outside of GOPATH.

[steelbrain]

FWIW, I've created a shell script for my project that I now use to compile go stuff. It temporarily links stuff into GOPATH, compiles and then unlinks. I get to keep my stuff out of GOPATH and into the repo and don't have to wait for any new features

[RichyHBM]

@steelbrain I agree!
Would you happen to have a gist of that script?

[bradfitz]

@steelbrain, the discussion of removing the requirement for $GOPATH is mostly happening in #17271. It is on our agenda to figure out in the next few months.

Please watch that issue but keep in mind https://golang.org/wiki/NoMeToo

I'm going to lock this issue so we can focus conversation in one place.