Go: proposal: cmd/go: automatically detect GOPATH

[adg]

Setting up a workspace/GOPATH is one of the big hurdles for new Go users. And maintaining (switching between) several workspaces is a common work flow for some Go developers.

This proposal is to make the GOPATH environment variable optional, and to have the go tool detect the GOPATH root on invocation. It would to this by walking up the tree from the cwd to find a directory named src, and setting GOPATH to the parent of that directory.

For new users, the getting started experience is something like:

mkdir -p src/github.com/user/hello
cd src/github.com/user/hello
vim hello.go
go install
hello

or to install something else:

mkdir src
cd src
go get github.com/docker/docker

Thoughts?

[broady]

/cc @jbuberel who was talking about this this morning.

[davecheney]

@adg thanks for raising this proposal. I thought it was going to be more along the lines of "lets have a default GOPATH value (and it might be $HOME)"

mkdir -p src/github.com/user/hello
cd src/github.com/user/hello
vim hello.go
go install
hello

I like this, but I think there are still a number of edge case that have to be explained en rote.

q. why do I need to make a sub directory, can't go just work in my current directory ?
a. no, not really, Go works with packages, and packages are a directory

q. oh, ok, well I put my code in src/hello.go and it didn't work !
a. ahh, well, src is special, you cannot put code directly in src/

q. ok, so I put my code in gocode/hello.go and it didn't work, you told me src/ was special so I didn't use it
a. err, right, sorry, you need to use src/, but you can't put any code in that, try src/hello/hello.go

q. ok, thanks for helping, that worked, but I have a question, why do I need all these directories, shouldn't Go be able to figure this out ?
a. _insert long technical explantation about package names being derived from import paths and how src/ would name no import path so technically not be importable into other package, not that that it matters 'cos its a main package_

q. what's a package, i just wanted to try helloworld.go ?!?

This is taken pretty much verbatim from my experiences having to justify the $PROJECT/src directory in gb.

[mdempsky]

I'd like to suggest that any automatic strategy to compute GOPATH should work anywhere within $GOPATH, not just within $GOPATH/src. E.g., it should at least also work within $GOPATH/bin and $GOPATH/pkg.

Easy variation of the proposed scheme is instead of looking for a parent directory named "src" (and then using _its_ parent as GOPATH), look for a parent directory with a "src" child directory (and use the parent itself as GOPATH).

Another option would be to add a new identifying file within the root directory. E.g., Bazel uses an empty file named "WORKSPACE" (see "Getting Started with Bazel") and Chromium's GN uses a file named ".gn" with config settings (see "What you need for a minimal GN build").

[minux]

I think we don't need to support $GOPATH/bin and $GOPATH/pkg,
because I can't think of a reason why I'd in one of those directory
and still use go build/install/test. Just handle the common case looks
good to me.

I also don't like introducing a new file to mark GOPATH.

[adg]

@davecheney I think at some point people need to learn about packages. I don't think we should design our entire build process around the very first new user experience. Can you suggest a way of avoiding this conversation that doesn't throw out everything we already have? (My gut reaction is "just tell them to put it in "src/hello" in the first place.")

@mdempsky I concur with @minux - I have no idea why you would care about resolving $GOPATH inside pkg or bin. I never change to those directories, ever. I also don't see the purpose in adding special files, when we already have a special directory (src).

Another consideration for this proposal is that it will theoretically break anyone that already has packages whose import paths have a src path component.

[hallas]

:+1: This change makes sense. I don't see what a default $GOPATH (for example at $HOME) would solve in terms of making it easier to get started with Go. I think if a user knows about $HOME (its significance) then he is proficient enough to setup a correct $GOPATH.

[davecheney]

@mdempsky I think the idea of "GOPATH is the parent of a directory called src/" works well.

@adg

I think at some point people need to learn about packages.

Agreed.

I don't think we should design our entire build process around the very first new user experience.

I'll have to disagree with you to some degree there :smile:

Can you suggest a way of avoiding this conversation that doesn't throw out everything we already have? (My gut reaction is "just tell them to put it in "src/hello" in the first place.")

I think I probably conflated a number of things in my response. From my experience there are several common pain points for newcomers when they see a path like src/github.com/user/hello

1. Why do I have to setup GOPATH, why can't I just put my stuff anywhere I want (or more likely where I put it in $INSERT_PREVIOUS_LANGUAGE). To that, I agree with you, the answers is "Because reasons".
2. The second is why do they have to put things in a subdirectory of a subdirectory src/ ? Again, it's probably better to stick with "reasons".
3. The third is the confusion that putting a github'ish url in the path. We've seen lots of users who think they have to push to github to compile.

Maybe this is just a matter of introducing the terms in order. Something like

---

There are two pieces of information you need to know to write Go code. The first is all Go code must live inside a workspace, we call that GOPATH. A GOPATH workspaces is any directory that contains the following three directories src/, bin/, and pkg/. You can create one like this

mkdir -p gocode/{src,bin,pkg}

So, gocode is your workspace, your GOPATH. gocode/src is where you will put your Go packages (we'll get to that in a second), gocode/bin is where the compiler will place any programs you compile, and gocode/pkg is the location that the compiler will cache files for later reuse to speed up compilation.

As mentioned above, all the Go code you write must be arranged into packages. What is a package ? A package is just a directory. The name of the directory must match the name of the package declaration that every file in that directory shares, that's why we call them packages.

Putting these two pieces of knowledge together, if you want to write a program called hello, it needs to be in the hello package, and knowing that packages need to live inside your workspaces' src/ directory, the path to this package must be gocode/src/hello

---

This explanation obviously trades off some degree of accuracy to avoid getting bogged down in the details, please don't beat me up for this.

[nightlyone]

What about go init [-path] where path defaults to $HOME/gopath and be done with it?

That could initialize the path, create an directory example too with a minimal main + package and fill this with some dummy content working on all major systems. It then also shows what it did (e.g. go get -v -x example/...), where it puts the stuff and what it actually is (e.g. a minimal binary, an example package, etc.)

When the $GOPATH is missing, suggest setting $GOPATH or calling go init to the user, if he doesn't know what we are talking about.

(Linux- )Distributions could then also easily integrate go init, if their policy needs defaults.

(Appropriate defaults for e.g. Windows are left out for the moment but can be easily added)

[mortdeus]

Is there a reason why we can't just build our default gopath in $GOROOT? Just add a $GOROOT/usr/$USER directory which is populated with it's own /bin, /src, /pkg and auto generated by the go tool.

Its one thing to have the developer explicitly state where they specify $GOPATH resides on their system, its another to allow go to automatically dig around for one.

[tylerdmace]

I think the original proposal is fine for those who understand package and workspace structure but I don't believe it to be helpful to new Go users as was suggested. The problem is that it's very likely that new Go users will be creating test directories within existing directory trees that contain code of different languages -- and src is likely to be a directory that exists all over within that same tree.

[mortdeus]

The go tool doesn't care if you have scripts/source written in other programming languages stored inside the same directory it found a set of *.go files in. It only cares that each declared package is confined within its own directory inside "$GOPATH/src".

I "get" what @adg is saying. Ideally when you use the command go install or go build, you have fed the tool a path (either by an explicit declaration or an implicit assumption that $PWD is your intended target). Given that the go tool already knows where the package you want to build resides, all it has to do is hop out into each subsequential parent directory until it finds itself inside a parent directory called "src". Which then the go tool can deduce that ../src is either a gopath that is already known to the go tool, or it's a base directory where the go tool can construct a new gopath if it is found that one doesn't already exist at said location.

I mean it definitely could work. It's just a really hooky programming environment convention if you ask me. It doesn't seem like it would always play nicely with Unix's file system security model either. (you develop your pkg in /usr/src, go installs executable in /usr/bin, which could bring about potential issues like system util name conflicts, or confused package managers, etc etc.

Personally, I think @adg's proposal makes the go tool far too "magic wand"-ey, for the amount of convenience the language would gain from his proposal's implementation.

[tylerdmace]

My concern wasn't so much that the code of other languages would be an issue; only that src is a common directory name and I figured there would be some potential conflicts in just assuming that you were working with a Go package's src directory when in fact it could be something entirely different. You gave a good example, @mortdeus -- /usr/src could be a conflicting directory if they happened to be working out of it. It's not likely but it could happen.

[mortdeus]

Also if we did do this we would have to adopt a policy that each gopath would be isolated from each other's view unless GOPATH has been explicitly set. The reason being the fact that the go tool can't resolve a lookup path precedence ambiguity with out a directorial cue from the user that expresses the preference of their intent.

[tylerdmace]

What if go could configure workspace configurations?

$ go create workspace /home/tyler/myworkspace

Then from anywhere you could do something like:

$ go myworkspace build

Maybe allow for a default or an active workspace:

$ go active workspace myworkspace

With an active workspace set you could simply do:

$ go build

The workspace name would be derived from the path passed at creation and could be referenced as simply.

To facilitate this, you’d need to allow for a GOCONF environment variable to point to a location where a simple config file (containing workspace names and GOPATHs for each) could be written/read by the tool and at that point perhaps the complexity of the situation is just the same; Thoughts?

[mortdeus]

I was also thinking about the idea for a go "workspace" project container object that essentially gives users one last degree of expressing package addressability. You wouldn't have to manually create, set, manage, and/or mess with them in any explicit way really.

Rather the workspace mechanism would work by walking up the tree until we find ourselves inside of a "src" directory. Once there we "cd .." into the parent base dir where we ideally want to set up the gopath we associate with the pkg were building. Since that parent base directory must have an identifier, we can use it to declare a workspace context for that GOPATH.

Syntactically the "workspace" identifier is treated just like a top level domain in all its usage contexts. For example if we wanted to call an executable that is in a specific $GOPATH/bin, we can just call go tool workspace.execname. Which is useful considering the fact that that GOPATH/bin might not be appended to the system path and would otherwise require us to call the exec using its full canonical file path identifier.

In the context of it's import path string, it would be just a top level domain.
import "workspace.gorepo.com/path/2/mypkg"

Then in $GOROOT, go get will fetch and install to $GOROOT/users/$USER/$workspace/src/path/2/mypkg

In the case that the top level domain is attachted to say github.com. We can just use the top level domain syntax to allude to a sub branch and/or tagged commit. And just create the workspace context name by appending the pkgname in front of the top level domain.

So lets say we have import "v2.github.com/mortdeus/mypkg", the go get tool will see the v2, then the github.com, then the user name, and then the pkgname. Go get will resolve this to the directory $GOROOT/users/mortdeus/mypkg.v2/src/github.com/mortdeus/mypkg

Then if we have a few executables called say mypkg/server, mypkg/client, mypkg/shell, mypkg/main
we can call the executables like 'go tool mypkg.server', 'go tool mypkg.shell', 'go tool mypkg.main' (or just 'go tool mypkg' in the special case of mypkg.main).

Idk its just an idea.

[mortdeus]

oh and in the case where we have a set of workspace ambiguity where we have say $GOROOT/users/{me | you}/gopkg/src/github.com/{my|your}/gopkg

Obviously we have a conflict here regarding which "gopkg" executables are tied to the globally resolved generic "gopkg" workspace identifier as far as the go tool is concerned. The best way I can think of resolving this is to have a $GOROOT/users/mortdeus/bin that links to the workspace's executables we prefer to be called in the case of a naming conflict.

[yiyus]

Another consideration for this proposal is that it will theoretically break anyone that already has packages whose import paths have a src path component.

A solution could be to say that the GOPATH is the parent of the top (instead of the bottom) src directory in the current directory. So, for example, when running go install from /home/user/src/github.com/project/src/package1, GOPATH would be set to /home/user.

I think it is more unlikely to break something this way and with the right wording it is not more complicated (I am assuming the fraction of users with GOPATHs inside src directories is neglegible). For new users, I think the most important point is showing a sensible error message when trying to build a package outside of a GOPATH (or with this proposal, outside of a src directory).

[XANi]

shouldn't it be something like $HOME/src/go instead ? src/ is a very generic name and there is a decent chance someone will be using it for something else than golang

[crawshaw]

I like @adg's proposal at the beginning of this issue of determining the GOPATH from the current working directory if the environment variable is not set.

There are several other interesting ideas proposed here, like a default GOPATH and a go init subcommand, that are compatible with the original proposal. They could be done instead of or as well as automatically detecting the GOPATH. Are any of them arguments against determining the GOPATH from the current working directory? If they are complementary they can be considered separately. If they are arguments against @adg's original proposal, I think it would be helpful to frame them as such.

[bictorman]

In my opinion auto-detecting GOPATH via src is too magical and can lead to unintended behaviour, specially since src is ubiquitous on developing environments.

As a multiple workspace and direnv user, I can vouch for a file .gopath generated via go init. It's unambiguous, most people are already familiar with the pattern thanks to git, and enables more extensibility as a potential source of metadata, like multiple paths, build flags, etc.

[ngrilly]

@adg's proposal would be a nice improvement to the tool chain. It doesn't change anything when GOPATH is already set, and when it's not, it simplifies the workflow. Many tools like git, hg or Node/npm already successfully implement a similar behaviour.

[rsc]

I really like the idea of auto-detecting GOPATH. I've been thinking about how to do this ever since I saw that @davecheney had done the equivalent in gb.

I'm not as sure about how. Finding src does seem to be the obvious choice, and it may work well. On the other hand, it may lead to wrong or confusing behavior if it found an unrelated src. At the least I think you'd need to say it's the highest-level src, so that if someone names their GitHub repo src and you're in /home/you/src/github.com/otherguy/src/pkg that uses GOPATH=/home/you.

Then there's the problem that any directory-based detection fails when you're outside the directory. You might be in $HOME or $HOME/bin and realize you need a command and run "go get that/command" and it fails because you're not in the src tree. That's annoying.

@davecheney's other points based on experience with gb are interesting too.

Counter-proposal, as suggested by Dave: If GOPATH is not set, it defaults to the user's home directory (aka $HOME on Unix, $USERPROFILE on Windows, $home on Plan 9).

The counter-proposal is not as flexible, but it's the suggested setting, it works outside the source tree, and it can't ever be confused by an inadvertently named parent directory.

I put the counter-proposal here mainly to play devil's advocate: if we're going to autodetect, we should make sure we are considering all the possible ways, not just one, and then pick the best.

As I go through the details, though, I'm starting to like default $HOME better than default magic search based on current directory.

[davecheney]

@rsc thanks for your comments.

re: finding the wrong src directory in a hierarchy. I think this is quite
unlikely, to the point that the advice could be 'don't do that'. There are
some Go repos on github (not naming any names, but those may be children of
the alphabet group) that include a '/src/' directory in the checkout, but
they could be said to be in the former category and should be fixed.

re: not being able to do 'go $CMD' anywhere on the filesystem is quite a
limitation. In both this and the previous case gb punts on this by making
it clear you arrange your work into projects and decide which project you
are working on by being present inside that directory -- a model well
understood by most git/svn/hg users. As a backup there is a -R flag to
override the root of the project detection logic, analogous to -C in modern
versions of git.

With that said, the place where this will bite will be go get from
scratch, that is to say, the first user experience; install go; go get
$BIGPROJECT; confusing error message as there is no /src/ to be found on
the filesystem and the user was following some potted advice from a project
README, not our getting started documentation.

In the face of this, defaulting $GOPATH to $HOME would

a. solve all of the issues above
b. mean that in the average case users have to set zero environment
variables to use Go, a moderate win for posix users, and a substantial win
for Windows users.

And, as you say, if people don't like the default, all they have to do is
set $GOPATH, just as they have for the last 5 years.

On Sat, Oct 24, 2015 at 2:54 PM, Russ Cox [email protected] wrote:

I really like the idea of auto-detecting GOPATH. I've been thinking about
how to do this ever since I saw that @davecheney
https://github.com/davecheney had done the equivalent in gb.

I'm not as sure about how. Finding src does seem to be the obvious
choice, and it may work well. On the other hand, it may lead to wrong or
confusing behavior if it found an unrelated src. At the least I think
you'd need to say it's the highest-level src, so that if someone names
their GitHub repo src and you're in /home/you/src/
github.com/otherguy/src/pkg that uses GOPATH=/home/you.

Then there's the problem that any directory-based detection fails when
you're outside the directory. You might be in $HOME or $HOME/bin and
realize you need a command and run "go get that/command" and it fails
because you're not in the src tree. That's annoying.

@davecheney https://github.com/davecheney's other points based on
experience with gb are interesting too.

Counter-proposal, as suggested by Dave: If GOPATH is not set, it defaults
to the user's home directory (aka $HOME on Unix, $USERPROFILE on Windows,
$home on Plan 9).

The counter-proposal is not as flexible, but it's the suggested setting,
it works outside the source tree, and it can't ever be confused by an
inadvertently named parent directory.

I put the counter-proposal here mainly to play devil's advocate: if we're
going to autodetect, we should make sure we are considering all the
possible ways, not just one, and then pick the best.

As I go through the details, though, I'm starting to like default $HOME
better than default magic search based on current directory.

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/12488#issuecomment-150750387.

[XANi]

I'd argue that $HOME/src might be pretty common place to put source code by developers and crapping all over it by default doesn't sound like a great idea, even something like $HOME/src/go would make it much better

[adg]

On 24 October 2015 at 14:54, Russ Cox [email protected] wrote:

As I go through the details, though, I'm starting to like default $HOME
better than default magic search based on current directory.

Sounds good to me.

I note the concerns about "crapping all over a user's home directory." But
I think putting stuff in $HOME/src isn't too objectionable, especially
since we make directories that directly reflect where the source came from
("github.com/...", etc).

I can foresee objections to $HOME/pkg being created automatically, but
issue #4719 might solve that for us.

Is $HOME/bin a problem? Seems like the logical place to "go install"
commands to IMO.

[XANi]

@adg why not $HOME/src/go ? Much less chance to collide with anyone's workflow ( I know few people using src/{source}/{repo} as their layout ) and if for some reason you need to clear/remove your environment you just remove one dir instead of sifting thru anything else you might have in src

[adg]

Because that would be change the way $GOPATH works.
The proposed change is to set a default GOPATH, not to change its
semantics.

On 13 November 2015 at 15:54, XANi [email protected] wrote:

@adg https://github.com/adg why not $HOME/src/go ? Much less chance to
collide with anyone's workflow ( I know few people using
src/{source}/{repo} as their layout ) and if for some reason you need to
clear/remove your environment you just remove one dir instead of sifting
thru anything else you might have in src

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/12488#issuecomment-156321155.

[bradfitz]

Until #4719 (killing the "pkg" dir, turning it into a cache in a system-specific user cache location), I can't get excited about GOPATH=$HOME. I'd be happy with GOPATH=$HOME/gopath as @nightlyone proposed earlier. It has the advantage that when we say "in your go path", it has the double meaning of either $GOPATH or "$HOME/gopath".

[rsc]

We could always rename pkg to gopkg.out.

[bradfitz]

We could always rename pkg to gopkg.out.

It's still kinda a turd in $HOME, whatever it's called. (even if it starts with a dot, which I know is sacrilege)

I'd prefer to address #4719 before doing an automatic $GOPATH.

[rsc]

gopkg.out was a joke (reference to gopkg.in). We can't reasonably rename pkg/ today.
I agree that creating $HOME/pkg automatically is not ideal.
But $HOME/gopath seems clearly the wrong default too.
That seems to exclude a static default.

That still leaves the dynamic approach of walking up from the parent directory looking for a src/ element (or perhaps looking for a dir with a src/ subdir, so that the go command would work in the bin directory too). Thoughts there? @davecheney, you've offered somewhat balanced pros and cons for the automatic detection; overall do you think it is a good idea for GOPATH?

[XANi]

It certainly makes a lot of things work out of the box, with "autodetect" setting gopath is no longer needed for usage like "run tests on CI server" (where $HOME is /var/lib/jenkins and you do not want to litter that but keep per-job files to its workspace dir) and building newly downloaded app "just works" (if it have src dir created ofc) even if your "main" dir have some old versions of package

[freeformz]

+1 on defaulting to $HOME if no $GOPATH is set. $HOME/pkg is ugly, but could be solved later.

[casey]

I think recursing upwards and looking for a file, like .gopath or .go to determine the $GOPATH is the best bet. I personally use $HOME/src for other things, and would be annoyed if go started using it. Also, src is present in many unix systems, see https://www.freebsd.org/cgi/man.cgi?hier(7) and https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard.

[dmitshur]

Is this proposal about helping people who don't know/understand GOPATH workspaces be able to use Go? Or is it about making it more convenient for people who know it but deal with many systems, CI, etc.?

[davecheney]

The former. Opting out of automatic detection is way, just set GOPATH.

On 11 Dec 2015, at 10:30, Dmitri Shuralyov [email protected] wrote:

Is this proposal about helping people who don't know/understand GOPATH workspaces be able to use Go? Or is it about making it more convenient for people who know it but deal with many systems, CI, etc.?

—
Reply to this email directly or view it on GitHub.

[Skarlso]

Please don't pollute $HOME with a default $GOPATH. It's already very crowded. I'd go +1 for $HOME/whatever.

I mean in that case, the delinquent would see a $HOME/src as well? And a $HOME/bin??? That's confusing as hell. Let's keep go where it belongs, in a neat module in it's own directory of some kind.

Couple of choices:

• $HOME/gopath ( I have no problems with that )
• $HOME/gohome
• $HOME/godefault
• $HOME/go (simple, yet elegant)
• $HOME/goworkspace
• $HOME/gouserspace
  ...

[rsc]

I am putting this proposal on hold.

[adg]

The time has come to revisit this, I believe.

[atomi]

why not? GOPATH=$(pwd):$(pwd)/vendor:$(somedefault_butpleasenotinhome)

Just to move the conversation forward Node has both local packages to the project and a global install location. But to be fair, I haven't had any issues prefixing go build with gopath.

[groob]

since nobody mentioned it already, linking the PYTHONPATH documentation here. By default, python uses _prefix_/lib/python_version_

https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH

[LaPingvino]

Another idea:

• Add the option to mark gopath by a special file in the directory (.gopath for example)
• Create a go path tool that checks if gopath is set, if not use said heuristics to find the probable directory and ask if this should be your gopath, if so, set the .gopath marker. After running always return/set correct gopath.
• When running a tool that needs gopath, use go path tool to get gopath if it's missing.

Advantages:

• Existing config doesn't change
• New users will have a gopath setup, but can refuse
• Heuristics can be applied
• Systems without working environment variables for any reason can configure a persistent gopath
• Git style did you mean education which can be improved on

Disadvantages:

• Heuristics may take time... (possible solution: if this happens, strongly suggest setting $GOPATH, also globally set $GOPATH at first run of tool)

[mem]

At some point I had something like this: https://gist.github.com/mem/21b0db6e6f672ba7c46cab457c34ae2b (I just rewrote and uploaded it for discussion purposes)

I share @bradfitz's view about setting GOPATH=$HOME, particularly because it tends to overwrite things I have purposefully put in ~/bin (and I don't like seeing the pkg dir there).

The _one_ problem I have with this is that it doesn't do anything for this use case:

$ cd ~/random/dir
$ mkdir -p src/hello
$ edit src/hello/main.go
# edit here
$ go install hello
$ hello
-bash: hello: command not found

this happens because GOPATH gets set to ~random/dir, but nothing happens with PATH (nor should it). What happens now is that you have to start telling people where the binary ended up in. In this specific case, the alternative is to tell them to type bin/hello instead of just hello, but that doesn't feel like a solution.

[quentinmit]

I have a counterproposal:

When $GOPATH is unset, behave like "git clone" or "svn checkout" - place the requested package in the current directory. Place the dependencies in a "vendor/" subdirectory under the newly-created directory.

Thus, each invocation of "go get" yields a self-contained directory which can be used to build and develop the go-gotten package.

Downsides:

• "vendor" doesn't currently work without $GOPATH (#14566); arguably this is just a bug
• This would leave you with an unclean working copy, or alternately, "go get" could exclude the vendor directory it creates, but this gets confusing if the repo already contains some checked-in vendored packages.

Motivation:

• Most users who "go get" do not care about seeing or modifying the dependencies; they just want to build or edit the package they directly reference.
• This more closely matches the behavior of ~other languages' dependency tools, such as easy_install, cpan, gem, etc.

[ucirello]

Once you get disciplined about Go workspaces, it turns out that it becomes very predictable where you need to stop to detect the GOPATH - pretty much similar to the algorithm to detect the vendor directory.

I have been using this flawlessly for sometime:

$ which gopathme
gopathme: aliased to origDir="$PWD";lastDir=""; while [ "$lastDir" != "$PWD" ]; do if [ -d src ]; then export GOPATH="$PWD"; export PATH="$GOPATH/bin:$PATH"; echo $PWD; break; fi; lastDir="$PWD"; cd ../; done; cd $origDir

I have wrapped go with an bash script that actually tests for GOPATH before forwarding the shell call, and it works just as fine.

[mcluseau]

Coming from #14566, the smallest set of things I want when I have a vendor directory outside the GOPATH is:

1. have my vendor directory is managed like $GOPATH/src;
2. be able to specify the package name of my current directory.
   This way, I can have my CI do reproductible builds using, for instance:

go build --local-package example.com/my-tools -o dist/my-command  ./cmd/my-command

Optionally, if go get can also pull to my vendor directory when outside of my GOPATH, that would be awesome.

[broady]

I have an alternative, which is also compatible with the "src" detection above.

We currently have a mechanism for enforcing package import paths (import path checking).
It indicates the import path of a particular package. We can use this to synthesize a GOPATH, and to import subdirectories or nearby directories without relative imports.

Consider

directory $HOME/foo/bar
    package bar // import "github.com/broady/bar"
    import "github.com/broady/bar/baz"

We expect "baz" to be a subdirectory of $HOME/foo/bar.
We can contract github.com/broady/bar to . and treat it like import "./baz".

I'm working on a prototype and will write up a proposal in more detail. Evidently it has similar drawbacks to relative imports (nowhere to put object files), but has the advantage of retaining explicitness and not conflicting with how go get works.

(and while I was writing this, @MikaelCluseau replied. This proposal addresses his second point)

[mcluseau]

On 07/14/2016 10:13 AM, Chris Broadfoot wrote:

I'm working on a prototype and will write up a proposal in more
detail. Evidently it has similar drawbacks to relative imports
(nowhere to put object files), but has the advantage of retaining
explicitness and not conflicting with how |go get| works.

git is using a .git folder; I wouldn't find it awful if go did the
same. It could be $PWD/.go or $PWD/vendor/.go or even
$PWD/vendor/.pkg.

There are only two hard problems in distributed systems:

1. Exactly-once delivery
2. Guaranteed order of messages
3. Exactly-once delivery

[peebs]

I'm in favor of the original walk up until you find src proposal or walk up to the highest level src dir.

As someone who works with a unique GOPATH per project with each project vendoring its dependancies -- I really like the gb model of auto detection of the GOPATH based on which project path you are in. Its just like git and it best matches the workflow I see developers use day to day which is switching between git repos that are all fully self contained gopaths in essence (or a project in gb terms).

I commonly see people work around existing gopaths by including build.shscripts that set the gopath to avoid the developer being required to set their environment when moving between git repos. In this development model it is expected that if you want to develop the project, you clone the repo with git. If you are consuming the project you are importing it with a vendor tool to an existing repo. Without the repo including the build script then every time I move between git repos I would have to change my GOPATH env again.

With this proposal someone who normally uses something like git clone goes from doing:
git clone https://github.com/coreos/mantle
cd mantle
(now one can call ./build or other bash scripts to work on the repo and create a gopath)
To doing:
git clone https://github.com/coreos/mantle mantle/src/github.com/coreos/mantle
cd mantle/src
And now I am in the gopath and can execute any go commands requiring the path to be setup correctly. In the first example things like go list still don't work out of the box whereas with the proposal you have a properly setup go path and can make better use of go tooling.

Basically, for this work style, this proposal would eliminate a lot of bash glue in the example repo above as long as the new git command make sense to developers. Also, its not like this proposal stops people from using the old clone style and bash scripts.

I understand there are different working styles to consider here, I just wanted to chime in that for the work style I commonly see I am in favor of the original proposal.

[mcluseau]

On 07/14/2016 11:14 AM, Patrick Baxter wrote:

I understand there are different working styles to consider here, I
just wanted to chime in that for the work style I commonly see I am in
favor of the original proposal.

I don't think we are incompatible: you are actually getting a GOPATH,
while I was discussing about being outside the GOPATH.

If we take the nice git approach, go should look for a .go directory
in the CWD or any parent in the same filesystem. This directory would
mark what is currently "$GOPATH/src", while $GOPATH/pkg would be
.go/pkg. To support CI builds, we only need either a command line
argument, or a marker to (optionnaly) set package name of the CWD. This
marker could be in the .go directory.

There are only two hard problems in distributed systems:

1. Exactly-once delivery
2. Guaranteed order of messages
3. Exactly-once delivery

[peebs]

If the original proposal gets adopted it might also make sense to make any vendor dir auto set as the gopath rather then continuing to walk up the file tree to a src. That way you could cd into the vendor directory and have the normal go tools work on just the vendored code but also allow go get to work as vendoring tool.

The vendoring stuff would need to be figured out more before the go get stuff would be useful inside the vendor dir. However, making the auto-detect work in the meantime for vendor as well might still be desirable. It might even affect plans around standardizing the location of the manifest file and how many vendor dirs can exist per project and where they should be.

[rakyll]

I am in the favor of the original proposal. Walking up to the src seems to be a great strategy unless it doesn't break packages with import paths with src inside. Do you have any input what's going to be the expected behavior if go is run inside a directory such as:

$GOPATH/src/github.com/someone/proj/src/cmd/cli

[XANi]

@rakyll I think almost every proposal will break _something_ one way or another but walk up to source/walk up to .dotfile seems intuitive enough

[kokes]

Having all Github code in BigQuery will help identify potential problems like the one with src in import paths. Obviously Github != all Go code, but it's a solid start.

[abextm]

How about a new go command, say go new. It would create a new project and could serve as a introduction to new users. On first run it may look like:

~ $ go new
Enter a project name. This typically looks like: github.com/user/project
> user/entered/name
You do not seem to have a $GOPATH set. This is required for go to manage dependencies.
$GOPATH location [/home/username/go]:
> 

The tool would then create the $GOPATH structure, and add it to wherever you add $GOPATH on your os. It might also copy the default go playground example, or similar, to the new project directory and cd there. It could also, optionally, add $GOPATH/bin to your $PATH, or at least give guidance on how go install works.

I think this would help new users understand the gopath structure hands-on, without having to change other tools.

[adg]

@Abextm this is an interesting idea, but would you mind filing a separate issue for it? I don't want to distract from the central discussion here.

[mem]

Thinking more about it, I'm confused about what problem this proposal is trying to solve.

I do have a workspace-centered workflow, which is very much like what this proposal would encourage. I do _not_ set a default value for GOPATH. When I start working on a new project I create a directory for it and set GOPATH to that directory. Something akin to this:

$ export GOPATH=$PWD/project-foo
$ PATH=$GOPATH/bin:$PATH
$ cd $GOPATH
$ mkdir -p src/project-foo
$ vim src/project-foo/main.go
$ go install project-foo
$ project-foo

The point being, everything here works, from compilation to running. Notice that I have to do two things: set GOPATH and set PATH. Setting PATH is arguably pure comfort and not required, but as far as I understand this proposal, it's trying to address newcomers to Go and people who are not used to working with Unix-like operating systems. The second point that might generate disagreement is the usage of go install vs go build. If we are talking about newcomers, go build might be enough, and the above can be changed to this:

$ export GOPATH=$PWD/project-foo
$ cd $GOPATH
$ mkdir -p src/project-foo
$ vim src/project-foo/main.go
$ go build project-foo
$ ./project-foo

Now we are one step away from this proposal:

$ mkdir -p project-foo/src/project-foo
$ cd project-foo
$ vim src/project-foo/main.go
$ go build project-foo
$ ./project-foo

We have traded not having to tell newcomers about an environment variable for telling them to prefer go build over go install. Granted we would have gained the ability for doing this:

$ mkdir -p project-foo/src/project-foo
$ cd project-foo/src/project-foo
$ vim main.go
$ go build
$ ./project-foo

but note that for this particular case we do not need GOPATH. We need GOPATH only if we introduce packages:

$ mkdir -p project-foo/src/project-foo
$ cd project-foo/src/project-foo
$ mkdir pkg
$ vim pkg/pkg.go
# ...
$ vim main.go
# add something that requires import "project-foo/pkg"
$ go build
$ ./project-foo

This fails with the current state and would work with this proposal.

The downside I'm seeing is that since this proposal encourages using go build over go install, that also means that it encourages _not_ having a pkg directory with a cache of all the packages (directly or indirectly) imported by project-foo, meaning slower build times.

Also note in that last example PWD is .../project-foo/src/project-foo, meaning that an user cd'ing around (in my experience, this is typical of people who do not feel comfortable with the command line, which might or might not equate the newcomers we are talking about here), will always be able to run go build project-foo and will end up with project-foo binaries all over the place. go install doesn't do that, which is another reason to recommend it over go build. But that means going back to the PATH problem.

Is there an additional required modification? Something like go install -run ..., meaning that it does what go install ... does, and it also runs the program. I don't really like it, but if the issue is environment variables, that's one way to solve it. Obviously another way is to extend go run to accept packages... :-P

[rodmur]

I'm a very new user of Go, so I basically just fall into the camp of wanting to test out helloworld.go. But I must say, it feels rather like 199x if I have to set an environment variable from my shell just to run my compiler or install a package. I get packages, I like them, but I also happen to like having reasonable defaults for tools and having those things in places like /etc/profile.d/ which a lot of Linux distributions use to setup the system wide default environment for a user. Why doesn't Go just use /etc/profile.d/ and set stuff there? More advanced users can always override those values for whatever they want to do. I'm not sure remaking the wheel, in this situation, is entirely necessary.

[metakeule]

I think the easiest way to help newcomers and don't break the existing way is the following:

_Change go get in a way, that if GOPATH is not set, it will save the dependencies into a subfolder of the current folder named vendor (if this folder is not there: create it)._

This way it is very easy: You can just build your package in the current folder and it will just work.

(A second step could be to advocate to create subpackages inside vendor/sub or vendor/lib and have the vendor folder added to the repo. Although you might argue that this would confuse the terminology and purpose of the vendor folder.)

[rakyll]

Why doesn't Go just use /etc/profile.d/ and set stuff there?

We don't need to set the env variable and mess with the environment. All we have to ensure the entire tooling has a sane default if nothing is set. Easy, dependency-free, cross-platform solution if the goal is to make GOPATH optional.

[Merovius]

FWIW a (arguably pretty naive)

curl https://api.godoc.org/packages | jq '.results[].path' | grep '/src/' | wc -l

turns up 4184 import paths currently on godoc.org that would not work with this proposal and might lead to surprising results if GOPATH ever is accidentally not set (whereas now, if it is not set, there will just be a safe error). They include a lot of vendored stuff and projects that use a custom GOPATH (and might thus even, individually, benefit from this) but I'd still find it surprising if a go get suddenly deposits stuff deeply below my $GOPATH (just because I happened to accidentally run a shell without GOPATH set explicitly).

Just as a different variant: There are system-specific environment variables that tell you where to put data or configs and the most correct solution under a modern linux might be to use something like $XDG_DATA_DIR/go. Not that I'm claiming that this is a good idea.

Personally, I think all of the things proposed here can lead to unwanted side-effects, if $GOPATH is not set, be that because it wasn't set yet, or because of an accidental non-login-shell or something at some point in time. Thus I'm in favor of either not changing the current behavior or to default to a predictable directory (and of those I'd use $HOME, but I'm openly biased as that's my current setup).

Also, in response to people suggesting using a subdirectory of $GOROOT, or (like python) _prefix_/whatever: This will disallow using a distribution-provided go installation with go get, as $GOROOT there is most often not user-writable.

[mem]

@Merovius I get your point, but doing a random sampling of the packages considered in that count, after looking at 5 of them, none of them are go-get'able, and in the cases that I looked at, the src directory is there because that's where their "workspace" starts. Look at the import statements in the packages: instead of e.g. import "github.com/kpumuk/metricsd/src/metricsd/config" it says import config.

[Merovius]

@mem fair enough (I don't think that matters particularly, but who am I to argue): Searching for /src/ on the HTML index (which only includes go-get'able packages) gives 963 hits.
But really, go-get'able or not is besides the point, in my opinion. They are buildable. And the negative side-effects are mostly independent of whether or not they intend you to set GOPATH to a subdir of the repo.

Anyway. I just wanted to provide some actual quantitative perspective on this. And state my technical objections to this proposal (namely that it makes it somewhat unpredictable what'll happen if GOPATH in some session isn't set, accidentally). I just don't think this heuristic is very good, is all.

[LaPingvino]

@Merovius my proposal actually takes care of this, because it never automatically sets, instead it asks for the first time and saves for later. Roll back to my proposal to see how that works.

Here again straight from my memory:

1. have a tool that looks if either $GOPATH is set or a .gopath hidden file indicates the $GOPATH
2. if only .gopath is found, set $GOPATH globally to that
3. if not, ask where to put it with a sensible default
4. if an answer is given, put .gopath there so it will automatically work the next time, and set $GOPATH
5. this tool should be run by go get etc if no gopath is set, just a simple invocation to check

[davecheney]

Hello,

What is the status of this proposal?

If it's stalled, I'd like to make my own, simpler, proposal along the lines of https://github.com/golang/go/issues/12488#issuecomment-137596005.

[adg]

@davecheney I'm happy for you to make a counterproposal here, unless it is not about automatically detecting GOPATHs at all.

[davecheney]

TL;DR - I want to propose that GOPATH defaults to $HOME.

On Mon, Sep 26, 2016 at 9:35 AM, Andrew Gerrand [email protected]
wrote:

@davecheney https://github.com/davecheney I'm happy for you to make a
counterproposal here, unless it is not about automatically detecting
GOPATHs at all.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/12488#issuecomment-249453949, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA-3HUF6GkN3RRthEjd9jelKKlCcqks5qtwU-gaJpZM4F3oTt
.

[bradfitz]

@davecheney, not until we nuke the "pkg" directory. I'd like to add proper caching to cmd/go early in the Go 1.9 cycle, and then we could probably ditch pkg.

[davecheney]

@branfitz is that because I suggested GOPATH=$HOME, if I suggested an
alternative location like GOPATH=$HOME/gocode would that unblock this
proposal so we don't have to wait til /pkg is removed ?

On Mon, Sep 26, 2016 at 9:53 AM, Brad Fitzpatrick [email protected]
wrote:

@davecheney https://github.com/davecheney, not until we nuke the "pkg"
directory. I'd like to add proper caching to cmd/go early in the Go 1.9
cycle, and then we could probably ditch pkg.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/12488#issuecomment-249454812, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA7gJLMgZcip7HinQlVCSho31u-Qyks5qtwljgaJpZM4F3oTt
.

[mcluseau]

@davecheney GOPATH defaulting to home won't fix the "complex CI build" issue, but is nice. If pkg is removed at some point, all we'll need will be to have a flag to specify the position of the CWD in the go import tree.

[davecheney]

If you don't want GOPATH=$HOME then all you have to do is set $GOPATH
manually. For 100% of Go users today, nothing has changed.

On Mon, Sep 26, 2016 at 10:00 AM, Mikaël Cluseau [email protected]
wrote:

@davecheney https://github.com/davecheney GOPATH defaulting to home
won't fix the "complex CI build" issue, but is nice. If pkg is removed at
some point, all we'll need will be to have a flag to specify the position
of the CWD in the go import tree.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/12488#issuecomment-249455191, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA2AZGcpxM-trMf7TCv_ghstjS6lFks5qtwsvgaJpZM4F3oTt
.

[bradfitz]

@branfitz is that because I suggested GOPATH=$HOME, if I suggested an
alternative location like GOPATH=$HOME/gocode would that unblock this
proposal so we don't have to wait til /pkg is removed ?

I wouldn't object, at least. But I (and I think others) would object to Go automatically creating a $HOME/pkg dumping ground, surprising users.

[crawshaw]

@bradfitz is there a plan to do away with pkg? It does not seem easy to me, it is an important performance optimization, especially if you are importing cgo packages.

What if an unset GOPATH implied pkg being somewhere off in a temp file system?

[davecheney]

To be clear, I want GOPATH to be _a_ default, I don't care what the default
is because it can be overridden by setting GOPATH explicitly.

On Mon, Sep 26, 2016 at 10:04 AM, Brad Fitzpatrick <[email protected]

wrote:

@branfitz is that because I suggested GOPATH=$HOME, if I suggested an
alternative location like GOPATH=$HOME/gocode would that unblock this
proposal so we don't have to wait til /pkg is removed ?

I wouldn't object, at least. But I (and I think others) would object to Go
automatically creating a $HOME/pkg dumping ground, surprising users.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/12488#issuecomment-249455370, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA69DmumnKzwQvmL8_4Yhlb6zOevyks5qtwvxgaJpZM4F3oTt
.

[mcluseau]

On 09/26/2016 11:03 AM, Dave Cheney wrote:

If you don't want GOPATH=$HOME then all you have to do is set $GOPATH
manually. For 100% of Go users today, nothing has changed.

I agree, I just want to underline the path that put me in this thread,
which is "how to build easily in gitlab-ci", where your git repo is in
$HOME/build/some-variable-part/repo-name. We need to be able to say "hey
go, this folder where you are is the import path for mycomp.com/my-project".

[davecheney]

The former. The latter may have to set an GOPATH environment variable, this
is no different to what they have done for Go 1.1 to Go 1.7.

On Fri, Dec 11, 2015 at 10:30 AM, Dmitri Shuralyov <[email protected]

wrote:

Is this proposal about helping people who don't know/understand GOPATH
workspaces be able to use Go? Or is it about making it more convenient for
people who know it but deal with many systems, CI, etc.?

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/12488#issuecomment-163785836.

[davecheney]

@MikaelCluseau in that case nothing changes, the CI user will have to set GOPATH, just as they do today.

[davecheney]

$HOME/build/some-variable-part/repo-name. We need to be able to say "hey
go, this folder where you are is the import path for mycomp.com/my-project".

I'd say this is out of scope. Various CI tools like travis have worked around the fact that the checkout of the go project is not the root of the workspace. My proposal to set a default GOPATH does not change this.

[mcluseau]

On 09/26/2016 11:10 AM, Dave Cheney wrote:

@MikaelCluseau https://github.com/MikaelCluseau in that case nothing
changes, the CI user will have to set GOPATH, just as they do today.

yes, that's why I said it doesn't solve the problem. But I understand it
can be seen as a different issue.

[bradfitz]

@bradfitz is there a plan to do away with pkg? It does not seem easy to me, it is an important performance optimization, especially if you are importing cgo packages.

Yes. That is #4719. It will still exist, but instead of being at $GOPATH/pkg, it'll be at a OS- and user-specific cache directory, and it'll be a cache of outputs from actions of content-addressable inputs. Which means go test -i as a necessary practice goes away, etc.

[davecheney]

This proposal only changes the experience for newcomers who have not chosen to set a GOPATH variable.

At the moment all the documentation says "you have to set a GOPATH", then people get distracted and upset because they don't understand why they have to do this.

By choosing a default GOPATH, we can make our documentatation easier because we can say things like

$ go get github.com/foo/bar

will check out the github.com/foo/bar repo into $HOME/gocode/src/github.com/foo/bar.

We don't need to distract newcomers with having to set an env var, all we need to do is put a little parenthetical note at the bottom of the page

$HOME/gocode is the default path to your go workspace. If you want to change this path, set the GOPATH variable to something of your choosing.

[LaPingvino]

I foresee problems here though when you e.g. start using a new shell and in the old situation the error message would inform you that it's not set up correctly yet. With your proposal you would silently get the code checked out and duplicated, and you end up with a mess, if you don't use the same GOPATH. in my proposal that doesn't happen, because any new shell will automatically pick up the same old directory, it works when you migrate to a new system and will gently put the new users through it.

[LaPingvino]

Also, if you add this stuff to an independent tool, you can backport it to old versions where necessary.

[davecheney]

@rsc as the proposal arbitrator would you welcome a proposal to provide a default GOPATH if one is not explicitly set in the environment?

[adg]

I think we're going with #17262.