dep init fails if in not in $GOPATH[...]/src/{somedir..}

[pbennett]

If GOPATH is foo/bar/ then running dep init from /foo/bar/src
yields:
determineProjectRoot: /foo/bar/src not in any $GOPATH
yet it most certainly is in the GOPATH.

I'll leave the commentary off about why I want this, but if this is going to be disallowed for now, a better error is needed.

[Mahito]

I could got same issue.

% echo $GOPATH
/Users/mahito/.go
% cd $GOPATH/src/sandbox
% dep init  
% ls
lock.json      manifest.json
% cd ..
% dep init
determineProjectRoot: /Users/mahito/.go/src not in any $GOPATH

This issue is happened every time in $GOPATH/src.
https://github.com/golang/dep/blob/master/context.go#L115 create srcprefix($GOPATH/src/) .
Next line compare path ($GOPATH/src) and srcprefix($GOPATH/src/) and get false, because len(path) is less length than len(srcprefix).

[machiel]

Yes, I've got the same issue, for private projects we work with $GOPATH as the project git root. Trying to do a dep init in $GOPATH yields:

determineProjectRoot: $GOPATH not in any $GOPATH

Though I was told this isn't the intended way of using the tool though, so it might be out of scope.

[sdboyer]

I think this approach to things should work fine, though because it's not a design case we've had in mind, there's likely to be at least a few hinky issues related to the base project root being empty. (Yeah...I can think of a few now.) The only important note to make is that the resulting repository will be useless for anyone trying to import it...but I imagine that's not a concern for anyone taking this approach.

To be clear - this approach means that dep will ensure there is exactly one version of all dependencies in your system, and all of them will be managed as a group. In general, if you leave all your third-party deps under vendor/, and keep all the stuff you actually work on (your monorepo-ish) directly in GOPATH not under vendor/, then this oughtta be OK.

At minimum, we need a better error here. Ideally, we'd support this as a first-class use case.

[tro3]

Yeah, this looks related to #201. I'll add an integration test for the cases described above then clean up the code.

[tro3]

Actually, this is not the same issue as #201 after all. This is more of a defintion question. The dependency import path is calculated as everything past $GOPATH/src, becuase that is where Go will look for the imports anyway. So if you start a project in $GOPATH/src, it can't figure out the import path. (I'm a little curious what $GOPATH/src/src would do.) I am too new to the subtleties of the Go import model to be able to make a call here, but it does sound to me like starting a repo directly in $GOPATH goes against the opinion of the tool, which assumes a $GOPATH/src, $GOPATH/pkg, $GOPATH/bin?

[machiel]

Just to clarify, I don't put my sourcecode directly in GOPATH.

My tree for most of my projects look somewhat like this: (Or vendor directly in GOPATH, if I use gb)

Hope this clear some things up :)

$GOPATH
├── bin
├── pkg
├── resources
│   ├── config
│   ├── templates
│   └── uploads
└── src
    ├── web
    ├── mail
    ├── notifier
    ├── server
    └── vendor

[sdboyer]

@tro3 yeah, this issue describes an alternate use pattern to what's typical; @Machiel's description is a good description of one variant of it. It can work within the general model of dep, sliding through on a technicality (actually, more like a door-left-carefully-ajar).

However, fixing the handling to make the "GOPATH/src as root" case work is more involved, more or less because of what you note - there is no import path to figure out. Even once we solve it in dep, gps is going to need a bit of special handling for empty root import paths to support this case.

...and, actually, some other things, too. Ugh. I need to open an issue on gps for it.

[tro3]

Oh, I see. Actually, I kind of like that use case, though presumably $GOPATH is then set per-project. (Still a newbie, here.)

So if it works as-is for $GOPATH (thru the door left ajar :-)) and as intended for $GOPATH/src/x, maybe just a better error message for dep init run in $GOPATH/src is needed? In any case, #201 is a different beast. If you want an explicit error called for this case, let me know and I'll add.

[typeless]

Looking forward to the progress on this. We have a proprietary project that has many small utilities built into a busybox-like executable and has some vendored packages from the stdlib and third-parties for those utilities. Currently the best solution is GB but it seems abandoned. We also have to integrate the project into some BSPs, including AOSP and other buildroot-based ones, which makes the process a little messy. Hopefully a canonical tool like dep will ease our job soon.

[sekrett]

@sdboyer so your design is to force everyone to put their project under src folder? I don't understand why I should group my repositories by language, instead I should be able to group by projects. For example I have a project, which contains one service written in Ruby and another in Golang. In general I would like to decide myself how to name and organize my folders.

[andrewrk]

why does dep depend on GOPATH? isn't the whole point to ignore GOPATH and use vendor as the place for dependencies, so projects can be independent?

[sdboyer]

@andrewrk sadly, dep can't liberate us from GOPATH on its own, as the compiler still requires it. and, being that dep's goal is to have as easy a transition into the toolchain as possible, we're focusing minimally on those areas that complement/extend the existing go toolchain, rather than supplanting existing systems.

given that we know we have to accept GOPATH's existence for now, we do rely on it for figuring out the root import path of the current project. this is important because we use it to determine which import paths are "internal" vs "external" when inspecting the code from the current project. A fair bit more on those mechanics here: https://github.com/golang/dep/pull/313#issuecomment-285918383

when we're ready to move away from GOPATH (it might look something like this), we'll probably add a field to Gopkg.toml for declaring the project root so that we no longer need infer it from filesystem position + GOPATH variable.

[andrewrk]

Ah. Thanks for this explanation. I was getting ridiculed in #go-nuts for the same question but your answer is clear and helpful.

[sekrett]

Another bad situation is when I run go dep from Gitlab CI. I have to create src directory and symlink the project to it. It works but the hack is ugly. Golang should not dictate path conventions.

[DarthHater]

Thanks for the explanation @sdboyer , I am trying to run a dep project in Jenkins and this was driving me insane. Leaving a comment here so I can know when/if this becomes possible.

[zhexuany]

An interesting observation. When I try to run dep init at a symlink but it failed and give me error message root project import: xxxx is not within any GOPATH/src. The interesting part is that current directory is actually a symlink to a project under GOPATH. Any solution for this one?

[tombh]

I'm really confused by all this. So just to be explicit: if you're developing a project and you want to use dep then that project needs to live on your computer under $GOPATH/src? If that's the case, where's the documentation for that.

So my current fix is to symlink my project into $GOPATH/src and have a separate CLI open at $GOPATH/src/myproject to run the dep commands in.

[sdboyer]

@tombh it's not in the docs, as the current docs are paltry, and GOPATH is one of those foundational realities that's just implied.

i realize that's a terrible answer in general, though. so, ive made it explicit in the docs overhaul - #1499.

[frostyplanet]

We hate to put our project in GOPATH, as the project itself might contain submodule for other project, or contain code written by other language.

[sdboyer]

@frostyplanet we know the reasons why it's important to do, and agree with them. it doesn't change the reality that it's not a choice we can make in dep right now.

[frostyplanet]

@sdboyer
There is some way to build a project which is not in gopath. We are using these practices for local develop and it's easy to pack up everything inside working dir to build RPM packages on dedicated build host.
All we want is a tool that download dependences into "vendor". "vendor" may or may not be managed by version control.

1) For simple code which does not have complex structure, *.go is right under working dir, like any other golang projects,
create a build directory (which was set as GOPATH) , sync the code into it and build,
Using this Makefile:
compile: internal_dep FORCE
@mkdir -p build/src/
@rsync -a --delete *.go vendor build/src/mypackage/
env GOPATH="$$(pwd)/build" go build -o build/$(PROG) $(PROJECT)
@echo "Build done"

2) Using this directory tree
repo
| -- src (code written in go)
| -- | -- mypackage1
| -- | -- mypackage2 (might be git submodule following practice 1)
| -- scripts
| -- xxx ( code not written in go )
| -- thirdparty
| -- | -- src
| -- | -- | -- thirdparty package

Set GOPATH with
export GOPATH="$(pwd):$(pwd)/thirdparty"
and then using "go get xxx", will download them into "thirdparty/src".

"go get" works fine with public code. For private code we have written some script to download them into "thirdparty" and deal with the branch/tag/commit thing, but still our script is not good at resolving dependency.

[andrewrk]

@sdboyer

sadly, dep can't liberate us from GOPATH on its own, as the compiler still requires it. and, being that dep's goal is to have as easy a transition into the toolchain as possible, we're focusing minimally on those areas that complement/extend the existing go toolchain, rather than supplanting existing systems.

Looks like Russ Cox has decided to fix this issue: https://research.swtch.com/vgo-intro

The most significant change, though, is the end of GOPATH as a required place to work on Go code. Because the go.mod file includes the full module path and also defines the version of every dependency in use, a directory with a go.mod file marks the root of a directory tree that serves as a self-contained work space, separate from any other such directories. Now you just git clone, cd, and start writing. Anywhere. No GOPATH required.

Great! It took a long time, but looks like we'll get a happy ending after all.

[sam3d]

Now you just git clone, cd, and start writing. Anywhere. No GOPATH required.

Yay! God that's such exciting news.

You can find a mirror for vgo (a prototype implementation of the spec outlined in @andrewrk's comment) at golang/vgo.

[sdboyer]

yep, addressing this was never really a question. just needed to get to the point where the changes could happen in the toolchain.

On February 21, 2018 1:29:02 PM EST, Andrew Kelley [e-mail] wrote:

@sdboyer

sadly, dep can't liberate us from GOPATH on its own, as the compiler
still requires it. and, being that dep's goal is to have as easy a
transition into the toolchain as possible, we're focusing minimally on
those areas that complement/extend the existing go toolchain, rather
than supplanting existing systems.

Looks like Russ Cox has decided to fix this issue:
https://research.swtch.com/vgo-intro

The most significant change, though, is the end of GOPATH as a
required place to work on Go code. Because the go.mod file includes the
full module path and also defines the version of every dependency in
use, a directory with a go.mod file marks the root of a directory tree
that serves as a self-contained work space, separate from any other
such directories. Now you just git clone, cd, and start writing.
Anywhere. No GOPATH required.

Great! It took a long time, but looks like we'll get a happy ending
after all.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/golang/dep/issues/148#issuecomment-367423733

[EddieOne]

So what is the fix?

[frayhan32]

In my case how i can solve this is, First i run the export for my GOPATH. Lets assume my GoPath would be. _/Users/your-username/Documents/mataharimall-development/www/go/_. After that make sure you have directory src under your gopath and your project directory under src folder

[codeAndxv]

you should use this command "dep init" in $GOPATH/src/your projectname

[sam3d]

It's annoying that we can't do it without using $GOPATH yet, but it's on the way!

[luisdavim]

I have a project that I can build from any path in my workstation and I can fetch all the dependencies with:

go get -t -d -v ./...

why does dep ensure fail in that scenario?

[notzippy]

Not sure if this is the same issue or not - but it is the same error.

If the GOPATH is based on symlinks the dep tool fails reporting that the realpath does not match any GOPATH. eg

$ export GOPATH=~/home/symlinkedpath/gopath
$ realpath $GOPATH
$ /mnt/dev/gopath
$ cd ~/home/symlinkedpath/gopath/src/test && dep ensure
/mnt/dev/gopath is not within a known GOPATH/src

My quick fix for this is running the dep tool by proxy and creating a new GOPATH based on the filepath.EvalSymlinks for each existing part in the GOPATH

    realPath := &bytes.Buffer{}
    for _, p := range filepath.SplitList(build.Default.GOPATH) {
        rp,_ := filepath.EvalSymlinks(p)
        if realPath.Len() > 0 {
            realPath.WriteString(string(filepath.ListSeparator))
        }
        realPath.WriteString(rp)
    }
        // Call dep using realPath for GOPATH