Go: cmd/go: relative imports do not work with vendoring

[titpetric]

1. What version of Go are you using (go version)?

go version go1.6.1 linux/amd64

1. What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

Linux serenity 4.3.0-1-amd64 #1 SMP Debian 4.3.5-1 (2016-02-06) x86_64 GNU/Linux
Running golang with Docker image, official golang build, latest image from docker hub.
Full reproduction script available with ./run

Source files: https://cdn.si/assets/go-gh-issue-vendoring.tgz

1. What did you do?

I created a small program, using a local subpackage.
main.go (using a subpackage + vendor) = experienced problem.
main2.go (using vendored import from main) = works as expected.

Subpackage is imported with: import "./apiservice". Import of vendored package from this subpackage results in a run/build error.

Also included is a test-runner using the official golang docker image. It's the _same_ environment, that I'm running the example in. The folder vendor was created with gvt fetch github.com/namsral/flag, full source provided.

1. What did you expect to see?

Expected output with main.go:

# go run main2.go
Hello world! Network port: 8080

1. What did you see instead?

# go run main.go
apiservice/apiservice.go:4:8: cannot find package "_/go/src/app/vendor/github.com/namsral/flag" in any of:
        /usr/local/go/src/_/go/src/app/vendor/github.com/namsral/flag (from $GOROOT)
        /go/src/_/go/src/app/vendor/github.com/namsral/flag (from $GOPATH)

Problem:

Import path for local subpackage is not honored. The dot gets converted to underscore, and as a consequence, it's looking for the vendor directory in the wrong path:

Expected path:

        /usr/local/go/src/app/vendor/github.com/namsral/flag (from $GOROOT)

[davecheney]

Your need to put your source in GOPATH. From the example you posted (sorry
I haven't run your script) you have not done this.

[titpetric]

@davecheney I've been putting it in GOROOT, I updated now to use GOPATH, same issue persists.

#!/bin/bash
printf "Expected:\n\n"
docker run --rm -it -v `pwd`:/go/src/app -w /go/src/app golang go run main2.go
printf "\nActual:\n\n"
docker run --rm -it -v `pwd`:/go/src/app -w /go/src/app golang go run main.go

# ./run_gopath
Expected:

Hello world! Network port: 8080

Actual:

apiservice/apiservice.go:4:8: cannot find package "_/go/src/app/vendor/github.com/namsral/flag" in any of:
        /usr/local/go/src/_/go/src/app/vendor/github.com/namsral/flag (from $GOROOT)
        /go/src/_/go/src/app/vendor/github.com/namsral/flag (from $GOPATH)

So, sorry, no. I also tried putting everything in /go instead of /go/src/app but that breaks both examples.

[davecheney]

I belive GOPATH is not set in your enviroment correctly. I believe this
because of the leading _/ in the error report. This is the path that the go
tool uses when source is _outside_ GOPATH or GOPATH is not set correctly.
If GOPATH is not set correctly, vendoring will not be used.

Is it possible to isolate your issue by removing docker?

[titpetric]

I don't have an environment outside of docker where I can run the tests. If I get one, I'll post the results, and if you have one, please just stick the sources from the tgz into $GOPATH/src/app and run go run main.go to see if the error occurs with you.

But, my sanity check here tells me that you shouldn't be so suspicious, go env reports /go (same as $GOPATH in bash), sources don't reference some /other or ../other locations. I might be quite new at this, but I'm pretty sure that nowhere in about 10-20 lines of code I'm going _outside_ of GOPATH, and I'm not touching it in any way. The obvious question is "what's the correct way to set up GOPATH?" if the official docker image doesn't do this. Source Dockerfile is here, if you can spot the issue: https://github.com/docker-library/golang/blob/master/1.6/Dockerfile

I'll attach my results if I can get my hands on on a local install of go somewhere.

[davecheney]

You shouldn't need docker to compile your tests -- they may not pass, but
you'll at least confirm that you can compile your code.

[davecheney]

I just checked out your sample. Sorry I should have done that before.

The root cause is you are using go run, which does not support vendoring.

I recommend you lay out your code in the prescribed package format as
documented here https://golang.org/doc/code.html

Then compile and run your program.

[titpetric]

I'm having the same result with go build, but I'll check the layout extensively. I followed the recommended layout from glide - in the main README, which should be this:

$GOPATH(/go)/src/your-project
  |__ main.go (Where your main code lives)
  |__ subpackage/
  |     |__ subpackage.go (You can create subpackages)
  |     \__ ...
  |__ ... (Other files or subpackages)
  \__ vendor/
        |__ github.com/namsral/flag
        |__ github.com/gorilla/mux
        \__ ...

So if I understand you, instead of using "subpackage" I should create src/subpackage and use import "subpackage" as a replacement? I'm just about to go AFK into a meeting, and I'll try this asap. But it really seems that I shouldn't. Importing subpackages which reside in the same level is done with "./package", which is a documented thing somewhere. It actually _works_... at least some packages use a slightly different approach to structure the sources in the same way, with the double-slash import option:

package main // import "github.com/janeczku/go-dnsmasq"
import "github.com/janeczku/go-dnsmasq/hostsfile"

Is there a way to achieve something similar for local development without a recognized VCS location? I believed until now that: import "./package" was it.

[davecheney]

Is there a way to achieve something similar for local development without
a recognized VCS location? I believed until now that: import "./package"
was it.

Yes, packages are just directories on disk. They follow the url based
convention to avoid namespace collision and to enable go get. You should
layout your code using the url based pattern, but there is no requirement
(unless you want someone else to go get your code) to push it to a remove
vcs location

[kostya-sh]

I think that currently local (relative) imports do not work with vendoring. I am not sure if it is even possible to make these features work together. Local imports do not work in some other cases as well (see for example http://stackoverflow.com/questions/10687627/relative-import-from-parent-directory).

[titpetric]

@davecheney It is just slightly awkward if you're using go get to mix remote packages from github, etc, with whatever you have locally. main.go and package/package.go with import "./package" would seem like a valid usage pattern.

@kostya-sh That's unfortunate. I guess I'd mark this issue as a feature request then, I would very much like local imports to work with vendoring.

[titpetric]

If anybody is coming here for a fix, this worked for me:

1. My app name is 'app' (/go/src/app),
2. instead of importing ./submodule, import app/submodule
3. Vendoring starts to work, everything works with go run.

I will not close this issue, just because I expect that this should be fixed in a future version. I hope someone agrees and tags/assigns the issue properly. Relative imports should work with vendoring, but I understand the motivations for FQDN imports. Considering that i'm using them from the main package, which is a reasonable "root" of the project, and if I'll ever reuse it from another package i would use the FQDN import from a VCS/other.

In the current state I would have to rewrite './' to $(basename $(dirname $0)) before doing go build/go run, which would require some external tooling like gb. Don't want external tools for something that simple.

[bcmills]

It looks like there were several different issues at play in this thread.

@titpetric, could you summarize the remaining problem that you want to be fixed, ideally as a list of commands that we can run in isolation to observe the problem? (A .tgz file is not a suitable format for a bug report, sorry.)

I suspect that this is another instance of #14566 / #18007, but since I don't really understand the problem statement it's hard to be sure.

[titpetric]

There's only one issue, vendoring doesn't play well with relative imports.

/main.go
~~~go
package main

import (
"fmt"
"./errors"
)

func main() {
fmt.Println(errors.Error404())
}
~~~

/errors/errors.go
~~~go
package errors

import (
e "github.com/pkg/errors"
)

func Error404() error {
return e.New("Not found")
}
~~~

Expected result: stdout Not found

Actual result:
~
errors/errors.go:4:2: cannot find package "_/go/src/app/vendor/github.com/pkg/errors" in any of:
/usr/local/go/src/_/go/src/app/vendor/github.com/pkg/errors (from $GOROOT)
/go/src/_/go/src/app/vendor/github.com/pkg/errors (from $GOPATH)
~

I verified the above with Go 1.6-1.11:

~
docker run --rm -it -v pwd:/go/src/app -w /go/src/app golang:1.6-alpine go run main.go
docker run --rm -it -v pwd:/go/src/app -w /go/src/app golang:1.7-alpine go run main.go
docker run --rm -it -v pwd:/go/src/app -w /go/src/app golang:1.8-alpine go run main.go
docker run --rm -it -v pwd:/go/src/app -w /go/src/app golang:1.9-alpine go run main.go
docker run --rm -it -v pwd:/go/src/app -w /go/src/app golang:1.10-alpine go run main.go
docker run --rm -it -v pwd:/go/src/app -w /go/src/app golang:1.11-alpine go run main.go
~

[bcmills]

Neither of the paths listed in that comment includes a vendor component. Is the whole thing in the vendor directory? If so, where are you running the go command?

Please provide a _list of specific commands that we can run_ to reproduce the problem, starting from an empty GOPATH. The details are important.

[titpetric]

I apologize, I believed I was clear that the vendor/ folder should come from go dep, it seems to have gotten lost in a comment edit (I was too much in a rush).

The github.com/pkg/errors repo should be vendored. You can use go get to retrieve it and then just copy it into the vendor/ folder to provide vendor/github.com/pkg/errors/*.

[titpetric]

Here you have it:

~
git clone [email protected]:titpetric/go-issues-15478.git
cd go-issues-15378
./test.sh
~

[bcmills]

Ok, this is definitely #18007.

When you use a relative path from within main.go by using go run with a filename, the imported package does not have an ordinary GOPATH-based import path; instead, it has a synthetic path derived from the relative file path. vendor support uses the import path to resolve the locations of vendor directories, so it cannot find the directory using the synthetic path.