Go: x/vgo: "use of internal package not allowed" unclear

[EwanValentine]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.10.1

Does this issue reproduce with the latest release?

Quite likely

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN="/Users/valene02/go/bin"
GOCACHE="/Users/valene02/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/valene02/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/p2/tx56hr5x6k73086v_zfnnnt8rbbmmt/T/go-build295382073=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Ran $ vgo build

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

For vgo to install dependencies and generate a valid go.mod file.

What did you see instead?

vgo got most of the way through the install process before finally throwing the following error:

../../../v/github.com/gomodule/[email protected]/redis/pool.go:28:2: use of internal package not allowed

I'm using redigo as a top level package, but I'm not directly importing the file that seems to be throwing the error pool.go.

[odeke-em]

Hello there @EwanValentine, thank you for filling this bug!

For starters, could you please post perhaps a way of others reproducing this bug? I ask because currently your reproduction is vgo build but not a repo, nor way to reproduce and diagnose the bug itself, nor even give it a succinct title name

The internal package not allowed error comes appears after Go1.5, but it shouldn't be an error unless vgo misdiagnoses a path as not existing in its parent's root as per
https://github.com/golang/go/blob/b7f1777a709e575cb6aa03c25a976d06d36cb6a6/src/cmd/go/internal/load/pkg.go#L734-L742

If you'd like to get your hands dirty, perhaps a fmt.Printf right above those lines e.g
an edit to $GOPATH/src/golang.org/x/vgo/vendor/cmd/go/internal/load/pkg.go

@@ -726,6 +726,9 @@
        return p
    }

+   fmt.Printf("\nsrcDir:  %q cleaned: %q\nparent: %q cleaned: %q\n\n",
+       srcDir, filepath.Clean(srcDir), parent, filepath.Clean(parent))
+
    // Internal is present, and srcDir is outside parent's tree. Not allowed.
    perr := *p
    perr.Error = &PackageError{

and then rerun go get -u golang.org/x/vgo and then try to reproduce your error.

Thank you.

[cznic]

@odeke-em

I tried to follow your instructions, hope it can help.

jnml@r550:~/src/golang.org/x/vgo> git status 
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   vendor/cmd/go/internal/load/pkg.go

no changes added to commit (use "git add" and/or "git commit -a")
jnml@r550:~/src/golang.org/x/vgo> git --no-pager diff
diff --git a/vendor/cmd/go/internal/load/pkg.go b/vendor/cmd/go/internal/load/pkg.go
index acd9187..541303d 100644
--- a/vendor/cmd/go/internal/load/pkg.go
+++ b/vendor/cmd/go/internal/load/pkg.go
@@ -726,6 +726,9 @@ func disallowInternal(srcDir string, p *Package, stk *ImportStack) *Package {
        return p
    }

+   fmt.Printf("\nsrcDir:  %q cleaned: %q\nparent: %q cleaned: %q\n\n",
+       srcDir, filepath.Clean(srcDir), parent, filepath.Clean(parent))
+
    // Internal is present, and srcDir is outside parent's tree. Not allowed.
    perr := *p
    perr.Error = &PackageError{
jnml@r550:~/src/golang.org/x/vgo> git --no-pager log -1
commit b85f7250588babd76bcf9001abf8128e629b1c2a (HEAD -> master, origin/master, origin/HEAD)
Author: Russ Cox <[email protected]>
Date:   Thu May 24 16:44:30 2018 -0400

    cmd/go/internal/modfetch: add newlines to new errors

    Change-Id: Id3dafa2381692e3665d6442e558cad7832379ac8
    Reviewed-on: https://go-review.googlesource.com/114497
    Run-TryBot: Russ Cox <[email protected]>
    Reviewed-by: Bryan C. Mills <[email protected]>
    TryBot-Result: Gobot Gobot <[email protected]>
jnml@r550:~/src/golang.org/x/vgo> go install
jnml@r550:~/src/golang.org/x/vgo> cd cznic/ql
/home/jnml/src/github.com/cznic/ql
jnml@r550:~/src/github.com/cznic/ql> git status 
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean
jnml@r550:~/src/github.com/cznic/ql> git --no-pager log -1
commit eccf3e9caa43fab70d7ddfdee5c3fb73fc76c2a1 (HEAD -> master, origin/master, origin/HEAD)
Author: Jan Mercl <[email protected]>
Date:   Wed Feb 28 10:13:17 2018 +0100

    Check defaults/constraints limits. Fixes #195.

            modified:   all_test.go
            modified:   storage.go
jnml@r550:~/src/github.com/cznic/ql> vgo build |& tee log
vgo: creating new go.mod: module github.com/cznic/ql
vgo: resolving import "github.com/cznic/b"
vgo: finding github.com/cznic/b (latest)
vgo: adding github.com/cznic/b v0.0.0-20180115125044-35e9bbe41f07
vgo: resolving import "github.com/cznic/golex/lex"
vgo: finding github.com/cznic/golex (latest)
vgo: adding github.com/cznic/golex v0.0.0-20170803123110-4ab7c5e190e4
vgo: resolving import "github.com/cznic/lldb"
vgo: finding github.com/cznic/lldb (latest)
vgo: adding github.com/cznic/lldb v1.1.0
vgo: resolving import "github.com/cznic/mathutil"
vgo: finding github.com/cznic/mathutil (latest)
vgo: adding github.com/cznic/mathutil v0.0.0-20180504122225-ca4c9f2c1369
vgo: resolving import "github.com/cznic/strutil"
vgo: finding github.com/cznic/strutil (latest)
vgo: adding github.com/cznic/strutil v0.0.0-20171016134553-529a34b1c186
vgo: resolving import "github.com/cznic/fileutil"
vgo: finding github.com/cznic/fileutil (latest)
vgo: adding github.com/cznic/fileutil v0.0.0-20180108211300-6a051e75936f
vgo: resolving import "github.com/cznic/internal/buffer"
vgo: finding github.com/cznic/internal (latest)
vgo: adding github.com/cznic/internal v0.0.0-20170905175358-4747030f7cf2
vgo: resolving import "github.com/cznic/sortutil"
vgo: finding github.com/cznic/sortutil (latest)
vgo: adding github.com/cznic/sortutil v0.0.0-20150617083342-4c7342852e65
vgo: resolving import "github.com/cznic/zappy"
vgo: finding github.com/cznic/zappy (latest)
vgo: adding github.com/cznic/zappy v0.0.0-20160723133515-2533cb5b45cc
vgo: resolving import "github.com/edsrzf/mmap-go"
vgo: finding github.com/edsrzf/mmap-go (latest)
vgo: adding github.com/edsrzf/mmap-go v0.0.0-20170320065105-0bce6a688712

srcDir:  "/home/jnml/src/v/github.com/cznic/[email protected]/buffer" cleaned: "/home/jnml/src/v/github.com/cznic/[email protected]/buffer"
parent: "/home/jnml/src/v/github.com/cznic/[email protected]" cleaned: "/home/jnml/src/v/github.com/cznic/[email protected]"


srcDir:  "/home/jnml/src/v/github.com/cznic/[email protected]" cleaned: "/home/jnml/src/v/github.com/cznic/[email protected]"
parent: "/home/jnml/src/v/github.com/cznic/[email protected]" cleaned: "/home/jnml/src/v/github.com/cznic/[email protected]"


srcDir:  "/home/jnml/src/v/github.com/cznic/[email protected]/file" cleaned: "/home/jnml/src/v/github.com/cznic/[email protected]/file"
parent: "/home/jnml/src/v/github.com/cznic/[email protected]" cleaned: "/home/jnml/src/v/github.com/cznic/[email protected]"


srcDir:  "/home/jnml/src/v/github.com/cznic/[email protected]" cleaned: "/home/jnml/src/v/github.com/cznic/[email protected]"
parent: "/home/jnml/src/v/github.com/cznic/[email protected]" cleaned: "/home/jnml/src/v/github.com/cznic/[email protected]"


srcDir:  "/home/jnml/src/v/github.com/cznic/[email protected]" cleaned: "/home/jnml/src/v/github.com/cznic/[email protected]"
parent: "/home/jnml/src/v/github.com/cznic/[email protected]" cleaned: "/home/jnml/src/v/github.com/cznic/[email protected]"

../../../v/github.com/cznic/[email protected]/btree.go:16:2: use of internal package not allowed
../../../v/github.com/cznic/[email protected]/memfiler.go:13:2: use of internal package not allowed
../../../v/github.com/cznic/[email protected]/buffer/buffer.go:36:2: use of internal package not allowed
jnml@r550:~/src/github.com/cznic/ql> 

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[cznic]

Speaking up, please reopen. Required info was submited 5 days ago.

[cespare]

Bad @gopherbot.

[rsc]

@EwanValentine's original problem is that vgo decided to use github.com/gomodule/redigo version v1.6.0, the latest v1 tag in that repo, but that tag is broken code. It contains imports assuming that the code is github.com/garyburd/redigo not gomodule. The working code is tagged v2.0.0 but the v2.0.0 tag is ignored because this is redigo not redigo/v2. Even so,

vgo get github.com/gomodule/[email protected]

will do the right thing (replacing v2.0.0 with the corresponding pseudo-version) and will make the build start working.

I will also look into making this error message print the name of the internal package being imported.

@cznic, your problem is a bit different. You are expecting internal to work across module boundaries, which I'm not sure we ever intended. That's tracked by #23970.

[gopherbot]

Change https://golang.org/cl/116761 mentions this issue: cmd/go: mention internal package path in 'use of internal package not allowed' error