Go: x/tools/gopls: error when using vendoring with 1.14

Created on 3 Mar 2020 · 16Comments · Source: golang/go

What did you do?

I was using gopls as usual with VS Code.

My environment uses vendoring (implicitly) with the new 1.14 automatic vendor detection.

What did you expect to see?

No errors.

What did you see instead?

An error pop up:

Your workspace is misconfigured: go [-m -json all]: exit status 1: go list -m: can't compute 'all' using the vendor directory (Use -mod=mod or -mod=readonly to bypass.)

It seems some command used by gopls is causing issues when -mod=vendor is set implicitly by the go tool.

Build info

golang.org/x/tools/gopls v0.3.3
    golang.org/x/tools/[email protected] h1:mTFqRDJQmpSsgDDWvbtGnSva1z9uX2XcDszSWa6DhBQ=
    github.com/BurntSushi/[email protected] h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/sergi/[email protected] h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
    golang.org/x/[email protected] h1:WG0RUwxtNT4qqaXX3DPA8zHFNm/D9xaBpxzHt1WcA/E=
    golang.org/x/[email protected] h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
    golang.org/x/[email protected] h1:OX66ZzpltgCOuBSGdaeT77hS2z3ub2AB+EuGxvGRBLE=
    golang.org/x/[email protected] h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
    honnef.co/go/[email protected] h1:sXmLre5bzIR6ypkjXCDI3jHPssRhc8KD/Ome589sc3U=
    mvdan.cc/xurls/[email protected] h1:KaMb5GLhlcSX+e+qhbRJODnUUBvlw01jt4yrjFIHAuA=

Go info

go version go1.14 linux/amd64

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/<redacted>/.cache/go-build"
GOENV="/home/<redacted>/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY="<redacted>"
GONOSUMDB="<redacted>"
GOOS="linux"
GOPATH="/home/<redacted>/go"
GOPRIVATE="<redacted>"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build121565687=/tmp/go-build -gno-record-gcc-switches"

Tools gopls

Source

johanbrandhorst

👍2

All 16 comments

Thanks for reporting! Can you share the full output of gopls -rpc.trace -v check path/to/file.go?

stamblerre on 3 Mar 2020

I see two suspect call sites:

https://github.com/golang/tools/blob/abb57c682a60b603ade8571659cde791b971ccf0/internal/lsp/cache/mod.go#L239
https://github.com/golang/tools/blob/cf670267be10c1b9044a9299196b3c9f2b9db125/go/packages/golist_overlay.go#L285

CC @matloob

bcmills on 3 Mar 2020

Thanks for reporting! Can you share the full output of gopls -rpc.trace -v check path/to/file.go?

I'm not able to reproduce it reliably - I'm not quite sure what triggers it, but I will try to get a trace.

johanbrandhorst on 3 Mar 2020

Based on the -json and the fact that the misconfiguration error comes up after we try to load the workspace, I would guess that the issue is the go list call in go/packages.

@bcmills: How should go/packages work around this - should we set the -mod=mod flag?

stamblerre on 3 Mar 2020

@stamblerre, as far as I can tell that use in go/packages is only used in the implementation of determineRootDirs, and determineRootDirs itself is only used in getPkgPath.

So I would recommend raising the abstraction level up a bit: go list accepts a directory argument, so (barring any remaining bugs) you can invoke go list $DIR to resolve the package path corresponding to filesystem directory $DIR:

~/src/golang.org/x/tools$ go1.13.8 list -f '{{.Dir}}' golang.org/x/xerrors
/usr/local/google/home/bcmills/pkg/mod/golang.org/x/[email protected]

~/src/golang.org/x/tools$ go1.13.8 list /usr/local/google/home/bcmills/pkg/mod/golang.org/x/[email protected]
golang.org/x/xerrors

bcmills on 3 Mar 2020

Ah, wait, that doesn't work because getPkgPath is trying to infer the hypothetical package path for a directory that does not (yet) actually contain a Go package.

bcmills on 3 Mar 2020

Thinking about this some more: is there a use-case for overlays outside of the main module? The module cache is generally read-only anyway, so folks shouldn't be adding files there.

I guess you might want to handle overlays for modules substituted in via replace directives..? But that won't work in vendor mode anyway, because the go command won't read the replacement directories: they really don't have a package path.

So that leaves:

Directories in the main module's vendor tree. You already know where the main module is, so you just need to detect that $DIR is within a subdirectory of its vendor tree, lop off the prefix, done. (No root search required.)
Directories in the main module. go list -m -f '{{.Dir}}' within $DIR to verify that it's actually still in the main module, lop off the prefix containing the main module directory, filepath.ToSlash, and done.
Directories in modules for which the main module contains a replace directive. But we currently require a go.mod file for replaced modules, and the replacement module's go.mod file is supposed to match the path that it replaces. So you can go list -m -f {{.Path}} {{.Dir}} within $DIR, then go list -m {{with .Replace}}{{.Dir}}{{end}} that module path from within the main module (to verify that that module really is the replacement for the module), and again lop off the directory prefix and filepath.ToSlash as above.

bcmills on 3 Mar 2020

Thanks for the detailed analysis, @bcmills! I'll leave the decision-making up to @matloob, since this is not an area of go/packages that I'm particularly familiar with, but your suggestions sound good to me.

stamblerre on 4 Mar 2020

I'm trying to understand what's going on here. Some questions:

@bcmills, you're asking if there's a use-case for overlays outside the main module because you're trying to figure out if we can get rid of the go list -m call that determines module roots for determining import paths, right? If so, the suggestion is to determine which module each directory is by doing a go list -m call for each of them, right? If so, I'm wondering if the go list -m -f '{{.Dir}} invocation for #2 will work if the directory doesn't exist.

My second question is why doesn't go list -m all work with -mod=vendor?

matloob on 4 Mar 2020

you're trying to figure out if we can get rid of the go list -m call that determines module roots for determining import paths, right?

Yep.

If so, the suggestion is to determine which module each directory is by doing a go list -m call for each of them, right?

Correct, with the exception of subdirectories of the main module's vendor directory.

If so, I'm wondering if the go list -m -f '{{.Dir}} invocation for #2 will work if the directory doesn't exist.

It will not, but it should be ok in general to run go list -m -f '{{.Dir}}' on the deepest parent of the proposed directory that actually does exist.

bcmills on 4 Mar 2020

My second question is why doesn't go list -m all work with -mod=vendor?

vendor/modules.txt only includes two sets of modules:

modules that provide packages transitively _imported by_ the main module
(as of Go 1.14) modules named explicitly in the main module's go.mod file.

In contrast, at least up through Go 1.14, go list -m all lists _all_ modules transitively _required by_ the main module's go.mod file. In general, that can be an arbitrarily larger superset of the two sets listed in vendor/modules.txt.

That means that the output of go list -m all would vary depending on the -mod flag, and since the default value for the -mod flag changes automatically in Go 1.14, I felt that could be very confusing. So instead, it emits an error, and the user can make an explicit choice to either set -mod=mod explicitly or find a different query that works more consistently with -mod=vendor.

bcmills on 4 Mar 2020

Thanks for the explanation. I'll start by implementing #2. If we need support for directories in the vendor tree or those that are replaced, we can implement #1 and #3 if requested by users. (My intention is to keep the implementation of overlays as simple and restricted as possible.

matloob on 4 Mar 2020

Is there any temporary workaround for this issue?