Go: cmd/go: `go get` fails when repository ends with `.go`

[wallyqs]

What version of Go are you using (go version)?

$ go version
go version go1.12 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env


GOARCH="amd64"

GOBIN=""

GOCACHE="/Users/himitsu/Library/Caches/go-build"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GOOS="darwin"

GOPATH="/Users/himitsu/repos/lkjh-dev"

GOPROXY=""

GORACE=""

GOROOT="/usr/local/Cellar/go/1.12/libexec"

GOTMPDIR=""

GOTOOLDIR="/usr/local/Cellar/go/1.12/libexec/pkg/tool/darwin_amd64"

GCCGO="gccgo"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD=""

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/xl/vzfvbyrs29s3t2mmh15ygmv00000gn/T/go-build506667951=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

When trying to fetch the latest version or a tag of the NATS Go client as follows, it fails with a no such file or directory error.

GO111MODULE=on go get github.com/nats-io/nats.go@latest
stat github.com/nats-io/nats.go: no such file or directory

GO111MODULE=on go get github.com/nats-io/[email protected]
stat github.com/nats-io/nats.go: no such file or directory

One workaround for this when using Go 1.12 is to append an extra slash at the end of the repository name and then add the explicit version as follows:

GO111MODULE=on go get github.com/nats-io/nats.go/@latest
...

# Explicit tag version
GO111MODULE=on go get github.com/nats-io/nats.go/@v1.8.0
go: finding github.com/nats-io/nats.go v1.8.0
go: downloading github.com/nats-io/nats.go v1.8.0
go: extracting github.com/nats-io/nats.go v1.8.0

What did you expect to see?

I expected go get to download the library as follows:

GO111MODULE=on go get github.com/nats-io/[email protected]
go: finding github.com/nats-io/nats.go v1.8.0
go: downloading github.com/nats-io/nats.go v1.8.0
go: extracting github.com/nats-io/nats.go v1.8.0

Or when not specifying a version either, expected for it to download the latest one:

GO111MODULE=on go get github.com/nats-io/nats.go
go: finding github.com/nats-io/nats.go v1.8.1
go: downloading github.com/nats-io/nats.go v1.8.1
go: extracting github.com/nats-io/nats.go v1.8.1

What did you see instead?

GO111MODULE=on go get github.com/nats-io/[email protected]
stat github.com/nats-io/nats.go: no such file or directory

Also when not specifying any version always fail, regardless of trailing slash or not:

GO111MODULE=on go get github.com/nats-io/nats.go/
stat github.com/nats-io/nats.go: no such file or directory

GO111MODULE=on go get github.com/nats-io/nats.go
stat github.com/nats-io/nats.go: no such file or directory

[bcmills]

Paths ending in .go usually refer to source files rather than package or module paths; presumably that is the problem here.

CC @jayconrod

[gopherbot]

Change https://golang.org/cl/180923 mentions this issue: cmd/goget: support package ended with '.go'

[letientai299]

@dmitshur, I'm still quite new to golang project workflow, so, thanks for the reminder, and please guide me through this second CL.

My fix in above CL is basically trying load patterns ended with .go as go files, and if it fails, fall back to loading the patterns as packages.

[dmitshur]

Thanks @letientai299. You should discuss the approach you've considered with @bcmills and @jayconrod here, since they're owners of cmd/go behavior, and see what they think.

trying load patterns ended with .go as go files, and if it fails, fall back to loading the patterns as packages.

I have a potential concern about it, in that it introduces unpredictable behavior. The same command invocation may act differently depending on whether or not a .go file exists and can be accessed without errors. I'm not sure of a better solution, I just wanted to point out the potential concern.

[bcmills]

Note that the same command also fails in GOPATH mode, at least as far back as 1.11.10:

$ GO111MODULE=off go1.12.5 get github.com/nats-io/nats.go
stat github.com/nats-io/nats.go: no such file or directory

$ GO111MODULE=off go1.11.10 get github.com/nats-io/nats.go
stat github.com/nats-io/nats.go: no such file or directory

Given the lack of other reports of this issue, perhaps nats-go should just pick a less ambiguous name for their module‽

(Or are there substantially more examples of this in the wild?)

[wallyqs]

@bcmills yes you also have to add trailing slash if not using Go modules. From a similar past issue that is open it seems that this may have been introduced around Go 1.11, still ok in Go 1.10: https://github.com/golang/go/issues/27096

[bcmills]

@wallyqs, note that the concrete example of such a package in #27096 is also nats-io/nats.go.

[wallyqs]

@bcmills I mean that the issue mentions others running into a similar issue, though we also commented on that issue. Btw thanks @letientai299 for looking at the possible fix!

[letientai299]

@dmitshur

I have a potential concern about it, in that it introduces unpredictable behavior. The same command invocation may act differently depending on whether or not a .go file exists and can be accessed without errors. I'm not sure of a better solution, I just wanted to point out the potential concern.

Previous implementation also have that issue. The code check if there's any pattern ends with .go and treat all of them as go files.

[wallyqs]

It looks like this behavior has worsened in Go 1.13, now it is not possible at all to go get the nats module repository:

# --- Go 1.13
GO111MODULE=on go get github.com/nats-io/nats.go/@v1.8.0
go: finding github.com v1.8.0
go: finding github.com/nats-io v1.8.0
go get github.com/nats-io/nats.go/@v1.8.0: malformed module path "github.com/nats-io/nats.go/": trailing slash

GO111MODULE=on go get github.com/nats-io/nats.go/@latest
go get github.com/nats-io/nats.go/@latest: malformed module path "github.com/nats-io/nats.go/": trailing slash

GO111MODULE=on /usr/local/go/bin/go get github.com/nats-io/nats.go
stat github.com/nats-io/nats.go: no such file or directory

GO111MODULE=on /usr/local/go/bin/go get github.com/nats-io/nats.go/
stat github.com/nats-io/nats.go: no such file or directory

GO111MODULE=on go get github.com/nats-io/nats.go@latest
stat github.com/nats-io/nats.go: no such file or directory

GO111MODULE=on go get github.com/nats-io/[email protected]
go: finding github.com v1.8.0
go: finding github.com/nats-io v1.8.0
stat github.com/nats-io/nats.go: no such file or directory

[derekcollison]

@bcmills Please keep us posted on this. Seems pretty bad behavior that has totally broke the NATS.io ecosystem. NATS has been with Go since 0.52 IIRC.

/cc @bradfitz @Sajmani

[wallyqs]

Only variation that seems to work is when disabling go modules and adding a trailing slash to go get:

GO111MODULE=off go get github.com/nats-io/nats.go
stat github.com/nats-io/nats.go: no such file or directory

GO111MODULE=off go get github.com/nats-io/nats.go/
# No `trailing slash` error

[gopherbot]

Change https://golang.org/cl/194600 mentions this issue: cmd/go: strip trailing slash from versioned arguments

[jayconrod]

CL 194600 should fix the regression in 1.13. I'd like to backport that for 1.13.1 if possible.

go get operates in two phases: it updates go.mod, then runs go build on the arguments (kind of). Both phases are having problems.

When we do the work needed to update go.mod, we weren't stripping trailing slashes from arguments with versions. So go get -d github.com/nats-io/nats.go/@latest would fail when we went to validate the proxy base URL, since trailing slashes aren't allowed in a module path. It works without the trailing slash though (go get -d github.com/nats-io/nats.go@latest succeeds).

If we do strip the trailing slashes, then the second go build phase fails because it treats arguments ending with .go as files, unlike the rest of go get. So CL 194600 will strip trailing slashes but preserve them for this phase.

[bcmills]

Note that the nats.go maintainers in particular knew that this repo name was a problem literally four days after renaming the repository and didn't roll back at that point.
(Compare the dates on https://github.com/golang/go/issues/27096#issuecomment-492380792 and https://github.com/nats-io/nats.go/commit/4c154eb68ac6b3c6d9532b1756095cbb6c85f026.)

I think I'm ok with a minimal change to restore compatibility with commands that happened to work in 1.12, but I can't justify sinking a lot of time into ongoing support for these names that never really worked to begin with.

[derekcollison]

@bcmills For ecosystems like NATS that have many client implementations it has become pretty common to name the client repo .. This is not just a NATS thing.

This is a silly problem that the Go team should resolve.

[derekcollison]

For instance - https://github.com/miscreant

[jayconrod]

@gopherbot please backport to 1.13, this was a regression

[gopherbot]

Backport issue(s) opened: #34243 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[derekcollison]

Thanks, much appreciated.

[wallyqs]

Thanks @jayconrod @bcmills just tried it building Go from master and it works great 🙌
Looking forward to the 1.13.1 release!

[halega]

1.13.1 is out, but the problem still exists:

$ go version
go version go1.13.1 windows/amd64

$ go get github.com/nats-io/nats.go/@latest
go get github.com/nats-io/nats.go/@latest: malformed module path "github.com/nats-io/nats.go/": trailing slash

$ go get github.com/nats-io/nats.go@latest
CreateFile github.com/nats-io/nats.go: The system cannot find the path specified.

How to add NATS client to a project?

[dmitshur]

@halega, Go 1.13.1 was a security release. Such releases contain only the relevant security fix and no other changes, in order to make them safer to update to. All other changes will be a part of the next minor release, Go 1.13.2, which will happen soon (beginning of October).

[gopherbot]

Change https://golang.org/cl/198459 mentions this issue: cmd/go: fix listing of ambiguous paths

[gopherbot]

Change https://golang.org/cl/198957 mentions this issue: cmd/go: fix listing of ambiguous paths

[derekcollison]

Was expecting this to be in 1.13.2 but it was not. Any estimations or timeframe to have this released?

[bcmills]

1.13.2 ended up being a security release, so everything not security-related got bumped to 1.13.3.

[derekcollison]

We just figured out that the fix was not in there. Do you folks have an eta?

[agnivade]

Should be out soon enough. Feel free to subscribe to https://github.com/golang/go/issues/34970.

[derekcollison]

Thanks!