Go: cmd/go: unclear error when server does not provide 'go-import' metadata for a module in a subdirectory

[graywolf]

What version of Go are you using (go version)?

$ go version
go version go1.13.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

GO111MODULE="on"

GOARCH="amd64"

GOBIN=""

GOCACHE="/home/wolf/.cache/go-build"

GOENV="/home/wolf/.config/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GONOPROXY=""

GONOSUMDB=""

GOOS="linux"

GOPATH="/home/wolf/go"

GOPRIVATE=""

GOPROXY="direct"

GOROOT="/usr/lib/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"

GCCGO="gccgo"

AR="ar"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD="/home/wolf/tmp/golang_test/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build652827916=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Created go module:

+$ tree .
.
└── bar
    ├── bar.go
    └── go.mod

1 directory, 2 files
+$ find . -name .git -prune -o -type f -print -exec cat {} \;
./bar/bar.go
package bar

func Bar() string {
        return "bar"
}
./bar/go.mod
module git.sr.ht/~graywolf/foo/bar

go 1.13

pushed it to the repository (https://git.sr.ht/~graywolf/foo) and tried to use it in following program:

package main

import (
        "fmt"
        "git.sr.ht/~graywolf/foo/bar"
)

func main() {
        fmt.Println(bar.Bar())
}

What did you expect to see?

+$ go run sr.go
bar

What did you see instead?

+$ go run sr.go
go: finding git.sr.ht/~graywolf/foo latest
build command-line-arguments: cannot load git.sr.ht/~graywolf/foo/bar: module git.sr.ht/~graywolf/foo@latest (v0.0.0-20191019153505-33a4721605aa) found, but does not contain package git.sr.ht/~graywolf/foo/bar

[bcmills]

Thanks for the clear steps to reproduce the problem.

The root cause seems to be that your HTTPS server is not serving a <meta name="go-import" […]> tag for the path git.sr.ht/~graywolf/foo/bar — it is only serving a tag for the /foo prefix:

example.com$ curl -sL https://git.sr.ht/~graywolf/foo | grep -A1 go-import
<meta name="go-import"
  content="git.sr.ht/~graywolf/foo git https://git.sr.ht/~graywolf/foo">

example.com$ curl -sL https://git.sr.ht/~graywolf/foo/bar | grep -A1 go-import

example.com$

That causes the go command to believe that there is no such module for that path.
Probably the best we could do in the go command is emit a better diagnostic for this case, but it's not obvious to me how we could detect it in general.

Or, perhaps we can improve the documentation for the remote import path protocol.

What documentation were you using when setting up the server?

CC @jayconrod @thepudds

[graywolf]

But github does not provide that either

+$ curl -sSf https://github.com/graywolf/foo | grep -A1 go-import
  <meta name="go-import" content="github.com/graywolf/foo git https://github.com/graywolf/foo.git">

+$ curl -sSf https://github.com/graywolf/foo/bar | grep -A1 go-import
curl: (22) The requested URL returned error: 404 Not Found

and over there it works just fine. As far as I can tell only difference is the .git in the repository path on github

+$ curl -sSf https://github.com/graywolf/foo | grep -A1 go-import
  <meta name="go-import" content="github.com/graywolf/foo git https://github.com/graywolf/foo.git">

+$ curl -sSf https://git.sr.ht/~graywolf/foo | grep -A1 go-import
<meta name="go-import"
  content="git.sr.ht/~graywolf/foo git https://git.sr.ht/~graywolf/foo">

Actually, when I try to import it as foo.git in "kinda" works even from the source hut

+$ cat sr2.go
package main

import (
        "fmt"
        "git.sr.ht/~graywolf/foo.git/bar"
)

func main() {
        fmt.Println(bar.Bar())
}
+$ go run sr2.go
go: finding git.sr.ht/~graywolf/foo.git latest
go: finding git.sr.ht/~graywolf/foo.git/bar latest
go: git.sr.ht/~graywolf/foo.git/bar: git.sr.ht/~graywolf/foo.git/[email protected]: parsing go.mod:
        module declares its path as: git.sr.ht/~graywolf/foo/bar
                but was required as: git.sr.ht/~graywolf/foo.git/bar

What documentation were you using when setting up the server?

I do not know, it's a hosted service not setup by me.

/cc @ddevault

[ddevault]

See this issue:

https://github.com/golang/go/issues/32260

The fix has landed for the next Go release.

[ddevault]

Err, no, that was cgo-specific. This may be a git.sr.ht bug.

[bcmills]

But github does not provide that either

GitHub is (unfortunately) a hard-coded special case at the moment, so that's probably why it works there. (But that means that GitHub is not a good example to follow when constructing your own go-import tags.)

See also #26134.

https://github.com/golang/go/blob/0e3e46f0d741745b6ddd78c05e137d7dd77ace92/src/cmd/go/internal/get/vcs.go#L1026-L1033

[graywolf]

It's really shame that biggest open source hosting is special cased so it cannot be used for comparing with other implementations. Are there any plans to de-hard-code them?

[ddevault]

It's also a shame that Golang has to be special cased for every repository hosting platform. I'd really rather have a more general solution than adding Go-specific meta tags to each repo.

[bcmills]

@ddevault, per https://golang.org/cmd/go/#hdr-Remote_import_paths, you can always use a .git path suffix in lieu of the go-import tag. (The go-import tag is only needed to tell the go command where the repo root is within the path and which VCS to use, and the suffix communicates both of those.)

[bcmills]

@graywolf we have no plan to remove the hard-coded paths at this time. Presumably as a first step we would have to get upstream fixes for them to implement the full go-import protocol. 😅

[leafrhythms]

Not about the error message or go get mechanisms, but sr.ht fixed its implementation of remote import protocol on Dec 19.
https://git.sr.ht/~sircmpwn/git.sr.ht/commit/2bdb70d928f1e41e1d4fa1e15e24246f0c37873a

[ddevault]

"Fixed". I still think it's really silly that a git repo has to have language-specific meta tags. There really ought to be a more universal standard.

[dimpase]

@ddevault complains he got blocked on this repo. Is it some kind of ban done by a bot?