Go: proxy.golang.org: returns not found for @v/list of a module deleted from the origin

[cuonglm]

What version of Go are you using (go version)?

$ go version

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

What did you do?

$ curl https://proxy.golang.org/github.com/belogik/goes/@v/list
not found: github.com/belogik/goes@latest: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/0c31a160b38443d5e33ca2a5e60e51734d83293b66dc3d9b0c12699f8d2b5cec: exit status 128:
    fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

What did you expect to see?

List of available versions.

What did you see instead?

Error.

The reason seems due to github.com/belogik/goes was recently removed by the author. If I query for specific version, proxy still returns the result.

$ curl https://proxy.golang.org/github.com/belogik/goes/@v/v0.0.0-20151229125003-e54d722c3aff.info
{"Version":"v0.0.0-20151229125003-e54d722c3aff","Time":"2015-12-29T12:50:03Z"}

[bcmills]

This proxy behavior makes sense to me, but it takes a bit of unpacking.

If the repo was removed entirely, then it would probably be more helpful to serve the list of known versions, but serving an error won't break any existing users — existing users all have some explicit version already. However, if the repo was converted from public to private, the only “correct” behavior is to serve a 404 or 410 so that the go command will fall back to direct access (possibly with private, local credentials) and get the up-to-date version list.

Given the privacy properties of hosting services, the proxy cannot, in general, distinguish between removing a repo and making it private.

CC @katiehockman @heschik @hyangah @jayconrod @matloob

[bcmills]

On the other hand, perhaps we _could_ add something to the proxy protocol that would allow it to indicate “this list is incomplete; you should check the next proxy regardless of its contents”. That would allow the proxy to serve a non-empty list without inhibiting fallback for private repos.

It would take some care (perhaps a new endpoint?) to ensure that that behavior doesn't break older versions of the go command, though.

[cuonglm]

This proxy behavior makes sense to me, but it takes a bit of unpacking.

If the repo was removed entirely, then it would probably be more helpful to serve the list of known versions, but serving an error won't break any existing users — existing users all have some explicit version already. However, if the repo was converted from public to private, the only “correct” behavior is to serve a 404 or 410 so that the go command will fall back to direct access (possibly with private, local credentials) and get the up-to-date version list.

Given the privacy properties of hosting services, the proxy cannot, in general, distinguish between removing a repo and making it private.

CC @katiehockman @heschik @hyangah @jayconrod @matloob

@bcmills I think in any case, an error should only be returned if proxy does not know anything about the module. Here at least one version v0.0.0-20151229125003-e54d722c3aff presents.

The use case for it is if I want to check a module exists in proxy or not.

[jayconrod]

IMO, if an upstream repository is not available, proxy.golang.org should serve a list of versions it has cached. It should not serve pseudo-versions in /@v/list, and I think the go command will ignore them anyway.

On the other hand, perhaps we could add something to the proxy protocol that would allow it to indicate “this list is incomplete; you should check the next proxy regardless of its contents”. That would allow the proxy to serve a non-empty list without inhibiting fallback for private repos.

Not sure a protocol change is needed for this. Proxies that have access to private modules should go first in the GOPROXY list. That avoids the need for additional fallback behavior, but it's also important for privacy: queries for private modules won't go to proxy.golang.org first. GONOPROXY / GOPRIVATE are also useful here.

[hyangah]

Proxy.golang.org returns known non pseudo versions if there are cached ones available for use. In this specific case, I think there are no non-pseudo versions.

[jayconrod]

Proxy.golang.org returns known non pseudo versions if there are cached ones available for use. In this specific case, I think there are no non-pseudo versions.

Hmm, should /@v/list return a 200 with an empty list then? It looks like /@latest for this module returns 200 with a pseudo-version.

The go command will request /@v/list first, and if that fails with any error (including 404, 410), we'll fall back to the next proxy. We only request /@latest if the request for /@v/list is successful but contains no suitable versions.

[katiehockman]

Hmm, should /@v/list return a 200 with an empty list then? It looks like /@latest for this module returns 200 with a pseudo-version.

Are you suggesting /@v/list return this empty list only in the case where the database lookup yields cached pseudo-versions exclusively? This is something we could do if necessary.

This is to force the go command to fallback to /@latest for this empty list result, rather than fall back to the next proxy, correct? Is that actually the ideal behavior in every case?

[bcmills]

Not sure a protocol change is needed for this. Proxies that have access to private modules should go first in the GOPROXY list.

Maybe? But in many cases there is no explicit proxy with access to private modules, and arguably it should be reasonable for users who don't mind leaking their paths to the proxy operator to use the default GOPROXY settings without bothering to set GOPRIVATE.

[bcmills]

Hmm, should /@v/list return a 200 with an empty list then? It looks like /@latest for this module returns 200 with a pseudo-version.

This is to force the go command to fallback to /@latest for this empty list result, rather than fall back to the next proxy, correct? Is that actually the ideal behavior in every case?

Hmm, those are good points. At the very least, /@v/list and /@latest should probably agree with each other: if the list of versions is authoritative, then knowledge of the latest version is also authoritative, and vice-versa.

[hyangah]

Hmm, should /@v/list return a 200 with an empty list then? It looks like /@latest for this module returns 200 with a pseudo-version.

Are you suggesting /@v/list return this empty list only in the case where the database lookup yields cached pseudo-versions exclusively? This is something we could do if necessary.

This is to force the go command to fallback to /@latest for this empty list result, rather than fall back to the next proxy, correct? Is that actually the ideal behavior in every case?

One caveat is that if the user will never learn about a fresh version if they use the default GOPROXY setting without GOPRIVATE setting. On the other hand, that's already true if a repo had some released versions before turning private.

[heschik]

After internal discussion, we think this behavior of proxy.golang.org is, if not an outright bug, undesirable. The module author is free to set GOPRIVATE and do whatever they need to do, but downstream users can't add a dependency on the module without knowing a valid pseudoversion somehow.

$ go get -x github.com/belogik/goes@latest
# get https://proxy.golang.org/github.com/@v/list
# get https://proxy.golang.org/github.com/belogik/@v/list
# get https://proxy.golang.org/github.com/belogik/goes/@v/list
# get https://proxy.golang.org/github.com/belogik/goes/@v/list: 410 Gone (0.165s)
# get https://proxy.golang.org/github.com/@v/list: 410 Gone (0.269s)
# get https://proxy.golang.org/github.com/belogik/@v/list: 410 Gone (0.270s)
# get https://github.com/?go-get=1
mkdir -p /.../pkg/mod/cache/vcs # git3 https://github.com/belogik/goes
# lock /.../pkg/mod/cache/vcs/0c31a160b38443d5e33ca2a5e60e51734d83293b66dc3d9b0c12699f8d2b5cec.lock# /.../pkg/mod/cache/vcs/0c31a160b38443d5e33ca2a5e60e51734d83293b66dc3d9b0c12699f8d2b5cec for git3 https://github.com/belogik/goes
cd /.../pkg/mod/cache/vcs/0c31a160b38443d5e33ca2a5e60e51734d83293b66dc3d9b0c12699f8d2b5cec; git ls-remote -q origin
# get https://github.com/?go-get=1: 200 OK (0.035s)
0.075s # cd /.../pkg/mod/cache/vcs/0c31a160b38443d5e33ca2a5e60e51734d83293b66dc3d9b0c12699f8d2b5cec; git ls-remote -q origin
# get https://github.com/belogik/goes
# get https://github.com/belogik/goes: 404 Not Found (0.097s)
go get github.com/belogik/goes@latest: module github.com/belogik/goes: git ls-remote -q origin in /.../pkg/mod/cache/vcs/0c31a160b38443d5e33ca2a5e60e51734d83293b66dc3d9b0c12699f8d2b5cec: exit status 128:
    fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

This isn't the user experience we want proxy.golang.org to give. I'll work on changing it to successfully return an empty list so that the go command will fall back to @latest.

[heschik]

The fix ended up being somewhat more complicated than I thought. We'll probably have it rolled out toward the end of next week.

[heschik]

Done.

[arschles]

in https://github.com/golang/go/issues/37079#issuecomment-582907606, @bcmills suggested a way to tell the go tool that:

"this list is incomplete; you should check the next proxy regardless of its contents"

In #1532 we're discussing a similar problem in the context of Athens deployments. We're also dealing "offline" deployments that have no access to the internet. This new response would help in the offline case too.

It would be best for our purposes if the go tool was able to check the next proxies and do a best effort merge of all the versions it gets, without failing the entire operation if any of the other proxies or direct fail (don't return success or this new response)

[marwan-at-work]

I'd love to double check the following assumptions as Athens should probably operate similarly to proxy.golang.org to avoid implementation confusion:

1. If the proxy ever returned a none 404/410, go build should fail immediately.
2. If the proxy returned a 404/410 in @v/list, then it will not go to @latest, it will then go to the next proxy.
3. If the proxy returned an empty list in @v/list then it will go to @latest and if that 404'd then it will choose the next proxy in a comma separated GOPROXY.
4. proxy.golang.org's /list endpoint will return a 410 if go list -m fails and there is nothing cached
5. proxy.golang.org's /list will return an empty list if go list -m fails but only pseudo versions are cached.
6. proxy.golang.org's list will return a potentially partial list of go list -m fails but there are some proper semver versions in the cache.
7. proxy.golang.org's list will return a merge of both go list -m and the cache if they both succeed.
8. proxy.golang.org will return one psuedo semver if cache is empty and go list -m succeeds.

The above is my assumption on how the go command and the public proxy works, but please let me know if any of it is inaccurate. Thanks!

[heschik]

The above looks about right to me. In general we will tolerate failures for a little while, so where you say "fails" it really means "fails for long enough that we give up." 8 is a little confusing: @latest is specified to only return one thing.

If you have more questions let's find somewhere else to go through them than this semi-arbitrary bug :)