Go: cmd/go: get fails on gitlab subgroups due to go-import meta tags referring to nonexistent repos

Created on 5 Sep 2019  ·  74Comments  ·  Source: golang/go

What version of Go are you using (go version)?

$ go version
go version go1.13 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output


$ go env

GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/Users/umputun/Library/Caches/go-build"

GOENV="/Users/umputun/Library/Application Support/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GONOPROXY=""

GONOSUMDB=""

GOOS="darwin"

GOPATH="/Users/umputun/go-home"

GOPRIVATE=""

GOPROXY="direct"

GOROOT="/usr/local/Cellar/go/1.13/libexec"

GOSUMDB="off"

GOTMPDIR=""

GOTOOLDIR="/usr/local/Cellar/go/1.13/libexec/pkg/tool/darwin_amd64"

GCCGO="gccgo"

AR="ar"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD="/Users/umputun/tmp/t/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/fx/rzs1_n8137qfktxcbt_2v8pc0000gp/T/go-build570104617=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

  1. Created a public project on gitlab https://gitlab.com/umputuntests/sub/example with a module gitlab.com/umputuntests/sub/example
  2. Tried to get it, i.e. go get -v gitlab.com/umputuntests/sub/example
  3. go get failed
get "gitlab.com/umputuntests/sub/example": found meta tag get.metaImport{Prefix:"gitlab.com/umputuntests/sub/example", VCS:"git", RepoRoot:"https://gitlab.com/umputuntests/sub/example.git"} at //gitlab.com/umputuntests/sub/example?go-get=1
get "gitlab.com/umputuntests/sub": found meta tag get.metaImport{Prefix:"gitlab.com/umputuntests/sub", VCS:"git", RepoRoot:"https://gitlab.com/umputuntests/sub.git"} at //gitlab.com/umputuntests/sub?go-get=1
go: finding gitlab.com/umputuntests/sub/example v0.0.2
go: downloading gitlab.com/umputuntests/sub/example v0.0.2
go: extracting gitlab.com/umputuntests/sub/example v0.0.2
go get gitlab.com/umputuntests/sub/example: git ls-remote -q https://gitlab.com/umputuntests/sub.git in /Users/umputun/go-home/pkg/mod/cache/vcs/410d993df4daac0579ff6b4402c511af12bd3d00851d05ab6183293d03664961: exit status 128:
    fatal: could not read Username for 'https://gitlab.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

What did you expect to see?

with prev versions of go (1.12.x) it works file:

 go get -v gitlab.com/umputuntests/sub/example
Fetching https://gitlab.com/umputuntests/sub/example?go-get=1
Parsing meta tags from https://gitlab.com/umputuntests/sub/example?go-get=1 (status code 200)
get "gitlab.com/umputuntests/sub/example": found meta tag get.metaImport{Prefix:"gitlab.com/umputuntests/sub/example", VCS:"git", RepoRoot:"https://gitlab.com/umputuntests/sub/example.git"} at https://gitlab.com/umputuntests/sub/example?go-get=1
go: finding gitlab.com/umputuntests/sub/example v0.0.2
go: finding github.com/stretchr/testify v1.3.0
go: finding github.com/stretchr/objx v0.1.0
go: finding github.com/davecgh/go-spew v1.1.0
go: finding github.com/pmezard/go-difflib v1.0.0
go: downloading gitlab.com/umputuntests/sub/example v0.0.2
go: extracting gitlab.com/umputuntests/sub/example v0.0.2
gitlab.com/umputuntests/sub/example/strategy
gitlab.com/umputuntests/sub/example

additional notes

I think this is both gitlab problem and go 1.13 change. Gitlab seems to return go-import tag on the parent, which doesn't exist and this seems to confuse "go get", trying to check module's parent.

curl "https://gitlab.com/umputuntests/sub?go-get=1"

<html><head><meta name="go-import" content="gitlab.com/umputuntests/sub git https://gitlab.com/umputuntests/sub.git" /></head></html>

In fact, gitlab returns good-looking go-import on any random path, for example:

curl "https://gitlab.com/random/xyz/something?go-get=1"
<html><head><meta name="go-import" content="gitlab.com/random/xyz git https://gitlab.com/random/xyz.git" /></head></html>
NeedsInvestigation Testing modules
Source
picture umputun
👍20

Most helpful comment

@xavivars, this has already been fixed at head and backported to the 1.13 branch. It will be included in Go 1.13.3.

picture bcmills on 12 Oct 2019
🎉6

All 74 comments

Same problem here...

magodo picture magodo on 5 Sep 2019

Same problem here

kinwyb picture kinwyb on 5 Sep 2019

Go 1.13 tries all module prefixes in parallel, and if another error other than "not-found" happens, the query fails. Of course the root cause is on gitlab side, but I think it's reasonable to ignore all errors if at least one attempt succeeds.
A quick workaround is clearing err if found is not empty in queryPrefixModules() in cmd/go/internal/modload/query.go.

leonshaw picture leonshaw on 5 Sep 2019

@bcmills @jayconrod

EDIT: As a reminder, please note our https://github.com/golang/go/wiki/noplusone policy. Comments like "Me too" do not add anything new to the discussion.

agnivade picture agnivade on 5 Sep 2019

I think it's reasonable to ignore all errors if at least one attempt succeeds.

I could see doing that if the provided Git URL itself returned a 404 or 410 status code, but note that gitlab.com today does not:

~$ curl -sI https://gitlab.com/nonexistentuser/xyz.git | grep HTTP
HTTP/1.1 401 Unauthorized
bcmills picture bcmills on 5 Sep 2019

I'm running into issues with GitLab subgroups as well, but I'm getting a different error:

$ go get -u example.com/group/subgroup/my-go-project.git
get "example.com/group/subgroup": found meta tag get.metaImport{Prefix:"example.com/group/subgroup", VCS:"git", RepoRoot:"https://example.com/group/subgroup.git"} at //example.com/group/subgroup?go-get=1
go: finding example.com/group/subgroup/my-go-project.git latest
go get example.com/group/subgroup/my-go-project.git: git ls-remote -q https://example.com/group/subgroup.git in C:\Users\dev\go\pkg\mod\cache\vcs\5af58fdf9c9bd0fd298253dde7a0add3f4e5d6be34206b0b46f6805ee365d4ea: exit status 128:
        remote: The project you were looking for could not be found.
        fatal: repository 'https://example.com/group/subgroup.git/' not found

It almost seems like it's assuming that the second part of the path must be the project. Going back to 1.12 fixes it.

Edit: Nevermind, re-ran it with -v and it's the same issue mentioned above with GitLab returning the parent.

tangtony picture tangtony on 5 Sep 2019
👍2

@bcmills I mean if a try on gitlab.com/group/project succeeds, we can ignore errors encountered trying gitlab.com/group and gitlab.com/group/project/submodule, no matter what they are.

leonshaw picture leonshaw on 5 Sep 2019

@leonshaw, maybe..? The problem with that is that it's asymmetric.

If _I_ have credentials for module example.com/foo containing subdirectory ./bar/baz, and _you_ have credentials for example.com/foo/bar containing subdirectory ./baz, and we both run go get example.com/foo/bar/baz, then we'll end up with different results for the exact same query.

The solution to that is to treat responses other than 404/410 as hard errors. Then I'll fail to fetch example.com/foo/bar, and you'll fail to fetch example.com/foo, and we'll both (consistently!) know that we don't have enough credentials to produce a consistent result.

(More generally: if servers want to avoid leaking information to unauthenticated users, then it's more robust to deny the existence of every module rather than claim the existence of every module — especially now that #29888 is fixed.)

bcmills picture bcmills on 5 Sep 2019

f I have credentials for module example.com/foo containing subdirectory ./bar/baz, and you have credentials for example.com/foo/bar containing subdirectory ./baz, and we both run go get example.com/foo/bar/baz, then we'll end up with different results for the exact same query.

I don't think this is possible, at least with giltab, as they don't have "directory-level permissions" but repository-level only. Probably the same for gihub.

Even in case, if a user has credentials to every repo on the server, go get will fail to get that non-existing parent, but instead of the authorization failure git ls-remote complains about "Could not read from remote repository."

> go get -v example.com/commons/pkg/multierror
get "example.com/commons/pkg": found meta tag get.metaImport{Prefix:"example.com/commons/pkg", VCS:"git", RepoRoot:"https://example.com/commons/pkg.git"} at //example.com/commons/pkg?go-get=1
get "example.com/commons/pkg/multierror": found meta tag get.metaImport{Prefix:"example.com/commons/pkg/multierror", VCS:"git", RepoRoot:"https://example.com/commons/pkg/multierror.git"} at //example.com/commons/pkg/multierror?go-get=1
go get example.com/commons/pkg/multierror: git ls-remote -q https://example.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
    > GitLab: The project you were looking for could not be found.
    fatal: Could not read from remote repository.
umputun picture umputun on 5 Sep 2019

If _I_ have credentials for module example.com/foo containing subdirectory ./bar/baz, and _you_ have credentials for example.com/foo/bar containing subdirectory ./baz, and we both run go get example.com/foo/bar/baz, then we'll end up with different results for the exact same query.

Is this something more fundamental? (A module can't be uniquely located by an import path.)

leonshaw picture leonshaw on 5 Sep 2019

at least with [gitlab], as they don't have "directory-level permissions" but repository-level only. Probably the same for [github].

GitLab and GitHub are not the only ways to host a module. Ideally, the semantics of the go command should generalize to self-hosted domains with arbitrary structure.

bcmills picture bcmills on 5 Sep 2019

Is this something more fundamental? (A module can't be uniquely located by an import path.)

A _module_ has a unique path. The mapping of _package paths_ to modules is not guaranteed to be 1:1, although the go command fails with an error message if any one package within the package import graph is provided by more than one module in the module graph.

bcmills picture bcmills on 5 Sep 2019

A _module_ has a unique path. The mapping of _package paths_ to modules is not guaranteed to be 1:1, although the go command fails with an error message if any one package within the package import graph is provided by more than one module in the module graph.

Yes, this is the problem. We need a way to avoid the collision, if possible. If it's simply treated as error, example.com/foo/bar will be unusable if example.com/foo exists.

leonshaw picture leonshaw on 5 Sep 2019

GitLab and GitHub are not the only ways to host a module. Ideally, the semantics of the go command should generalize to self-hosted domains with arbitrary structure.

Probably, self-hosted giltab used by a majority of self-hosted git systems. However, this new behavior seems to conflict not with giltab only, but gitea as well and may conflict with any system surprised by parent discovery calls. I appreciate the generalized approach and concerns about asymmetrical behavior in some theoretical cases, but the current problem if very practical one.

umputun picture umputun on 5 Sep 2019

I have most of a solution coded up, but there is a bad interaction with proxy fallback that I still need to work out.

Even then, my expectation is that a server that wishes to indicate “there is no repository here” (for a repository explicitly mentioned in a go-import <meta> tag) will serve a 404 or a 410 for the URL. I'm not sure whether gitlab.com does that for authenticated users, but for unauthenticated users they're currently serving 401s instead.

bcmills picture bcmills on 5 Sep 2019

Change https://golang.org/cl/194560 mentions this issue: cmd/go/internal/modfetch: report the module path for errors in (*codeRepo).Versions

gopherbot picture gopherbot on 10 Sep 2019

Change https://golang.org/cl/194561 mentions this issue: cmd/go/internal/modfetch/codehost: treat nonexistent repositories as “not found”

gopherbot picture gopherbot on 10 Sep 2019

@gopherbot, please backport to 1.13: this is a regression, and prevents previously-working modules from being fetched.

bcmills picture bcmills on 10 Sep 2019
👍2

Backport issue(s) opened: #34215 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

gopherbot picture gopherbot on 10 Sep 2019

I've implemented the approach described in https://github.com/golang/go/issues/34094#issuecomment-528410476. That should solve the problem if GitLab is correctly serving 404 or 410 responses for nonexistent paths for signed-in users.

However, I don't know whether they're doing that, so please help me confirm.

@umputun (or one of the other folks with this problem): please try a go command built from head and let me know whether your problem is resolved. (The easy way to try a go command built from head is to install golang.org/x/dl/gotip, then run gotip download and use gotip in place of the regular go command.)

bcmills picture bcmills on 11 Sep 2019

Change https://golang.org/cl/194679 mentions this issue: [release-branch.go1.13] cmd/go/internal/modfetch/codehost: treat nonexistent repositories as “not found”

gopherbot picture gopherbot on 11 Sep 2019

@bcmills I have tried to test it with both public gitlab as well as self-hosted and it doesn't seem to work. I have also tried it with full credentials and git's "insteadOf" mapping from https:// to git@... and it failed either. Both are working fine with 1.12.

You can try it with public repo from the original ticket, i.e. go get -v gitlab.com/umputuntests/sub/example vs gotip get -v gitlab.com/umputuntests/sub/example.

That should solve the problem if GitLab is correctly serving 404 or 410 responses for nonexistent paths for signed-in users.

I don't think giltab does it. To me it looks like they return 401 on any non-existent paths. Probably adding 401 to the list of the errors on the parent discovery you consider as "missing module" may fix it.

umputun picture umputun on 11 Sep 2019

Probably adding 401 to the list of the errors on the parent discovery you consider as "missing module" may fix it.

I don't think that would be appropriate. Per RFC 2616 §10.4.2 (emphasis mine):

If the request already included Authorization credentials, then the 401 response indicates that authorization has been refused for those credentials. If the 401 response contains the same challenge as the prior response, and the user agent has already attempted authentication at least once, then the user SHOULD be presented the entity that was given in the response, since that entity might include relevant diagnostic information.

I can certainly make the case that we should interpret a repo 404 as “not found”, but a server that _explicitly_ tells the go command to check a particular repo really should not _also_ tell the go command that it is not authorized to know whether that repo actually exists.

At the very least, if a more RFC-compliant response code is not feasible I'd like to hear the rationale from someone at GitLab.

bcmills picture bcmills on 11 Sep 2019
👍1

The ticket I have opened on gitlab was closed by them as a duplicate to another, 2y old ticket they refused to address.

From what I understand, gitlab considers 404 on non-existent repos as a security issue, i.e. someone could realize what private repo exists. I don't think it has anything to do with authorized status - 401 seems to be returned on any nonexisting path regardless of the auth status.

Generally speaking, such behavior is a little bit odd, but not obviously wrong. go get is trying to access parent(s) of the repo which may not be repo at all, but some protected page/service. Sure it will be nice if gitlab return no go-import in such a case, but I guess they consider it security issue as well.

umputun picture umputun on 11 Sep 2019

gitlab considers 404 on non-existent repos as a security issue, i.e. someone could realize what private repo exists.

Ironically, GitHub seems to take the opposite approach: they return 404 uniformly for unauthenticated users.

Generally speaking, such behavior is a little bit odd, but not obviously wrong. go get is trying to access parent(s) of the repo which may not be repo at all, but some protected page/service.

For the example of gitlab.com/umputuntests/sub/example, the cat is already out of the bag: the non-401 response for https://gitlab.com/umputuntests/sub/example already reveals the existence of the entities gitlab.com/umputuntests and gitlab.com/umputuntests/sub.

The same would seem to be true for private subgroups for appropriately-credentialed users.
So the “security issue” explanation does not seem to hold water.

Moreover, given that #29888 has been addressed in Go 1.13, it seems that they could simply stop serving the go-import meta tags in 401 responses in order to avoid revealing information to unauthenticated users.

bcmills picture bcmills on 11 Sep 2019

the non-401 response for https://gitlab.com/umputuntests/sub/example already reveals the existence of the entities gitlab.com/umputuntests and gitlab.com/umputuntests/sub

Not sure I understand. Both gitlab.com/umputuntests/sub and gitlab.com/umputuntests/sub/example are public. If you try anything completely random, you will get 401 for any non-existent as well as any private repo. By doing this they are trying to hide the presence of private repos.

http https://gitlab.com/random-user/very-very-random-repo/sub/blah.git
HTTP/1.1 401 Unauthorized

Http access to the real private repo as well as to non-existent repo redirects to gitlab.com/users/sign_in

that they could simply stop serving the go-import meta tags in 401 response

I see what you mean, but technically it is not entirely accurate - they serve meta tags with 200 status, not 401

umputun picture umputun on 11 Sep 2019

If you try anything completely random, you will get 401 for any non-existent as well as any private repo.

Sure, but the go command is not trying “anything completely random”. It is trying prefixes of the requested import path, and repositories to which it was directed via <meta> tags at those prefixes. And we are assuming that the user has sufficient credentials to at least reveal the existence of _some_ module with a matching prefix.

bcmills picture bcmills on 11 Sep 2019

Understood. I believe the root cause is meta, gitlab returns unconditionally with 200 on any path, but for the end-user, it looks and feels like a regression in go 1.3 - before update modules worked with gitlab, and after the update, it stopped to work.

I'm not sure what changed with the parent discovery between 1.2 and 1.3, but if this is due to "Go 1.13 tries all module prefixes in parallel, and if another error other than "not-found" happens, the query fails." can this logic be adjusted to succeed if the repo has go.mod regardless of the parrent status? If such behavior is not acceptable generally, maybe some env param to make gitlab happy and allow such forgiving modules discovery?

umputun picture umputun on 11 Sep 2019

can this logic be adjusted to succeed if the repo has go.mod regardless of the parrent status?

The presence or absence of a go.mod file seems largely independent. We could perhaps go back to ignoring errors for shorter paths if a longer path succeeded, but I would rather we get the server fixed to return semantically-appropriate responses.

bcmills picture bcmills on 12 Sep 2019

Agree, get the server fixed is the right way, but pragmatically, chances they are going to fix it very slim.

From what I see, as of now "1.13 -> gitlab" can be addressed in 4 ways:

  • gitlab change the behavior and makes it correct from go get (>=1.13) point of view. This unlikely to happen soon (ever?)
  • ignoring errors for shorter path - sounds doable, but you the expert.
  • Writing some kind of go-git-proxy sitting between "go get" and gitlab repo. It may work, but this thing is going to be ugly, with url rewrites and extra configuration of git. Doable if no other choice, but ignoring errors seems to be a better one.
  • There is one more workaround - define a bunch of rewrites in go.mod, one for each gitlab's module. This one will do the job, but it is not just an ugly hack but partially defeats the purpose of modules.
replace gitlab.example.com/commons/pkg/mongo => gitlab.example.com/commons/pkg/mongo.git v1.0.3
umputun picture umputun on 12 Sep 2019

@umputun, note that if the module is already in the build list, we don't bother searching all possible paths for it. So one workaround, if you already know the module or repo path, is to simply run something like

go mod edit -require gitlab.com/umputuntests/sub/example@master

in order to prime the appropriate paths before running any remaining go get subcommands.

foo.example.com$ go1.13 mod init foo.example.com
go: creating new go.mod: module foo.example.com

foo.example.com$ go1.13 mod edit -require gitlab.com/umputuntests/sub/example@master

foo.example.com$ go1.13 list -m all
go: finding gitlab.com/umputuntests/sub/example master
go: finding github.com/davecgh/go-spew v1.1.0
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/stretchr/objx v0.1.0
go: finding github.com/stretchr/testify v1.3.0
foo.example.com
github.com/davecgh/go-spew v1.1.0
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.0
github.com/stretchr/testify v1.3.0
gitlab.com/umputuntests/sub/example v0.0.2

bcmills picture bcmills on 12 Sep 2019
👍6

I'd still like to hear from someone at @gitlabhq (maybe @stanhu?) to get their take on this.

bcmills picture bcmills on 12 Sep 2019

Agree, get the server fixed is the right way, but pragmatically, chances they are going to fix it very slim.

👋 thanks for the ping. I've read https://gitlab.com/gitlab-org/gitlab-ce/issues/67020#note_212856862 and the thread above, and want to confirm I understand the preferred fix.

@umputun by "right way", do you mean returning 404 rather than 401?

Since this used to work in 1.12, could you help me understand the changes made on the go side in 1.13 that means this is problematic? I'm just trying to get a handle on the broader context.

jamesramsay picture jamesramsay on 13 Sep 2019

@jamesramsay correct. I believe (@bcmills correct if I wrong) 404 will fix the problem.

Another way to fix it - return 404 on GET <path>?go-get=1 for non-repo paths, i.e. groups, subgroups and anything else what is not a repository.

umputun picture umputun on 13 Sep 2019

@jamesramsay correct. I believe (@bcmills correct if I wrong) 404 will fix the problem.

Another way to fix it - return 404 on GET <path>?go-get=1 for non-repo paths, i.e. groups, subgroups and anything else what is not a repository.

Technically, GET <path>.git and GET <path>?go-get=1 should return the same type of error in this case.

leonshaw picture leonshaw on 13 Sep 2019

@jamesramsay

I've read […], and want to confirm I understand the preferred fix.

With CL 194679 in, there are several possible fixes:

  1. You could continue to serve <meta name="go-import" […]> tags for all URLs that could potentially exist as repositories, but serve 404s or 410s (instead of 401s) for requests for the actual repository (https://[…].git) when that the user is authorized to access any repository that has the requested path as a prefix.

    • For example: if the user is authorized to see https://gitlab.com/umputuntests/sub/example.git, then a response for https://gitlab.com/umputuntests/sub.git or https://gitlab.com/umputuntests.git should have code 404 or 410, not 401.

  2. You could remove the <meta name="go-import" […]> tags from https://[…]?go-get=1 responses for all URLs that don't correspond to actual repositories. (The HTTP response code does not matter as long as it is not a redirect.)

    • To preserve path secrecy, you would presumably want to also remove <meta […]> tags for repositories that the requester is not authorized to see. Unfortunately, that might cause go get to fail for users on Go 1.12 due to #29888 (which is fixed in 1.13).

  3. You could remove the <meta name="go-import" […]> tags from https://[…]?go-get=1 responses for all URLs that don't correspond to actual repositories _when the user is authorized to access a repository that has the requested path as a prefix_.

    • For example, if the user is authorized to see https://gitlab.com/umputuntests/sub/example.git, then a response for https://gitlab.com/umputuntests/sub?go-get=1 should omit the <meta […]> tag, since the user already knows that that URL does not correspond to an actual repository.
bcmills picture bcmills on 13 Sep 2019

@leonshaw

Technically, GET <path>.git and GET <path>?go-get=1 should return the same type of error in this case.

That's true, but turns out not to matter that much: the go command intentionally looks for <meta name="go-import" […]> tags even within 404 (or 401) responses.

(That is: the _content_ is what matters for the ?go-get=1 request, but the _status_ is what matters for the .git request, and the two are mostly orthogonal.)

bcmills picture bcmills on 13 Sep 2019

@jamesramsay

Since this used to work in 1.12, could you help me understand the changes made on the go side in 1.13 that means this is problematic?

In CL 173017, I changed the go command's algorithm for mapping an unresolved import path to the corresponding module path.

It was previously a sequential search starting from the longest path, working downward to progressively shorter paths, and the error handling was a bit sloppy.

Now, it searches in parallel for modules at all possible prefixes of the import path, which can significantly reduce overall latency (especially when a proxy is involved). However, the fact that the search occurs in parallel means that we end up trying a lot of potential module paths that turn out not to be valid.

We're fine with an invalid module path, _as long as_ the error is that the module does not exist.

  • The main mechanism for a server to indicate “the module does not exist” is to not serve a <meta name="go-import" […]> tag for the ?go-get=1 request for that path.

  • As of CL 194679 (which is merged but not released), we also support a secondary mechanism: if the server provides a <meta […]> tag with an HTTPS URL, and that URL serves status 404 or 410, we interpret that to mean “the module would be at that URL if it existed, but it's not there so it doesn't exist.”

bcmills picture bcmills on 13 Sep 2019

@bcmills - I had a chance to test this issue with gitea. For 1.13 the situation is similar to gitlab's, i.e. the same regression (worked with 1.12 and fails on 1.13). However, with gotip it worked fine. gitea seems to return go meta on any path, pretty much the same way as gitlab. However for those non-existing repos gitea, unlike gitlab, returns 404.

umputun picture umputun on 16 Sep 2019
👍1

@jamesramsay, note the corresponding Gitea issue in https://github.com/go-gitea/gitea/issues/8095.

bcmills picture bcmills on 18 Sep 2019

Thanks @bcmills and @umputun for all that context.

Our plan is to start returning a 404 in response to the first stage ?go-get=1 request if the project doesn't exist or if the user is not sufficiently authenticated. We're tracking the issue https://gitlab.com/gitlab-org/gitlab/issues/30612. We don't have capacity to look into this in the current development cycle, but I'll try to have this scheduled soon. Of course, anyone from the community would also be welcome to contribute a fix too 😄

jamesramsay picture jamesramsay on 19 Sep 2019
👍1

Is there a reason we can't do the module search in parallel but still walk the results in reverse order of specificity, thus ignoring errors unless every path had an error?

The permissions argument seems like a pretty distant edge case. In general, Go assumes one module path means one source code repository. If you were to return different results based on user identity, you could do that at the git layer in the same way you might at the ?go-get= resolution layer. Either way, you're seriously breaking the Go view of the world, and you'll probably know you are, but have deliberately decided to take on any pain associated with differing code results.

The way things are now implemented, that potential for pain for offenders now becomes real pain for anybody who relied on the more obvious behavior before, including indirectly, as in the case of those simply using these version control systems.

andrewstuart picture andrewstuart on 21 Sep 2019
👍3

Is there a reason we can't do the module search in parallel but still walk the results in reverse order of specificity, thus ignoring errors unless every path had an error?

Yeah, given the timeframe for fixes to the serving paths we should probably just do that.

(I'm not particularly happy about burying errors in general, but it seems like the least-unpleasant solution here.)

bcmills picture bcmills on 24 Sep 2019
👍3

Change https://golang.org/cl/197059 mentions this issue: cmd/go: suppress errors in package-to-module queries if the package is already found

gopherbot picture gopherbot on 24 Sep 2019

Change https://golang.org/cl/197063 mentions this issue: [release-branch.go1.13] cmd/go: suppress errors in package-to-module queries if the package is already found

gopherbot picture gopherbot on 24 Sep 2019

@umputun and others: this should be fixed at head and on release-branch.go1.13. Could someone with access to a private GitLab subgroup please try a build from release-branch.go1.13 and confirm?

(The instructions are here, but instead of git checkout master you'll want to run git checkout release-branch.go1.13.)

bcmills picture bcmills on 25 Sep 2019
🎉3

Just tested the latest build on release-branch.go1.13 and can confirm it works with a private GitLab subgroup.

tangtony picture tangtony on 26 Sep 2019
🎉4 👍2

works fine.

umputun picture umputun on 26 Sep 2019

Thanks for confirming.

In general we don't have a lot of different configurations of Git servers running in our test environment, so please do be sure to try out Beta and RC releases to catch any future regressions. (We do our best to test the cases we know about, but we don't know about all of the possible interactions.)

I'm going to leave this open to add a regression test on our end — I think this behavior, at least, we can replicate.

bcmills picture bcmills on 26 Sep 2019

Same problems...

GerryLon picture GerryLon on 11 Oct 2019
👎1

@GerryLon, see https://github.com/golang/go/wiki/NoPlusOne.

bcmills picture bcmills on 11 Oct 2019

Does this mean that this will get fixed for Go1.14, but will still not work for 1.13?

xavivars picture xavivars on 12 Oct 2019

@xavivars, this has already been fixed at head and backported to the 1.13 branch. It will be included in Go 1.13.3.

bcmills picture bcmills on 12 Oct 2019
🎉6

I've just tried with Go 1.13.3 but I still failed in go get my subgroup repository.

$ go version
go version go1.13.3 linux/amd64

$ go get -v gitlab.com/foo/bar/example
get "gitlab.com/foo/bar": found meta tag get.metaImport{Prefix:"gitlab.com/foo/bar", VCS:"git", RepoRoot:"https://gitlab.com/foo/bar.git"} at //gitlab.com/foo/bar?go-get=1
get "gitlab.com/foo/bar/example": found meta tag get.metaImport{Prefix:"gitlab.com/foo/bar", VCS:"git", RepoRoot:"https://gitlab.com/foo/bar.git"} at //gitlab.com/foo/bar/example?go-get=1
get "gitlab.com/foo/bar/example": verifying non-authoritative meta tag
go get gitlab.com/foo/bar/example: git ls-remote -q https://gitlab.com/foo/bar.git in /home/koala/Workspace/gopath/pkg/mod/cache/vcs/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: exit status 128:
    The project you were looking for could not be found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

P.S. My repository is a private repository. I don't know if it is a matter or not.

yookoala picture yookoala on 18 Oct 2019
👍2

@yookoala, you probably need to configure a git credential helper, or add a .netrc file containing credentials. (The go command is not intended to prompt for interactive logins.)

See https://git-scm.com/docs/gitcredentials and #29953.

bcmills picture bcmills on 18 Oct 2019
👍1

I'm still seeing an issue with 1.13.3 on linux. It's pulling the code, but fails with an error.

go version go1.13.3 linux/amd64

GOPRIVATE=gitlab.my-org.com/* go get gitlab.my-org.com/subgroup/test/go-micro-example
go: finding gitlab.my-org.com/subgroup/test/go-micro-example latest
# runtime
/usr/local/go/src/runtime/stubs_x86.go:10:6: stackcheck redeclared in this block
        previous declaration at /usr/local/go/src/runtime/stubs_amd64x.go:10:6
/usr/local/go/src/runtime/unaligned1.go:11:6: readUnaligned32 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/alg.go:321:40
/usr/local/go/src/runtime/unaligned1.go:15:6: readUnaligned64 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/alg.go:329:40
six-arm picture six-arm on 18 Oct 2019

@six-arm, that error message seems unrelated to the issue reported here. Please open a new issue with steps to reproduce.

Given that there is no src/runtime/stubs_x86.go file on release-branch.go1.13, I suspect that there is some problem with your GOROOT setting vs. the go command in use.

bcmills picture bcmills on 18 Oct 2019

go version 1.13.3 fixed the problem. I have tried it against public (gitlab hosted) repo as well as self-hosted giltab. In both cases worked as expected. thx @bcmills and others helped to diagnose and address the problem.

should I close the ticket? This was my first ticket here and I'm not sure what the rule.

umputun picture umputun on 19 Oct 2019

@umputun, thanks for confirming! I'm planning to leave the issue open for adding a more realistic regression test in 1.14 (probably during the freeze).

bcmills picture bcmills on 19 Oct 2019

go version 1.13.3 fixed the problem. I have tried it against public (gitlab hosted) repo as well as self-hosted giltab. In both cases worked as expected. thx @bcmills and others helped to diagnose and address the problem.

should I close the ticket? This was my first ticket here and I'm not sure what the rule.

My go version:

C:\Users\my-gitlab>go version
go version go1.13.3 windows/amd64

simple description:
with private subgroup can not be go get, see below:
my-group are public
my-private-group and it's project sample all are private
my-group and it's project sample all are public
private-demo are private

C:\Users\my-gitlab>go get git2.my-gitlab.net.cn/my-group/my-subgroup/sample
go: finding git2.my-gitlab.net.cn/my-group/my-subgroup/sample latest
go: downloading git2.my-gitlab.net.cn/my-group/my-subgroup/sample v0.0.0-20191022081546-6534fce037e9
go: extracting git2.my-gitlab.net.cn/my-group/my-subgroup/sample v0.0.0-20191022081546-6534fce037e9

C:\Users\my-gitlab>go get git2.my-gitlab.net.cn/my-group/my-private-subgroup/sample
go get git2.my-gitlab.net.cn/my-group/my-private-subgroup/sample: git ls-remote -q origin in D:\workspace\code\go\pkg\mod\cache\vcs\8314af4c2579c0ee8b974f039f8246d179834d34c834680bfe59ce583888c820: exit status 128:
        remote: The project you were looking for could not be found.
        fatal: repository 'https://git2.my-gitlab.net.cn/my-group/my-private-subgroup.git/' not found

C:\Users\my-gitlab>go get git2.my-gitlab.net.cn/my-group/private-demo
go: finding git2.my-gitlab.net.cn/my-group/private-demo latest
go: downloading git2.my-gitlab.net.cn/my-group/private-demo v0.0.0-20191022084509-90eb539586c8
go: extracting git2.my-gitlab.net.cn/my-group/private-demo v0.0.0-20191022084509-90eb539586c8

So, have you tested private group or subgroup on gitlab?

Thanks anyway!

GerryLon picture GerryLon on 22 Oct 2019

have you tested private group or subgroup on gitlab?

No, I didn't but tried it now with similar setup - commons group public, pkg subgroup public and syncs repo private (internal in gitlab's permissions). I see very similar results:

go version go1.13.3 darwin/amd64

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
go get git.internal.com/commons/pkg/syncs: git ls-remote -q https://git.internal.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
    > GitLab: The project you were looking for could not be found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

Not sure if important, but to access private repos from go I use insteadOf rewrite:

 [url "[email protected]:"]
     insteadOf = https://git.internal.com/
umputun picture umputun on 22 Oct 2019
👍2

@umputun, thanks for the additional detail.

This line in particular:

get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1

seems to indicate a bug or misconfiguration on the server side. If the repo is hosted at git.internal.com/commons/pkg/syncs, then the server should reply with that as the prefix and https://git.internal.com/commons/pkg/syncs.git as the repo root (instead of the git.internal.com/commons/pkg subgroup as shown).

Did that repository work with Go 1.12?

bcmills picture bcmills on 22 Oct 2019

doesn't work with 1.12 either

go version go1.12.12 linux/amd64, from golang:1.12 container:

go get -v git.internal.com/commons/pkg/syncs

Fetching https://git.internal.com/commons/pkg/syncs?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg/syncs?go-get=1 (status code 200)
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at https://git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
Fetching https://git.internal.com/commons/pkg?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg?go-get=1 (status code 200)
Fetching https://git.internal.com/commons/pkg?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg?go-get=1 (status code 200)
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at https://git.internal.com/commons/pkg?go-get=1
Fetching https://git.internal.com/commons?go-get=1
Parsing meta tags from https://git.internal.com/commons?go-get=1 (status code 200)
Fetching https://git.internal.com?go-get=1
Parsing meta tags from https://git.internal.com?go-get=1 (status code 200)
go get git.internal.com/commons/pkg/syncs: git ls-remote -q origin in /go/pkg/mod/cache/vcs/82b580717104e8b7bf1de06ef2b675a3ca5bf7d044f776fac4629633677f22da: exit status 128:
    > GitLab: The project you were looking for could not be found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

However with ~/.netrc everyting seems to work (1.13.3):

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg/syncs", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg/syncs.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
go: finding git.internal.com/commons/pkg/syncs v1.1.0
go: downloading git.internal.com/commons/pkg/syncs v1.1.0
go: extracting git.internal.com/commons/pkg/syncs v1.1.0
get "git.internal.com/commons/pkg/multierror": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg/multierror", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg/multierror.git"} at //git.internal.com/commons/pkg/multierror?go-get=1
git.internal.com/commons/pkg/multierror
git.internal.com/commons/pkg/syncs

So, looks like .gitconfig insteadOf doesn't play well with go mods, but .netrc is fine.

umputun picture umputun on 22 Oct 2019
👍2

.netrc is what I would recommend anyway, so it sounds like the overall state is improving, at least. (.netrc wasn't working well in 1.12.)

bcmills picture bcmills on 22 Oct 2019

have you tested private group or subgroup on gitlab?

No, I didn't but tried it now with similar setup - commons group public, pkg subgroup public and syncs repo private (internal in gitlab's permissions). I see very similar results:

go version go1.13.3 darwin/amd64

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
go get git.internal.com/commons/pkg/syncs: git ls-remote -q https://git.internal.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
  > GitLab: The project you were looking for could not be found.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.

Not sure if important, but to access private repos from go I use insteadOf rewrite:

 [url "[email protected]:"]
     insteadOf = https://git.internal.com/

it can not work as expected.

GerryLon picture GerryLon on 23 Oct 2019
👎1

I'm still facing this issue with go version go1.13.3 darwin/amd64

sha256x picture sha256x on 26 Oct 2019
👍3

@sha256x, “this issue” as reported by its original author has been verified as fixed. If you're seeing a similar symptom, please open a new issue with details and steps to reproduce it.

bcmills picture bcmills on 28 Oct 2019

@bcmills @sha256x @umputun There is still the issue where with git-based URL rewriting in combination with GitLab groups you cannot successfully go get ... a repo unless you tell go get in the import path where the git repo actually is, such as foo.bar.org/user/group1/group2/project.git -- and then go modules throw up because the canonical path of that module is foo.bar.org/user/group1/group2/project without any .git adornments. Am I supposed to change the canonical import paths of my private Go modules now to carry that .git adornment?

TheDiveO picture TheDiveO on 14 Nov 2019

have you tested private group or subgroup on gitlab?

No, I didn't but tried it now with similar setup - commons group public, pkg subgroup public and syncs repo private (internal in gitlab's permissions). I see very similar results:

go version go1.13.3 darwin/amd64

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
go get git.internal.com/commons/pkg/syncs: git ls-remote -q https://git.internal.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
    > GitLab: The project you were looking for could not be found.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

Not sure if important, but to access private repos from go I use insteadOf rewrite:

 [url "[email protected]:"]
     insteadOf = https://git.internal.com/

it can not work as expected.

Can you please explain? (otherwise, I cannot see how your reply might be of use to those who are not enlightened enough, such as me.)

TheDiveO picture TheDiveO on 14 Nov 2019

@TheDiveO We had a similar problem, we solved it by using the replace directive. For example:

require (
    private.gitlab.instance/project/team/somename v1.0.0
)

replace (
    private.gitlab.instance/project/team/somename => private.gitlab.instance/project/team/somename.git v1.0.0
)

However please be aware that this solution does not work cleanly when trying to upgrade that said package to non-tagged versions, for example you would need to do something like:

go get private.gitlab.instance/project/team/somename.git@somecommithash

Which then will fail with a message similar to:

go: finding private.gitlab.instance/project/team somecommithash
go: finding private.gitlab.instance/project/team/somename.git somecommithash
go: downloading private.gitlab.instance/project/team/somename.git vXYZ-somecommithash
go: extracting private.gitlab.instance/project/team/somename.git vXYZ-somecommithash
go get: private.gitlab.instance/project/team/somename.git@vXYZ-somecommithash: parsing go.mod:
    module declares its path as: private.gitlab.instance/project/team/somename
            but was required as: private.gitlab.instance/project/team/somename.git

We literally have to copy vXYZ-somecommithash, manually edit go.mod and then overwrite that value, changing it to something like

require (
    private.gitlab.instance/project/team/somename v1.0.0
)

replace (
    private.gitlab.instance/project/team/somename => private.gitlab.instance/project/team/somename.git vXYZ-somecommithash
)
MarioCarrion picture MarioCarrion on 21 Nov 2019
🎉1 👍1

@TheDiveO, @MarioCarrion: if you are having issues with GitLab subgroups for reasons unrelated to the root cause originally reported for this issue, please open a new issue with details and steps to reproduce. Thanks.

bcmills picture bcmills on 5 Dec 2019

I didn't get to the more realistic regression test during the 1.14 cycle, and by now I don't remember which cases were remaining to be tested. 😞

I'm happy to review further improvements to GitLab integration testing, but I don't think we need to leave this issue open.

bcmills picture bcmills on 21 Feb 2020

GitLab Community Edition 12.3.5 self-hosted
golang 1.13.8 & 1.14
this issue still there

treeman-zhou picture treeman-zhou on 26 Feb 2020

@Yi-Hunter, “this issue” as reported by its original author has been verified as fixed. If you're seeing a similar symptom, please open a new issue with details and steps to reproduce it.

bcmills picture bcmills on 26 Feb 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ajstarks picture ajstarks  ·  3Comments
bbodenmiller picture bbodenmiller  ·  3Comments
michaelsafyan picture michaelsafyan  ·  3Comments
stub42 picture stub42  ·  3Comments
mingrammer picture mingrammer  ·  3Comments