Go: cmd/go: behavior of go list ... is not documented in module mode

[perillo]

go help packages documents the special path all both for gopath and module mode:

"all" expands to all packages found in all the GOPATH trees. For example, 'go list all' lists all the packages on the local system. When using modules, "all" expands to all packages in the main module and their dependencies, including dependencies needed by tests of any of those.

However the ... pattern is documented only for gopath module:

An import path is a pattern if it includes one or more "..." wildcards, each of which can match any string, including the empty string and strings containing slashes. Such a pattern expands to all package directories found in the GOPATH trees with names matching the patterns.

How is the ... pattern supposed to work in module mode?

As a test, I ran go list ... | wc -l inside the github.com/golang/mod (golang.org/x/mod) directory with different versions of the go tool. The directory is outside GOPATH. The results are:

• go1.11.13: 501
• go1.12.16: 526
• go1.13.8: 632
• go1.14rc1: 634

As a side note, how can I tell go list to list only packages in the main module?

[perillo]

Unrelated to this issue, I also noted that with GO111MODULE=off and GOPATH set to a not existing directory:

• go1.13.8 list ...: 300
• go1.13.8 list std cmd: 376

This seems to contrast with the documentation saying Such a pattern expands to all package directories found in the GOPATH, since it is matching package directories in GOROOT.

The result is the same with GO111MODULE=on and inside an empty module (where go list all matches no packages).

[dmitshur]

/cc @jayconrod @bcmills @matloob per owners.

[perillo]

I think that go list ... should never match packages outside the main and active modules.

The difference between go list ... and go list std cmd is that the latter also includes the vendor directories.

[bcmills]

It is true that we have not documented the ... _package_ pattern in module mode. The intent is for it to have the same meaning as in GOPATH mode: every package that can be found in any of the available package roots. (In GOPATH mode, the available package roots are GOROOT/src, and GOPATH/src for each entry in GOPATH. In module mode, the available package roots are GOROOT/src and each active module.)

The increase from Go 1.11 to 1.12 is due to the change from vendor/golang_org/x to internal/x for packages vendored into the standard library (see CL 147443).

The increase from Go 1.12 to 1.13 is due to the inclusion of the cmd subtree plus the addition of crypto/ed25519. (The previous lack of cmd packages was a bug present since Go 1.11: note that the cmd subtree was always included in GOPATH mode.)

The minor changes from Go 1.13 to 1.14 are due to changes in internal packages plus the addition of hash/maphash.

[bcmills]

The difference between GOPATH mode and module mode in Go 1.13 is due to the requirement that each package have a unique path in module mode: for example, the package reported as cmd/vendor/golang.org/x/sys/unix would be just one of many possible packages known as golang.org/x/sys/unix in GOPATH mode.

Arguably the lack of such a listing entirely in GOPATH mode is a bug, but it is a fairly minor one and at this point almost certainly not worth fixing.

[perillo]

There is still something missing, in module mode.

For go1.13, the number of packages in GOROOT is 376 (and ... only matches 300 packages).

Inside the golang.org/x/mod module:

• go1.13 list all: 170
• go1.13 list std cmd all: 392
• go1.13 list std cmd ...: 708

Am I missing something?

[bcmills]

Per https://tip.golang.org/cmd/go/#hdr-Package_lists_and_patterns:

When using modules, "all" expands to all packages in the main module and their dependencies, including dependencies needed by tests of any of those.

Since no package in the main module can import a package main from cmd, that is a strict subset of ....

[perillo]

Is it only me that thinks this is really complex?

[bcmills]

The ... pattern is just not all that useful. It's an edge-case of a general wildcard. So, yeah: edge cases are complex, but generally avoidable. (The all pattern is usually what you want instead, and that one _is_ well-defined.)

[perillo]

It could be make useful if ... matches only the packages in the main module, since there are no other means to get that list.

[myitcv]

@perillo can you not do

go list $(go list -m)/...

[bcmills]

I think we do need a pattern that matches only the packages in the main module, but ... is not that pattern.

(./... is such a pattern, but it only works in the module root.)

[bcmills]

@myitcv: go list $(go list -m)/... will catch packages in nested modules too.

[bcmills]

@jayconrod, @matloob, and I were discussing such a pattern this morning, in fact.

(We're thinking maybe mod, since it's a three-letter shortening like std and cmd and less likely than main to be used as the name of someone's local toy module.)

But that really ought to be filed as a separate issue.

[perillo]

@bcmills I tested go list $(go list -m)/... from inside the golang.org/x/tools directory and it does not list the golang.org/x/tools/gopls sub module.

[bcmills]

It would if x/tools depended on gopls, which it (intentionally) does not.

I think cloud.google.com/go has that sort of structure, though.

[myitcv]

Ok, one last try 😄

go list -f "{{\$p := .}}{{with \$p.Module}}{{if eq .Path \"$(go list -m)\"}}{{\$p.ImportPath}}{{end}}{{end}}" $(go list -m)/...