Go: cmd/go: enable listing direct dependency updates

[icholy]

Intro

The recommended approach to listing available module updates is to run the following command.

go list -u -m all

This displays every dependency along with newer available versions.

go list -u -m all
github.com/icholy/utm
github.com/google/go-cmp v0.5.0 [v0.5.1]
github.com/pkg/errors v0.8.1 [v0.9.1]
github.com/spf13/pflag v1.0.3 [v1.0.5]
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 [v0.0.0-20200709230013-948cd5f35899]
golang.org/x/net v0.0.0-20190311183353-d8887717615a [v0.0.0-20200707034311-ab3426394381]
golang.org/x/sync v0.0.0-20190423024810-112230192c58 [v0.0.0-20200625203802-6e8e738ad208]
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a [v0.0.0-20200722175500-76b94024e4b6]
golang.org/x/text v0.3.0 [v0.3.3]
golang.org/x/tools v0.0.0-20190624222133-a101b041ded4 [v0.0.0-20200723000907-a7c6fd066f6d]
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
gotest.tools/v3 v3.0.2

My go.mod only specifies two dependencies.

module github.com/icholy/utm

go 1.14

require (
    github.com/google/go-cmp v0.5.0
    gotest.tools/v3 v3.0.2
)

Problem

The go list output gets completely unmanageable when the dependency tree grows. When listing available module updates, I'm only interested in my direct dependencies. Those are the only ones I usually have control over.

The go list command has a -json flag which makes it possible to feed into external tools such as https://github.com/psampaz/go-mod-outdated. However, I find it strange that I need external tooling to ask "which modules in go.mod are outdated"

Proposal

I propose a mechanism for limiting the go list output to direct dependencies.

go list -u -m direct

Relates to #40323

[mvdan]

We've discussed this multiple times before with the tooling/modules folks, mostly on Slack and on the golang-tools monthly calls. The only written down notes I could find right now are https://github.com/golang/go/issues/27887#issuecomment-432848126; @bcmills @rogpeppe @jayconrod @myitcv might have more links.

[icholy]

The provided solution uses go lists template ability to filter out indirect dependencies.

go list -f '{{if not .Indirect}}{{.}}{{end}}' -u -m all

I don't think it's reasonable to expect users to type this out every time. A bash alias can be created, but that's not portable and has the same problems as using an external tool (you have to know about it, and set it up).

The common use-case for people developing module tooling is not the same as for people consuming the module tooling. I think that most people just want to know if they have any dependencies to go get -u. Usually in Go, simple thing are simple to do, but complex things are still possible. go list only gives us that second half.

edit: I suppose what I actually want is a veneer on top of go list (ex: go mod updates).

[jayconrod]

I think this would be good to have. It's pretty common to want to list or upgrade direct dependencies of packages in the main module. It's too hard to do that with go list right now. Some thoughts though:

• go get -d direct should work, too, not just go list -m -u direct.
• We need a clear definition of direct. When lazy loading is implemented (#36460), the set of modules required in go.mod will expand, so I don't think it would make sense to define direct module dependencies as those required in go.mod. Even today, there may be // indirect requirements, and those dependencies may not be listed anywhere else. Maybe it should be defined at the package level as "the set of packages directly imported by packages in the main module".

[icholy]

An alternative to direct is imported.

go list -m -u imported

[thepudds]

Some overlap here with #28424.

[rsc]

Thanks @thepudds. Sorry we missed discussing that one. What do you think about this proposal vs the other?

[rsc]

/cc @jayconrod @bcmills; what's the status of this?

[jayconrod]

cc @matloob

I think the idea is good, but I'm not sure about the CLI specifics.

• The direct keyword seems a little terse. I'm not sure go list direct will make sense to someone unfamiliar with it. Any other keyword suggestions?
• I'd prefer direct over imports, since we'd want direct to cover both package and module dependencies, depending on whether -m is used.
• Would go list -m direct just list requirements in go.mod? That won't be all that useful after lazy loading (#36460) because go.mod will list all modules needed to build packages in the main module, not just those providing directly imported packages.
• direct seems like more of a function than a set. In this proposal, we're talking about direct dependencies of the main module, but we might want to be able to get the direct dependencies of another set of packages: go list direct(example.com/m/...). Personally, I'd love to have a query language like bazel query, with functions like rdeps and somepath. But that's certainly overcomplicating things. If we eventually did that, direct on its own could still imply the main module or its packages.

[bcmills]

The direct keyword seems a little terse.

Heh, I had a bit of the opposite reaction! All of the existing meta-packages (all, std, cmd) are three letters, so I was kind of hoping we could find another suitable three-letter pattern. (But I can't think of one at the moment.)

[bcmills]

Would go list -m direct just list requirements in go.mod?

I think it should list only those requires that are not labeled with an // indirect comment.
That admittedly assigns semantic meaning (as an input) to comments that were previously only an output (produced, but not semantically consumed by, the go command), but I think it's more in line with user intent for the direct keyword.

Or, we could disallow the direct pattern with the -m flag, although that would arguably make the direct pattern less useful.

[bcmills]

direct seems like more of a function than a set.

I agree that it would be nice for go list to support querying “the packages directly imported by the named packages”, but I think that feature should be mostly orthogonal to the one proposed here.

(I think the hypothesized go list query is more closely related to the -deps flag than to the all pattern.)

[icholy]

Or, we could disallow the direct pattern with the -m flag, although that would arguably make the direct pattern less useful.

Using it with -m is my primary use-case.

[rsc]

It sounds like people are saying:

• In go list direct, direct expands to a list of all the packages directly imported by the current module.
• In go list -m direct, direct expands to a list of all the modules providing those packages.

That is, direct becomes a pseudo-name like std and cmd and all that expands to this meaning in all commands, not just go list. But like with go list -m all, the meaning of direct changes for go list -m (as it must, since -m is about modules, not packages).

Do I have that right? Does anyone object to that?

[icholy]

Re #28424

It seems that the direct pseudo-name should also work with go get.

go get -u direct

[rsc]

Based on the discussion, this seems like a likely accept.

[rsc]

(If accepted, this would not be until Go 1.17. There's already a lot for Go 1.16.)

[mvdan]

In go list -m direct, direct expands to a list of all the modules providing those packages.

Just so I understand, this would be a subset of the modules listed in go.mod without // indirect after a go mod tidy, right? Since lazy module loading can also add lines to go.mod for indirect modules without // indirect as far as I remember.

[rsc]

No change in consensus, so accepted.