Go: cmd/go: separate direct and indirect dependencies in go.mod

Created on 27 Sep 2018  ·  10Comments  ·  Source: golang/go

To summarize, the ordering of the go.mod file, with direct and indirect dependencies intermixed, makes it hard to view direct dependencies that I actually care about.

A proposed fix would be...

module github.com/org/repo

require (
    github.com/some/dependency v1.2.3
    github.com/another/dependency v4.5.6

    github.com/some-indirect/dependency v1.2.3 // indirect
    github.com/another-indirect/dependency v4.5.6 // indirect
)

What version of Go are you using (go version)?

go1.11 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/<<REDACTED>>/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/<<REDACTED>>/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/<<REDACTED>>/go/src/<<REDACTED>>/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lq/y1k6czrx4gld0j4wrjvy1w6r0000gn/T/go-build624209139=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Run go mod tidy in a newly initialized project.

What did you expect to see?

An orderly go.mod file, with dependencies I care about and use directly separated out from indirect dependencies.

What did you see instead?

Direct and indirect dependencies intermixed in the go.mod file, leading toward difficulty in see modules I use and care about.

NeedsDecision modules
Source
picture jamesgoodhouse
👍23

Most helpful comment

@jamesgoodhouse I understand this is not exactly what you asked for, but note that go list is pretty flexible, and can show you your direct dependencies today, I believe. For example:

$ go list -f '{{if not .Indirect}}{{.}}{{end}}' -m all                        
 github.com/you/hello
 rsc.io/quote v1.5.2

That -f filter defaults to showing the current module (the first line of output), and then shows your direct dependencies (along with their current versions). If you wanted something different more focused, there is also a .Main boolean (if you wanted to exclude the main/current module), and .Path if you only wanted to the module name without the version. You can see more details in the module-specific portion of this section of the doc:

https://tip.golang.org/cmd/go/#hdr-List_packages_or_modules

Here is the go.mod for that particular above example:

$ cat go.mod
module github.com/you/hello

require rsc.io/quote v1.5.2

In contrast, for that same example above, a go list -m all also shows indirect dependencies:

$ go list -m all
github.com/you/hello
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
rsc.io/quote v1.5.2
rsc.io/sampler v1.3.0

Again, I understand it might be more convenient to be able to more easily see direct vs. indirect in go.mod itself (for example if you are browsing someone else's repo), but wanted to at least mention some things that are supported today in the core go tooling already.

picture thepudds on 25 Oct 2018
👍6

All 10 comments

/cc @bcmills @rsc

agnivade picture agnivade on 27 Sep 2018

@gopherbot, please add label modules

thepudds picture thepudds on 21 Oct 2018

@jamesgoodhouse you might also want to see the discussion in #26955 and #26913, especially this comment https://github.com/golang/go/issues/26913#issuecomment-411976222

thepudds picture thepudds on 21 Oct 2018

Note, the list of indirect dependencies in your go.mod will approach zero over time as the set of transitive dependencies adopt Go modules.

That said, a more useful way to look at your dependencies is golang.org/x/tools/cmd/digraph. e.g. 

$ GO111MODULE=off go get golang.org/x/tools/cmd/digraph
$ cd /tmp
$ git clone https://github.com/gobuffalo/buffalo
Cloning into 'buffalo'...
$ cd buffalo/
$ go mod download
go: finding github.com/gobuffalo/mw-contenttype v0.0.0-20180802152300-74f5a47f4d56
go: finding github.com/gobuffalo/mw-paramlogger v0.0.0-20181005191442-d6ee392ec72e
...
$ go mod graph | digraph succs github.com/gobuffalo/buffalo
github.com/codegangsta/[email protected]
github.com/dgrijalva/[email protected]+incompatible
github.com/dustin/[email protected]
github.com/fatih/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected]+incompatible
github.com/gobuffalo/[email protected]+incompatible
github.com/gobuffalo/[email protected]+incompatible
github.com/gobuffalo/[email protected]
github.com/gorilla/[email protected]
github.com/gorilla/[email protected]
github.com/gorilla/[email protected]
github.com/kr/[email protected]
github.com/markbates/[email protected]
github.com/markbates/[email protected]
github.com/markbates/[email protected]
github.com/markbates/[email protected]
github.com/markbates/[email protected]
github.com/markbates/[email protected]
github.com/markbates/[email protected]
github.com/monoculum/[email protected]
github.com/nicksnyder/[email protected]
github.com/pkg/[email protected]
github.com/sirupsen/[email protected]
github.com/spf13/[email protected]
github.com/spf13/[email protected]
github.com/spf13/[email protected]
github.com/stretchr/[email protected]
golang.org/x/[email protected]
golang.org/x/[email protected]
golang.org/x/[email protected]
gopkg.in/alexcesaro/[email protected]
gopkg.in/[email protected]

digraph -help gives more details:

digraph: queries over directed graphs in text form.

Graph format:

  Each line contains zero or more words.  Words are separated by
  unquoted whitespace; words may contain Go-style double-quoted portions,
  allowing spaces and other characters to be expressed.

  Each field declares a node, and if there are more than one,
  an edge from the first to each subsequent one.
  The graph is provided on the standard input.

  For instance, the following (acyclic) graph specifies a partial order
  among the subtasks of getting dressed:

        % cat clothes.txt
        socks shoes
        "boxer shorts" pants
        pants belt shoes
        shirt tie sweater
        sweater jacket
        hat

  The line "shirt tie sweater" indicates the two edges shirt -> tie and
  shirt -> sweater, not shirt -> tie -> sweater.

Supported queries:

  nodes
        the set of all nodes
  degree
        the in-degree and out-degree of each node.
  preds <label> ...
        the set of immediate predecessors of the specified nodes
  succs <label> ...
        the set of immediate successors of the specified nodes
  forward <label> ...
        the set of nodes transitively reachable from the specified nodes
  reverse <label> ...
        the set of nodes that transitively reach the specified nodes
  somepath <label> <label>
        the list of nodes on some arbitrary path from the first node to the second
  allpaths <label> <label>
        the set of nodes on all paths from the first node to the second
  sccs
        all strongly connected components (one per line)
  scc <label>
        the set of nodes nodes strongly connected to the specified one

Example usage:

   Show the transitive closure of imports of the digraph tool itself:
   % go list -f '{{.ImportPath}}{{.Imports}}' ... | tr '[]' '  ' |
         digraph forward golang.org/x/tools/cmd/digraph

   Show which clothes (see above) must be donned before a jacket:
   %  digraph reverse jacket <clothes.txt
myitcv picture myitcv on 21 Oct 2018

@jamesgoodhouse I understand this is not exactly what you asked for, but note that go list is pretty flexible, and can show you your direct dependencies today, I believe. For example:

$ go list -f '{{if not .Indirect}}{{.}}{{end}}' -m all                        
 github.com/you/hello
 rsc.io/quote v1.5.2

That -f filter defaults to showing the current module (the first line of output), and then shows your direct dependencies (along with their current versions). If you wanted something different more focused, there is also a .Main boolean (if you wanted to exclude the main/current module), and .Path if you only wanted to the module name without the version. You can see more details in the module-specific portion of this section of the doc:

https://tip.golang.org/cmd/go/#hdr-List_packages_or_modules

Here is the go.mod for that particular above example:

$ cat go.mod
module github.com/you/hello

require rsc.io/quote v1.5.2

In contrast, for that same example above, a go list -m all also shows indirect dependencies:

$ go list -m all
github.com/you/hello
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
rsc.io/quote v1.5.2
rsc.io/sampler v1.3.0

Again, I understand it might be more convenient to be able to more easily see direct vs. indirect in go.mod itself (for example if you are browsing someone else's repo), but wanted to at least mention some things that are supported today in the core go tooling already.

thepudds picture thepudds on 25 Oct 2018
👍6

The build definition file in any system should be easily readable and understandable. I understand that typically go.mod is read and handled by the go tool, and that the need for // indirect dependencies will decrease as adoption of modules increases. However, this is still something that we will have to deal with for a while, and requiring an external tool just to read your dependencies easily isn't a good solution.

Cluttering the require block with // indirect dependencies surrounding the direct dependencies (as they are sorted alphabetically) makes it much more difficult to actually find the direct dependencies, which is the more relevant information for people when looking at the file.

Simply adding a blank line and separating the direct and indirect dependencies would go a long way in cleaning up the presentation of the go.mod file.

DemonWav picture DemonWav on 1 Nov 2018

Adding some notes per a discussion with @DemonWav on Slack.

In my experience with modules thus far, a) I never made changes to a go.mod by hand and b) any questions I have about my dependencies I ask via tools.

I agree that it's easier to glance at your go.mod to check your immediate dependencies (ignoring // indirect for a second), but I don't know what question it helps you answer. Because that list includes your test dependencies... and doesn't include your transitive non-test dependencies.

myitcv picture myitcv on 2 Nov 2018

@DemonWav, the go.mod file is oriented toward machines, not people. go list is for people, and even then you need to be specific about which question you actually want to answer. (The common questions tend to get their own flags, so feedback on which operations are useful-but-difficult is valuable.)

To answer the question “what are the modules that provide the direct imports of my non-test packages?”, you could do something like:

go list -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' \
    $(go list -f '{{range .Imports}}{{if ne . "C"}}{{.}} {{end}}{{end}}' ./...) \
    | sort -u

See also https://github.com/golang/go/issues/27900.

bcmills picture bcmills on 2 Nov 2018

It's valuable to can read go.mod on github before cloning the lib. To have an idea of the dependencies of a new lib for example.

https://research.swtch.com/vgo-module
| My goals for the file format were that it be (1) clear and simple, (2) easy for people to read

_However about the annoying // indirect it's a motivation to send PR to make the old libs module aware!_

flibustenet picture flibustenet on 2 Nov 2018
👍1

file this under the ease of use category.. If the auto formatter can make a few of our lives easier, why not do it? It should also sort the dependencies alphabetically for even easier browsing.

chirino picture chirino on 31 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

dominikh picture dominikh  ·  3Comments
longzhizhi picture longzhizhi  ·  3Comments
bradfitz picture bradfitz  ·  3Comments
rsc picture rsc  ·  3Comments
ashb picture ashb  ·  3Comments