Stack: "ls dependencies" should have an option to maintain the tree structure

[sschuberth]

Currently, "ls dependencies" just outputs a flat list of package names and versions of all transitive dependencies of a project. However, it would be nice to maintain the tree hierarchy of dependencies to see their parent / child relationship. On the console, the tree output could look similar to Maven's mvn dependency:tree output, but ideally there would also be an option to output to a machine readable format like JSON.

[dbaynard]

Hello @sschuberth . That sounds like a helpful enhancement. How much of what you would like to do can you achieve with stack dot --external (perhaps with --include-base, too)?

[sschuberth]

I was thinking about (mis-)using stack dot, too, but while it basically provides the parent -> child relationship, it's a bit inconvenient to parse, also because it repeats the parent for each child. The versions are missing, too, but together with the output from stack ls dependencies I could probably get what I want. I'll give it a try.

[dbaynard]

That's all useful to know. It may be relatively straightforward to reuse the code required for the dot output. Would you be willing to contribute a PR yourself?

[sschuberth]

I'm sorry, I'm not a Haskell guy (and I don't intend to become one 😉), I'm just trying to add basic Haskell support to the OSS Review Toolkit.

[dbaynard]

Oh, that's an interesting project. Is there anything else that would be helpful for you to achieve that, from stack?

It might also be possible for you to do what you intend using the hackage or stackage APIs directly (depending on scale), though they are quite limited. If stack meets your other requirements, then great.

[sschuberth]

Is there anything else that would be helpful for you to achieve that, from stack?

Thanks for asking. Besides listing dependencies as a tree (optionally in a machine readable format), it would be nice if the shown meta-data would not only include the name and version, but also the dependency's declared license (as an SPDX identifier). That would be the very minimum, IMO. More meta-data like the download URL would also be interesting and could be added, or simply a link to the respective hackage or stackage (REST?) APIs endpoints where more information can be gathered by the client (the program using the meta-data output) on demand.

In general, from a performance perspective it would be nice if all this meta-data could be gathered without internally downloading and unpacking the whole (source code of a) package, i.e. ideally meta-data for a package and the package itself should be separated, similar to Maven POM and JAR files.

using the hackage or stackage APIs directly

While I've found http://hackage.haskell.org/api, I didn't find something similar for stackage. Is there? Edit: I just found https://groups.google.com/forum/#!topic/stackage/9h_RA-jNz_g.

[dbaynard]

Perhaps we should broaden the issue, then. It seems like a useful addition, though it may not be a priority for the core team.

So to summarize,

• [x] List dependencies as tree
• [ ] List dependencies in machine readable format

• [ ] List license as SPDX in addition to name and version

  Most cabal files don't use these identifiers; internally Cabal (which processes these files and is different to cabal-install, the provider of the command line program cabal) uses an enumeration, though, so you may wish to take a look thee.

• [ ] List URI too

In general, from a performance perspective it would be nice if all this meta-data could be gathered without internally downloading and unpacking the whole (source code of a) package, i.e. ideally meta-data for a package and the package itself should be separated, similar to Maven POM and JAR files.

stack downloads and unpacks the hackage tarball (~100MB compressed), which contains the cabal files where the metadata is located; this might update incrementally (I've never investigated). The cabal files themselves are reachable directly (/package/:package/:cabal.cabal). So the trade-off is a space vs latency one. If you're happy with the space use, stack would be useful for this.

I didn't find something similar for stackage

I think I might have just tried the hackage endpoints when I was looking for snapshot information… or I may have checked the api calls in the debugger. I think the hackage api is most relevant, though — I only needed stackage because of stack snapshots. That said, it may be useful to add the hackage mirrors, if you go that way.

[dbaynard]

@borsboom — it may be worth supporting this enhancement to support the OSS review toolkit project mentioned earlier.

[borsboom]

I'm very much in favour. I've often wished for an ls-dependencies that shows the tree structure. That said, I'm going to un-assign myself as it's very unlikely I'll actually be able to work on implementing it.

[bitemyapp]

$ cargo tree
lefortovo v0.1.0 (file:///home/callen/work/lefortovo)
├── bytes v0.4.8
│   ├── byteorder v1.0.0
│   └── iovec v0.1.2
│       └── libc v0.2.42
├── clap v2.25.0
│   ├── ansi_term v0.9.0
│   ├── atty v0.2.2
│   │   └── libc v0.2.42 (*)
│   ├── bitflags v0.9.1
│   ├── strsim v0.6.0
│   ├── term_size v0.3.0
│   │   └── libc v0.2.42 (*)
│   ├── textwrap v0.6.0
│   │   ├── term_size v0.3.0 (*)
│   │   └── unicode-width v0.1.4
│   ├── unicode-segmentation v1.1.0
│   ├── unicode-width v0.1.4 (*)
│   └── vec_map v0.8.0
├── hyper v0.12.5
│   ├── bytes v0.4.8 (*)
│   ├── futures v0.1.21
│   ├── futures-cpupool v0.1.8
│   │   ├── futures v0.1.21 (*)
│   │   └── num_cpus v1.6.2
│   │       └── libc v0.2.42 (*)
│   ├── h2 v0.1.10
│   │   ├── byteorder v1.0.0 (*)
│   │   ├── bytes v0.4.8 (*)
│   │   ├── fnv v1.0.6
│   │   ├── futures v0.1.21 (*)
│   │   ├── http v0.1.7
│   │   │   ├── bytes v0.4.8 (*)
│   │   │   ├── fnv v1.0.6 (*)
│   │   │   └── itoa v0.4.1
│   │   ├── indexmap v1.0.1
│   │   ├── log v0.4.3
│   │   │   └── cfg-if v0.1.4
│   │   ├── slab v0.4.0
│   │   ├── string v0.1.0
│   │   └── tokio-io v0.1.7
│   │       ├── bytes v0.4.8 (*)
│   │       ├── futures v0.1.21 (*)
│   │       └── log v0.4.3 (*)
│   ├── http v0.1.7 (*)
│   ├── httparse v1.2.3
│   ├── iovec v0.1.2 (*)
│   ├── itoa v0.4.1 (*)
│   ├── log v0.4.3 (*)
│   ├── net2 v0.2.33
│   │   ├── cfg-if v0.1.4 (*)
│   │   └── libc v0.2.42 (*)
│   ├── time v0.1.37
│   │   └── libc v0.2.42 (*)
│   ├── tokio v0.1.7
│   │   ├── futures v0.1.21 (*)
│   │   ├── mio v0.6.14
│   │   │   ├── iovec v0.1.2 (*)
│   │   │   ├── lazycell v0.6.0
│   │   │   ├── libc v0.2.42 (*)
│   │   │   ├── log v0.4.3 (*)
│   │   │   ├── net2 v0.2.33 (*)
│   │   │   └── slab v0.4.0 (*)
│   │   ├── tokio-executor v0.1.2
│   │   │   └── futures v0.1.21 (*)
│   │   ├── tokio-fs v0.1.1
│   │   │   ├── futures v0.1.21 (*)
│   │   │   ├── tokio-io v0.1.7 (*)
│   │   │   └── tokio-threadpool v0.1.4
│   │   │       ├── crossbeam-deque v0.3.1
│   │   │       │   ├── crossbeam-epoch v0.4.3
│   │   │       │   │   ├── arrayvec v0.4.7
│   │   │       │   │   │   └── nodrop v0.1.12
│   │   │       │   │   ├── cfg-if v0.1.4 (*)
│   │   │       │   │   ├── crossbeam-utils v0.3.2
│   │   │       │   │   │   └── cfg-if v0.1.4 (*)
│   │   │       │   │   ├── lazy_static v1.0.1
│   │   │       │   │   ├── memoffset v0.2.1
│   │   │       │   │   └── scopeguard v0.3.3
│   │   │       │   └── crossbeam-utils v0.3.2 (*)
│   │   │       ├── futures v0.1.21 (*)
│   │   │       ├── log v0.4.3 (*)
│   │   │       ├── num_cpus v1.6.2 (*)
│   │   │       ├── rand v0.4.2
│   │   │       │   └── libc v0.2.42 (*)
│   │   │       └── tokio-executor v0.1.2 (*)
│   │   │   [dev-dependencies]
│   │   │   └── tokio-io v0.1.7 (*)
│   │   ├── tokio-io v0.1.7 (*)
│   │   ├── tokio-reactor v0.1.2
│   │   │   ├── futures v0.1.21 (*)
│   │   │   ├── log v0.4.3 (*)
│   │   │   ├── mio v0.6.14 (*)
│   │   │   ├── slab v0.4.0 (*)
│   │   │   ├── tokio-executor v0.1.2 (*)
│   │   │   └── tokio-io v0.1.7 (*)
│   │   ├── tokio-tcp v0.1.0
│   │   │   ├── bytes v0.4.8 (*)
│   │   │   ├── futures v0.1.21 (*)
│   │   │   ├── iovec v0.1.2 (*)
│   │   │   ├── mio v0.6.14 (*)
│   │   │   ├── tokio-io v0.1.7 (*)
│   │   │   └── tokio-reactor v0.1.2 (*)
│   │   ├── tokio-threadpool v0.1.4 (*)
│   │   ├── tokio-timer v0.2.4
│   │   │   ├── futures v0.1.21 (*)
│   │   │   └── tokio-executor v0.1.2 (*)
│   │   └── tokio-udp v0.1.1
│   │       ├── bytes v0.4.8 (*)
│   │       ├── futures v0.1.21 (*)
│   │       ├── log v0.4.3 (*)
│   │       ├── mio v0.6.14 (*)
│   │       ├── tokio-codec v0.1.0
│   │       │   ├── bytes v0.4.8 (*)
│   │       │   ├── futures v0.1.21 (*)
│   │       │   └── tokio-io v0.1.7 (*)
│   │       ├── tokio-io v0.1.7 (*)
│   │       └── tokio-reactor v0.1.2 (*)
│   ├── tokio-executor v0.1.2 (*)
│   ├── tokio-io v0.1.7 (*)
│   ├── tokio-reactor v0.1.2 (*)
│   ├── tokio-tcp v0.1.0 (*)
│   ├── tokio-timer v0.2.4 (*)
│   └── want v0.0.5
│       ├── futures v0.1.21 (*)
│       ├── log v0.4.3 (*)
│       └── try-lock v0.2.2
├── hyper-tls v0.2.1
│   ├── bytes v0.4.8 (*)
│   ├── futures v0.1.21 (*)
│   ├── hyper v0.12.5 (*)
│   ├── native-tls v0.1.4
│   │   └── openssl v0.9.14
│   │       ├── bitflags v0.9.1 (*)
│   │       ├── foreign-types v0.2.0
│   │       ├── lazy_static v0.2.8
│   │       ├── libc v0.2.42 (*)
│   │       └── openssl-sys v0.9.14
│   │           └── libc v0.2.42 (*)
│   │           [build-dependencies]
│   │           ├── gcc v0.3.51
│   │           └── pkg-config v0.3.9
│   ├── tokio-io v0.1.7 (*)
│   └── tokio-tls v0.1.4
│       ├── futures v0.1.21 (*)
│       ├── native-tls v0.1.4 (*)
│       ├── tokio-core v0.1.17
│       │   ├── bytes v0.4.8 (*)
│       │   ├── futures v0.1.21 (*)
│       │   ├── iovec v0.1.2 (*)
│       │   ├── log v0.4.3 (*)
│       │   ├── mio v0.6.14 (*)
│       │   ├── scoped-tls v0.1.0
│       │   ├── tokio v0.1.7 (*)
│       │   ├── tokio-executor v0.1.2 (*)
│       │   ├── tokio-io v0.1.7 (*)
│       │   ├── tokio-reactor v0.1.2 (*)
│       │   └── tokio-timer v0.2.4 (*)
│       └── tokio-io v0.1.7 (*)
├── pretty_env_logger v0.2.3
│   ├── ansi_term v0.11.0
│   ├── env_logger v0.5.10
│   │   ├── atty v0.2.2 (*)
│   │   ├── humantime v1.1.1
│   │   │   └── quick-error v1.2.2
│   │   ├── log v0.4.3 (*)
│   │   ├── regex v1.0.1
│   │   │   ├── aho-corasick v0.6.5
│   │   │   │   └── memchr v2.0.1
│   │   │   │       └── libc v0.2.42 (*)
│   │   │   ├── memchr v2.0.1 (*)
│   │   │   ├── regex-syntax v0.6.1
│   │   │   │   └── ucd-util v0.1.1
│   │   │   ├── thread_local v0.3.5
│   │   │   │   ├── lazy_static v1.0.1 (*)
│   │   │   │   └── unreachable v1.0.0
│   │   │   │       └── void v1.0.2
│   │   │   └── utf8-ranges v1.0.0
│   │   └── termcolor v0.3.6
│   └── log v0.4.3 (*)
└── tokio-core v0.1.17 (*)

cargo tree has nice output. Cf. https://github.com/sfackler/cargo-tree

I think stack ls dependencies needs to go unchanged as other tools reuse the output for things like license checking, but a --tree flag or similar would be nice.

[sschuberth]

And so do Maven (mvn dependency:tree), Gradle (gradle dependencies), and several other package managers / dependency tools / build systems / whatever you call them...

[creichert]

NPM does as well:

$ npm ls | head
[email protected] /home/christopher/dev/proj
├── [email protected]
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ └── [email protected]
│ │ │ ├─┬ [email protected]

@sschuberth I'm sure you've seen stack ls dependencies --license? It's not SPDX format afaik, but may still be a useful interim solution.

$ stack ls dependencies --separator=- --license | head
Cabal-BSD3
HTTP-BSD3
HUnit-BSD3
HsOpenSSL-PublicDomain
JuicyPixels-BSD3
MonadRandom-OtherLicense
QuickCheck-BSD3
RSA-BSD3
SHA-BSD3
StateVar-BSD3

@borsboom Why does --license discard the version number?

[sschuberth]

The problem with NPM is that it can only show the tree for installed dependencies. But ideally I'd like to see the list / tree of dependencies before I even install them. Maven and Gradle can do that just fine.

Thanks @creichert, I've seen stack ls dependencies --license, but I've already implemented a solution that uses a mix of calling stack and parsing *.cabal files directly from Hackage.

[akshaymankar]

I was exploring the newcomer friendly tag and came across this. It looks interesting and also seem like nobody is try to implement this feature. I am going to take a dig at implementing this, please give me a shout if somebody is already doing it.

[dbaynard]

@akshaymankar For machine readable output I suggest json (unless @sschuberth has a better suggestion). It might be simple enough to implement some aeson instances and then plumb them in.

SPDX is well defined.

For the URI, I'm not sure what to go with. I don't really understand the use case, so perhaps @sschuberth can explain.

[sschuberth]

JSON is indeed fine from my perspective. But if e.g. more common in the Haskell community, YAML could be another option.

URI / URL was referring to the direct download location for the source code to the package. At the example of the memory package version 0.14.11, the URL would be https://hackage.haskell.org/package/memory-0.14.11/memory-0.14.11.tar.gz. The use-case is to perform automated license checks by scanning the source code of the package.

[akshaymankar]

The URL in the JSON doesn't really work, because a package could either be from hackage, a local package, an archive specified by URL or a git repository. So, I guess a URL to a will not always be possible.

I am not sure what the best possible format for this would be. @sschuberth, do you have any suggestions/preferences?

@dbaynard I also couldn't find a place where stack figures out URL from PakcageLocation object. Can you please point me to it, so I don't invent the logic again.

[qrilka]

@akshaymankar what's the problem to use file:// for local packages and git:// for git?

[qrilka]

And I think you'll need a custom function for that, I don't think there's already some code doing that.

[sschuberth]

I was also about to say a URL should work:

• a package could either be from hackage -> Use a http(s) URL to hackage.haskell.org
• a local package -> Use a file:// URL
• an archive specified by URL -> Use a http(s) URL to a non-hackage place
• a git repository -> Here's the problem. Git supports a variety of transports, and in particular not all Git repos can be actually accessed via the git:// protocol (as that protocol offers no authentication). So either we come up with a convention that says that http(s) URLs that refer to Git repos have to end in .git (which is the common case anyway), or we introduce some additional meta-data field to indicate which type of URL this is. If packages are never going to be hosted in some other VCS than Git, the former approach would probably work fine. But e.g. for Mercurial there is no convention that the http(s) URL to a Mercurial repo should end in .hg, and we'd need the second approach of additional meta-data.

[dbaynard]

@dbaynard I also couldn't find a place where stack figures out URL from PakcageLocation object. Can you please point me to it, so I don't invent the logic again.

And I think you'll need a custom function for that, I don't think there's already some code doing that.

I think @qrilka was answering the question you asked of me.

[akshaymankar]

I think file:// is a good idea for local packages.
For git and hg repositories, even if we go with any URL specification, we still won't be able to show the commit in the URL. So, I'm leaning towards the second approach of additional metadata.

Maybe it could look something like this for git (and similar for hg):

{
  "name": "foo",
  "version": "1.0.0.0",
  "license": "MIT",
  "location": {
    "type": "git",
    "url": "<URL provided by user>",
    "commit": "<sha>"
  }
}

For hackage:

{
  "name": "foo",
  "version": "1.0.0.0",
  "license": "MIT",
  "location": {
    "type": "hackage",
    "url": "<computed hackage URL>"
  }
}