Nixpkgs: Multiple outputs for haskell ecosystem

Created on 13 Oct 2014 · 23Comments · Source: NixOS/nixpkgs

Haskell closures could be greatly reduced if:

a) the shared libraries that come with ghc that all haskell shared libraries link to were in a separate output from the compiler, and
b) documentation were in a separate output from the binary/library being built

cc @peti

enhancement closure-size haskell

Source

shlevy

👍3

Most helpful comment

filename_stuff :: String

wow

shlevy on 14 Jun 2016

😄4

All 23 comments

Yes, I completely agree. It's probably not even hard to implement that in the generic Cabal builder.

peti on 13 Oct 2014

:+1:

copumpkin on 14 Oct 2014

@peti Maybe this has been fixed already?

Profpatsch on 5 Mar 2016

No, we don't support multiple outputs at the moment.

peti on 7 Mar 2016

https://github.com/NixOS/nixpkgs/issues/10541 could also be improved.

peti on 28 Apr 2016

https://github.com/NixOS/nixpkgs/issues/8390 has interesting discussion about this subject.

peti on 28 Apr 2016

b) is partially addressed at https://github.com/NixOS/nixpkgs/pull/16167 (at least for 7.10.x and 8.0.1)

rasendubi on 14 Jun 2016

Testing a patch to move libraries in a separate output now. Will report back soon.

Don't know how to split static and dynamic libraries apart, though.

rasendubi on 14 Jun 2016

Do we build static libraries by default? Should we? @peti

shlevy on 14 Jun 2016

We build both shared and static libraries by default.

peti on 14 Jun 2016

Generally in nixpkgs we only build shared and static you have to request specifically, is there a reason haskell is different?

shlevy on 14 Jun 2016

So far, Cabal used to implode when you tried to run builds that produce dynamic libraries only. Not sure how well recent version fare in that regard. Also, static libraries are very useful to produce binaries that don't depend on GHC at runtime.

peti on 14 Jun 2016

But if we do this split, that latter won't be an issue, right? Since having the runtime statically linked vs dynamically shouldn't make a huge difference in runtime deps (and will be much less wasteful if you have multiple binaries)

shlevy on 14 Jun 2016

There is also the issue of cabal-generated source files on build which can be (and are!) referenced by packages to get stuff from different paths, e.g. https://hackage.haskell.org/package/gitit-0.12.1.1/docs/src/Paths_gitit.html.
Those are generated by this brilliant peace of engineering: https://github.com/haskell/cabal/blob/master/Cabal/Distribution/Simple/Build/PathsModule.hs

Profpatsch on 14 Jun 2016

😕1

filename_stuff :: String

wow

shlevy on 14 Jun 2016

😄4

Starting with ghc 8.0.2 ./Setup.hs should support --libdir and --dynlibdir flags, first one is for static libraries and second for dynamic. Docs, and commit - https://github.com/haskell/cabal/pull/4011

sopvop on 25 Nov 2016

The Paths_pkgname module problem is real. A lot of haskell packages install data and use Paths_pkgname to access it. So even if you compile executable statically, you can still get 1.5GiB closure if you are not careful.
I propose to add following to generic-builder.nix:
Extra argument which defaults to false

{ args
, splitDataOutput ? false # backwards compatible
}

Conditional mutiple-outputs and configure flags

{
outputs = if splitDataOutput then [ "out" "data"] else ["out"];
defaultConfigureFlags = [
    "--verbose" "--prefix=$out" "--libdir=\\$prefix/lib/\\$compiler" "--libsubdir=\\$pkgid"
    (optionalString splitDataOutput "--datadir=$data/\\$datasubdir")
   other stuff ];
}

If it turns out that older cabal or ghc has problems with it, we can always back out.
At first this can be used to manually fix problematic packages, and later maybe cabal2nix
can automate that by examining PackageDescription.dataFiles.

sopvop on 25 Nov 2016

Ah, disregard that. Paths_pkgname it also has libdir in there.

sopvop on 25 Nov 2016

I'd hope we can split libdir and prefix.

With dead code elimination, most of the stuff in Paths_pkgname should be stripped out. If libdir is used then in Paths_pkgname is a very edge case we can fix per package.