Nixpkgs: Add binary snapshots of GHC versions newer than 7.4.x for quicker bootstrapping

[copumpkin]

The current bootstrap process goes from a binary 7.4 to 7.8 to 7.10 to 8.0 which as you might expect, takes _ages_. I'm getting conflicting reports on whether the bootstrap situation is better on Linux, but I think it'd be fairly straightforward to update the GHC bootstrap package to be 7.10 for Darwin. Is there a reason not to do that? The current situation makes things rather painful when you do a lot of stdenv work.

cc @peti @pikajude

[peti]

GHC 7.4.x is the latest version we have a binary snapshot of, hence that's the latest compiler we can use for bootstrapping. This is the same on all platforms. Adding binary snapshots of newer compilers is entirely feasible, it's just a boring task no-one bothered to do yet.

[copumpkin]

Ah okay thanks, just wasn't sure if there was a deeper reason than that. I might take a peek at making a new one (for Darwin at least) later if I'm sitting around bored waiting for several GHCs to build 😄

[vcunat]

Isn't it possible to "make larger jumps" in this sequence?

[dtzWill]

I suppose the lack of incentive here is because once someone gets GHC versions that work for a platform folks mostly grab various versions from the cache?

Anyway, this caused me a fair bit of pain (on Linux, but same problem really) while working on https://github.com/NixOS/nixpkgs/pull/21163 so thought I'd drop a line over here :).

[dtzWill]

To answer @vcunat's question, it looks like roughly you can build using ~4 minor versions earlier although it seems to change. The release notes for each describe the requirements, apparently.

For example, it seems 7.10.3 can be built with 7.6 or newer:

https://www.haskell.org/ghc/download_ghc_7_10_3#sources

Looks like we can skip a stage by either

• Creating 7.6.3 binary package and using it to build everything after it and before 8.0.1. Luckily everything after it (7.8 onwards) can be built with enableParallelBuilding which helps quite a bit. This shortens the build chain for more versions but requires putting together a 7.6.3 bindist.
• If building 7.10 quickly is less of an issue now that 8.0.1 is out (I couldn't say), we could instead simply change 8.0.x+ to build using 7.8.4 (skipping the need to build 7.10.3). According to https://www.haskell.org/ghc/download_ghc_8_0_1#sources 7.8 is all that's needed.

For my needs, the second option sounds better and I'm testing that currently to see if anything obviously breaks...

Thoughts?

[dtzWill]

It seems ghcHEAD (8.1) requires 7.10 which makes my proposed solution a short-lived one at best. Hrmph.

[vcunat]

Why not simply always bootstrap from the oldest possible version, e.g. 8.1 from 7.10? Such a scheme should always use the shortest number of steps, given a fixed set of binary versions.

[Ericson2314]

True, we have enough parallel capability that shrinking the length of bootstrap is arguably more important than reducing redundancy among intermediate steps.

edit oh but in the context of this that means more binary blobs hmm.

[vcunat]

I didn't mean adding more binary ghc versions; that seems mostly an independent question.

[domenkozar]

What's the status, anyone working on this? It would help a lot doing mass changed to GHC.

[peti]

I am not aware of anyone working on this topic (and I, personally, am not).

[peti]

Closing due to inactivity.

[domenkozar]

Let's keep this one open, we need to get it done.

[domenkozar]

Did anyone check what will GHC 8.2.1 require?

[dhess]

This is also needed to build Haskell packages on armv7l and aarch64, as those architectures do not have a binary GHC for 7.4.2. (In fact, armv7l doesn't appear to have a binary dist from haskell.org until 7.10.3, and aarch64 doesn't have one at all. Perhaps a Debian .deb could be used instead. https://packages.debian.org/search?keywords=ghc)

[dhess]

Here is a WIP for a binary derivation for GHC 8.0.1. It uses Debian's binary distribution so that it can be used to bootstrap GHC for armv7l and aarch64.

https://github.com/dhess/nixpkgs-channels/commit/9d36babcbd89772947e5da47a8107781dc38e0e1

In this changeset, I've modified the ghc802 derivation to use this binary distribution, for much faster bootstrapping.

It's still a bit rough and I could use some guidance on a few things. My incentive here was to get GHC built for ARM -- all else was secondary.

• Because it uses Debian's binary distributions, it cannot be used on Darwin. Darwin would need special-casing before this changeset could be committed.

• Existing ghcXBinary derivations do not define a Haskell package set, only a compiler, and therefore they do not include hscolour, which is used for more recent GHC derivations to include colorized HTML documentation. I've followed this pattern here with the GHC 8.0.1 binary distribution, which means that any GHC derivations that are bootstrapped from it will not include colorized documentation. Is there a way to get hscolour for a bootstrap-only derivation without needing a from-scratch build of the bootstrapping compiler?

• In the current changeset, I've only used it to bootstrap GHC 8.0.2, but I imagine it could be used to bootstrap at least some older GHC versions, as well.

• The patchelfed Debian ghc binary suffers from this issue: https://github.com/NixOS/patchelf/issues/99. It appears to be only a cosmetic issue.

• For some reason, when compiling for armv7l, I had to add llvm_37 to propagatedBuildInputs, otherwise GHC complained that it could not tell which version of llvm was installed. (This was not necessary on x86_64, but didn't hurt, either.)

• I have only tested this on x86_64 and armv7l. If anyone with some build horsepower wants to give it a go on i686 or aarch64, let me know how it goes.

Suggestions on how to address these issues are welcome.

Also note that you may need to define a 1G or larger swapfile on your armv7l build host to get a bootstrapped GHC to build on that platform. I had several GHC 8.0.2 builds run out of memory on my Jetson TK1 (2GB RAM) until I defined a swapfile.

[cleverca22]

i also have some notes on the min versions and an unfinished PR i need to make in https://gist.github.com/cleverca22/81e6f84355412f8e13c685a1bac6f6d3

[dhess]

The WIP I mentioned above should be obsoleted by #29688, once that's merged.

[domenkozar]

FYI: https://github.com/NixOS/nixpkgs/pull/33054 is ready for review.

[peti]

master at 635c10140fc276469a905e2750ff9a260da52c1a contains @domenkozar's improvements. Please verify that everything works as expected.

[domenkozar]

https://github.com/NixOS/nixpkgs/pull/33129 wires up boot packages in a way that each ghc now directly depends on a binary ghc for building.

@dhess if you can work on aarch64 from here on, that would be great. I've only tested linux.