Cabal: Use GHC CI infrastructure to produce release artifacts.

[phadej]

The current CI is good for daily development, but GHC CI has wider platform coverage, which would be great for providing binary releases.

Yet, the setup have to be made.

[bgamari]

Anyone interested should get in touch with me. I'm happy to advise.

[samuela]

Hey @bgamari I'm interested in taking a stab at this! What needs to be done?

[bgamari]

@samuela, great! Essentially, the task is to write a .gitlab-ci.yml similar to GHC's which uses the same build environments to build and package cabal-install. The GHC build environments already include cabal-install (since it's necessary to build GHC's Hadrian build system) so the task of building cabal-install should be fairly straightforward.

I have put up a quick experiment here. Feel free to use it as a starting point. The remaining work is the following:

• [x] fix the logic (i.e. the find command in my experiment) for locating and copying the executable
• [ ] modify the Linux job in my experiment to rather use the CentOS 7 build instead of Debian 9 (this ensures that we build against an old glibc, allowing us to use the same executable across all glibc-based distributions)
• [ ] Possibly freeze the Hackage index date?
• Add jobs for (using the GHC .gitlab-ci.yml as a template):
  
  
  
  • [ ] AArch64 Linux
  
  
  • [ ] ARMv7 Linux
  
  
  • [ ] Darwin (macOS)
  
  
  • [ ] Windows
  
  
  • [ ] x86-64 FreeBSD
  
  
  • [ ] x86-64 Alpine Linux (musl)
  
  
  • [ ] any other platforms that I have forgotten
  
  
• [ ] add --enable-split-sections to the cabal command line
• [ ] build and package the documentation?
• [ ] possibly package the build products into a tarball?
• [ ] cache the Cabal store?

The platforms which don't use Docker are a bit tricky; on these platforms (Windows, Darwin, FreeBSD) GHC's .gitlab/ci.sh script is responsible for installing a GHC binary distribution. You'll need to replicate this functionality.

I would be happy to set up a chat to discuss any details if this would be helpful. Alternatively, find me in #ghc on irc.freenode.net.

Thanks for volunteering!

[hasufell]

fix the logic (i.e. the find command in my experiment) for locating and copying the executable

    - mkdir -p out
    - cabal update
    - cabal new-install --install-method=copy --overwrite-policy=always --installdir=$(pwd)/out cabal-install

If you want to use new-build and then find the exe, do:

    - mkdir -p out
    - cabal update
    - cabal new-build
    - cp "$(cabal new-exec --verbose=0 --offline sh -- -c 'command -v cabal')" out/cabal

[bgamari]

Thanks @hasufell! I adopted the v2-install approach in my branch.

[samuela]

@samuela, great! Essentially, the task is to write a .gitlab-ci.yml similar to GHC's which uses the same build environments to build and package cabal-install. The GHC build environments already include cabal-install (since it's necessary to build GHC's Hadrian build system) so the task of building cabal-install should be fairly straightforward.

Thanks for all the info! Since this project is hosted on GitHub whereas GHC is hosted on GitLab, we'll need to do something different than .gitlab-ci.yml, right? It looks as though cabal is currently using GitHub Actions and Travis. So is the ultimate goal to do these builds on cabal GH Actions, cabal Travis, or ghc gitlab CI?

[samuela]

@bgamari Oh, I think I may have misunderstood! So your fork lives on gitlab and has a .gitlab-ci.yml file. So which of these CI systems is/should be producing the bindists?

[bgamari]

@samuela the plan here is to use GHC's GitLab CI to prepare the cabal-install binary distributions since then we can easily piggy-back on the existing work put into GHC's infrastructure (not to mention, neither GitHub Actions nor Travis support the range of platforms supported by GHC).

[samuela]

Ok, so in order to start adapting https://gitlab.haskell.org/bgamari/cabal-build-test/-/commit/50b81bcf1208dbd7ea566d0a2e0fa428dbdd4d00 to work with centos, we need a new docker image in ci-images. Here's an attempt: https://gitlab.haskell.org/samuela/ci-images/-/commit/b50c3c8049321d8542e7bc29bb03c4e815765621 but it fails with

Step 10/25 : RUN curl -L https://downloads.haskell.org/~ghc/$GHC_VERSION/ghc-$GHC_VERSION-armv7-centos7-linux.tar.xz | tar -Jx
 ---> Running in 1ee41616a1d9
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   178  100   178    0     0    217      0 --:--:-- --:--:-- --:--:--   217
xz: (stdin): File format not recognized
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/bash -o pipefail -c curl -L https://downloads.haskell.org/~ghc/$GHC_VERSION/ghc-$GHC_VERSION-armv7-centos7-linux.tar.xz | tar -Jx' returned a non-zero code: 2

So we'll need ghc-8.8.3-armv7-centos7-linux.tar.xz first...

[hasufell]

There is armv7 for https://downloads.haskell.org/~ghc/8.2.2/ (debian 8). You could try to bootstrap from there.

[bgamari]

@samuela I would say let's stick with building on Debian 9 on ARMv7 and AArch64. We don't currently build CentOS bindists on these platforms; it's something that ideally we would ideally do but this will take time and most people tend to use Debian-based distributions on these platforms anyways.

[samuela]

Ok, I've started some work on this branch. Here are the pipeline results. So far armv7-linux-deb9 and aarch64-linux-deb9 seem to be working but x86_64-freebsd failed.

[hasufell]

@samuela nice, a few comments

1. I think the FreeBSD runner doesn't have a cabal-3.x yet. I also manually update it in the ghcup CI. You can download FreeBSD binaries here: https://hasufell.de/d/d3e215db133e4fcaa61e/ (I signed the hash file). @hvr hasn't responded to my request of adding those tarballs to downloads.haskell.org yet.

2. I believe the idea was to add this CI in the actual cabal repository and then build the current in-tree version of cabal-install: https://gitlab.haskell.org/bgamari/cabal-build-test I believe your version currently builds from hackage? @phadej there seems to be no cabal mirror on gitlab.haskell.org yet?

[samuela]

I think the FreeBSD runner doesn't have a cabal-3.x yet. I also manually update it in the ghcup CI. You can download FreeBSD binaries here: https://hasufell.de/d/d3e215db133e4fcaa61e/ (I signed the hash file). @hvr hasn't responded to my request of adding those tarballs to downloads.haskell.org yet.

Hmm, this is unfortunate. Ideally we could just update the Docker image? I guess that would require having the downloads up on downloads.haskell.org? For the sake of this issue should we just go ahead and skip freebsd? Or will we need all the platforms in order to switch over?

I believe the idea was to add this CI in the actual cabal repository and then build the current in-tree version of cabal-install: https://gitlab.haskell.org/bgamari/cabal-build-test I believe your version currently builds from hackage?

I'm a little confused where this CI is supposed to live. We've got a bunch of different repos flying around across GitHub and GitLab! I'm happy to put them wherever. I interpreted this comment to mean that they should be added to GHC's GitLab repo, but perhaps I misunderstood?

[bgamari]

Hmm, this is unfortunate. Ideally we could just update the Docker image?

@samuela, right, I'm not opposed to updating the Docker image to pull FreeBSD binaries from @hasufell's server for now.

Or will we need all the platforms in order to switch over?

We don't, although it would be nice to do it in one swoop.

I believe the idea was to add this CI in the actual cabal repository and then build the current in-tree version of cabal-install: https://gitlab.haskell.org/bgamari/cabal-build-test I believe your version currently builds from hackage?

No, https://gitlab.haskell.org/bgamari/cabal-build-test is a fork of this repository and builds Cabal and cabal-install from said repository.

I'm a little confused where this CI is supposed to live. We've got a bunch of different repos flying around across GitHub and GitLab! I'm happy to put them wherever. I interpreted this comment to mean that they should be added to GHC's GitLab repo, but perhaps I misunderstood?

The idea is that we add a .gitlab-ci.yml to this repository (as I did in my experiment). https://gitlab.haskell.org/ can then host a mirror https://github.com/haskell/cabal which CI will run on. The binary artifacts that are produced from this CI can then be used as release artifacts.

[samuela]

So a lot of magic is happening in GHC's .gitlab/ci.sh. It seems like a bit of code smell to copy-pasta the whole thing into cabal to get builds working on non-Docker platforms. A few options off the top of my head:

• Just copy-paste the script into cabal.
• Download .gitlab/ci.sh off of the GHC repo, and then run it from there. Has the disadvantage that GHC can unknowingly break cabal CI.
• Move .gitlab/ci.sh into ci-images or another independent repo.

@bgamari @hasufell wdyt?

[hasufell]

@samuela I think the idea was to start off where @bgamari left in his experimental branch: https://gitlab.haskell.org/bgamari/cabal-build-test/-/blob/build-test/.gitlab-ci.yml

The GHC one is certainly overkill.

[bgamari]

Precisely. Very little of the complexity (maybe none?) in the GHC script is needed by Cabal, which is a standard Haskell package. As suggested by @hasufell, starting with my experiment is probably the best way to begin.

[23Skidoo]

This would be pretty great.

[samuela]

Ok, I've been working on this branch and specifically .gitlab-ci.yml. The following architectures are working so far:

• aarch64-linux-deb9
• armv7-linux-deb9
• x86_64-linux (deb9)
• x86_64-freebsd
• x86_64-linux-alpine
• x86_64-darwin

I think the only thing missing now is windows...

[phadej]

@Mistuke do you have your cabal building scripts somewhere publicly available?

[phadej]

Also @hvr, could you share the scripts you use.

The setup samuela made seems simple enough, but it misses making actual tarball and providing license report + plan.json which IIRC we want to provide along (or inside) the tarballs.

[Mistuke]

@phadej my CI runs a version of this

> cd "C:\TeamCity\buildAgent\work\9a40cc4bbeeda94c"
> del /F /S /Q "C:\TeamCity\buildAgent\work\9a40cc4bbeeda94c\hsbin"
> mkdir "C:\TeamCity\buildAgent\work\9a40cc4bbeeda94c\hsbin" 
> cabal v1-install alex --bindir="C:\TeamCity\buildAgent\work\9a40cc4bbeeda94c\hsbin"
> cabal v1-install happy --bindir="C:\TeamCity\buildAgent\work\9a40cc4bbeeda94c\hsbin"
> cabal --version
> alex --version
> happy --version 
> cabal  update
> cabal  --store-dir="E:\CabalStore" configure --extra-lib-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\lib" --extra-include-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\include" all --allow-older=cabal-testsuite:Cabal
> cabal  --store-dir="E:\CabalStore" build --extra-lib-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\lib" --extra-include-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\include" cabal
> cabal  --store-dir="E:\CabalStore" build --extra-lib-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\lib" --extra-include-dirs="E:\GHC\x86_64\ghc-8.4.3\mingw\include" cabal-install
> bash -c "find dist-newstyle/build/x86_64-windows/ghc-8.4.3/ -iname cabal.exe -print0 | xargs -I{} -0 cp -v {} ."
> E:\GHC\tools\cv2pdb.exe cabal.exe
> strip -s cabal.exe
> E:\GHC\tools\upx.exe -9 cabal.exe
> gpg --output cabal.exe.sig --detach-sig cabal.exe
> E:\GHC\tools\7zip\7z.exe a -bt -y cabal.zip cabal.exe cabal.exe.pdb cabal.exe.sig

[samuela]

I'm not at all familiar with windows, and I also don't have a windows machine to test on. But if anyone else is familiar/can contribute then I think we'd be all done here! Is there anything else we'd be missing at that point?

[bgamari]

I can handle Windows. Thanks @samuela!

[hasufell]

Can we trigger those release builds for the previous 2-3 versions?

[phadej]

Please hold your horses. In other words do not use these builds for anything yet. Let's make the setup work first and then propagate it. I don't want to see yet another feature left half-way because "it seems to work".

[hasufell]

Let's make the setup work first and then propagate it.

What's left to do? :)

[phadej]

@hasufell I see many unchecked checkboxes in https://github.com/haskell/cabal/issues/6616#issuecomment-615913991

[Mistuke]

Can we trigger those release builds for the previous 2-3 versions?

Also please don't change the binaries on the download site. This would break a lot of things. I'm sure no one was planning on doing that but just wanted to be sure 🙂

[hasufell]

@samuela what's left of the checkboxes in https://github.com/haskell/cabal/issues/6616#issuecomment-615913991 ?

[samuela]

• [x] fix the logic (i.e. the find command in my experiment) for locating and copying the executable

All done here.

• [ ] modify the Linux job in my experiment to rather use the CentOS 7 build instead of Debian 9 (this ensures that we build against an old glibc, allowing us to use the same executable across all glibc-based distributions)

Out of scope per https://github.com/haskell/cabal/issues/6616#issuecomment-616166143.

• [ ] Possibly freeze the Hackage index date?

Unclear whether this is required?

• Add jobs for (using the GHC .gitlab-ci.yml as a template):

  • [ ] AArch64 Linux
  • [ ] ARMv7 Linux
  • [ ] Darwin (macOS)
  • [ ] Windows
  • [ ] x86-64 FreeBSD
  • [ ] x86-64 Alpine Linux (musl)
  • [ ] any other platforms that I have forgotten

These are all done except for windows.

• [ ] add --enable-split-sections to the cabal command line

Done in https://gitlab.haskell.org/samuela/cabal-build-test/-/commit/1964e1c8.

• [ ] build and package the documentation?
• [ ] possibly package the build products into a tarball?
• [ ] cache the Cabal store?

Not sure which of these are hard requirements. Right now, we're not doing any of them AFAIK.

[hasufell]

@phadej so what is blocking further steps?

[phadej]

I'll take a look.

[phadej]

I'm having really a bad day, but I say it here anyways. I don't see how we can officially support FreeBSD or ARM as e.g. Windows eats occasionally days of work.

In other words, it would be simpler and clearer for us to provide binaries for GHC's Tier 1 platforms only. We could have scripts for say FreeBSD or/and ARM in place, but if those fail, it won't hold release process or change any priorities.

EDIT: I'd rather support FreeBSD and ARM than Windows, but if one have to choose (and we seems to be forced to) than Windows wins.

[Mistuke]

I'm having really a bad day, but I say it here anyways. I don't see how we can officially support FreeBSD or ARM as e.g. Windows eats occasionally days of work.

I'm sorry you're having a bad day, I know what those are like. But I wonder why you think Windows is so much harder. I build cabal nightly in different configuration and have done so for 2 years now with the only time I had to make any changes was when I hit a GHC bug recently.

So I wonder if Windows isn't more difficult because of the CI setup, and not Windows itself.

In other words, it would be simpler and clearer for us to provide binaries for GHC's Tier 1 platforms only. We could have scripts for say FreeBSD or/and ARM in place, but if those fail, it won't hold release process or change any priorities.

EDIT: I'd rather support FreeBSD and ARM than Windows, but if one have to choose (and we seems to be forced to) than Windows wins.

What exactly is the thing that is increasing your maintenance burden. I'd be happy to go over the configuration with you instead of having you stab around in the dark. You know where to find me.

[phadej]

What exactly is the thing that is increasing your maintenance burden. I'd be happy to go over the configuration with you instead of having you stab around in the dark. You know where to find me.

This CI issue is just a reminder that heterogeneous targets are tricky. There are occasionally issues with Windows, like this CI or segfaulting (https://gitlab.haskell.org/ghc/ghc/issues/17926). We don't have even thought about signing cabal binaries for macOS (which I guess will be required), I hope that whatever GHC will do could help cabal as well there.

So adding ARM and/or FreeBSD to the set which one should keep working all the time doesn't feel good now. I do think that they will have less unique issues then mac or win, but I don't want to promise anything.

I'm also really worried what the rumored ARM based mac would mean to all of us.

[phadej]

And to clarify even more. I somehow know Linux, and understand mac. So can reasonably deal with most issues on those. If it weren't you @mistuke, than Windows story would be a lot worse. You deserve a lot of thanks for helping us.

But I have no idea about FreeBSD or OpenBSD, and what issues ARM might bring to the table. It's an unknown territory with no "support network". I guess if someone who would champion these areas (and don't want to change or rewrite all the rest of cabal because of that), then it would be great.

[hasufell]

@phadej FreeBSD support should really be no issue. I can spin up a FreeBSD vm in a few seconds, do tests, build stuff. In fact, the first few ghcup versions were all manually built on FreeBSD with manually built cabals. I haven't experienced major issues. The most problematic OS is macOS and it isn't easy to get your hands on a VM.

[samuela]

@phadej I'm sorry to hear this is turning out to be so stressful. I understand first-hand that maintaining an OSS project involves a lot of work, much of it thankless. I think we should brainstorm ways to make this all easier to maintain.

It sounds to me like there are two questions at play here: (a) How much can/should be put into CI? and (b) What platforms should be supported and at what "support tiers"?

In terms of (a), I claim that having a CI should greatly reduce the workload and stress required in publishing a release and maintaining stability across a variety of platforms. Codifying all of this ad hoc logic into CI means that we have a single, reproducible, and version-controlled source-of-truth for the build and release process across all the platforms. This way, eg. @Mistuke could contribute with his unique Windows knowledge, and others could contribute to platforms of their own interest/ability.

Now the question (b) of which platforms should be supported seems slightly thornier. One significant advantage of having a CI in place is that then the onus of support for each platform is shifted to committers when submitting PRs, and off of maintainers. Ie, platform-specific issues are caught on each build/test triggered on every PR. The easiest strategy in my mind would be to set up CI with the platforms listed in https://github.com/haskell/cabal/issues/6616#issuecomment-615913991, and then if certain platforms prove to be too onerous then a discussion can be had about whether or not to allow failures for those targets in CI.

[phadej]

It sounds to me like there are two questions at play here: (a) How much can/should be put into CI? and (b) What platforms should be supported and at what "support tiers"?

Please send an email to [email protected] to discuss that. I have put way too much time (month of work in the beginning of this year) into current CI setup which works reliably enough to actually be useful but that might be my familiarity bias. GitHub Actions don't offer not-self-run ARM nor FreeBSD runners, so adding them in current setup is not possible. Changing CI is not what I want to be part of. In principle it should be easy, "run ./validate.hs".

I promise to stay out of that discussion.

[phadej]

To add

I'm not completely happy with GitHub actions setup either.

• You cannot restart individual jobs, only all jobs in a workflow at once.
  
  
  
  • Fortunately our tests aren't as fluky as it used to be
  
  
• The continuonsly updating base images of macOS and Windows.

But on the other hand

• It's free
• Setup is relatively simple
  
  
  
  • Accessible for modification to all Cabal contributors
  
  
• It's a lot faster than previous travis setup
  
  
  
  • This is something I'd like not to give up. I can trust jobs to complete in 1-2 hours since creation.
  
  
• There weren't hardware issues during half year we have used it
  
  
  
  • There were occasional software "events non delivered" outrages on GitHub, but those affected all CI providers.
  
  

[samuela]

Well ARM support could be accomplished through self-hosted runners, which is actually the same as what the GitLab CI does. We could probably scavenge a Raspberry Pi or two to run those. I personally don't care about FreeBSD, but that can be run in docker on an Ubuntu runner, so it's not out of the question. Should we just forge ahead with GitHub Actions then? The work done here can be migrated to use GitHub Actions?

[samuela]

The Raspberry Pi folks seem to be moving towards 64 bit builds, which makes armv7 less important in my mind. Of course aarch64 is still very much needed, and many devices will be running armv7 for a while to come.