Haskell-language-server: Continuous integration

[bubba]

Preferably I would like to see just one CI provider used

Wishlist:
1) Supports Linux, Windows, macOS
2) Can produce binaries and deploy them to GitHub releases
3) Doesn't have a (too strict) time limit
4) Has reasonable cache size

Appveyor and Travis CI are probably out of the question because of 1. and 3. respectively.
That leaves us with Azure or the GitHub actions. Github actions has had some issue with cache sizes in the past

[Avi-D-coder]

We could use Azure for Windows and something else for Linux and macOS.
Azure seems to be quite slow for hie.

[jneira]

It seems github is using azure underneath, but it has 20 parallel jobs. Cache size is not usable for us though as hie needs several gb to get reasonable build times

[jneira]

Also, ghcide and hie using already azure is a good point. However, i've been figthing it lately and it is not all reliable i would like.

[bubba]

It seems github is using azure underneath, but it has 20 parallel jobs. Cache size is not usable for us though as hie needs several gb to get reasonable build times

I did manage to build hie with cache with GitHub actions on lsp-test recently, since they've bumped the cache size to 2GB per job: https://github.com/bubba/lsp-test/actions/runs/29897031

My main concern with GitHub actions at the moment is the windows support for actions/setup-haskell, since it seems to only support linux currently. I had to add Mac support via a fork. It should be possible but we might have to set that up ourselves

[jneira]

Oh so maybe i misinterpret the comment about increasing cache: https://github.com/actions/cache/issues/6#issuecomment-571290282
I did understand that the 2 Gb limit was for the repo (i.e. all jobs) and no for each job

[bubba]

Yeah, I think it's per cache too so you can potentially have multiple caches per job. That is to be battle tested though!

[jneira]

Mmm maybe it worths to give github a try, using for windows the manual setup of cabal/stack/ghc using choco like in azure config of hie

EDIT: I've just checked what is the max cache used in hie and it is 1,850.0 MB for stack linux jobs, so it is dangerously close to the actual limit. Maybe it could be small though
CORRECTION: macos stack jobs are eating 2,346.8 MB :fearful: but they are caching ghcs as well

[jneira]

This time i would try to use directly the build system to install executables in ci too

[alanz]

What have we decide on this? Shall I just copy over the circleci stuff in the interim, to just have a smoke test at least?

[jneira]

Well, i think there is some value in to have at least two ci providers. If one of them changes its conditions or limits and it become unusable or it is broken for a while we will have the other one still active.
circleci is really fast and it starts with a custom docker image with the initial setup done so i think we can keep it permanently.

[Tehnix]

It seems GitHub Actions now have a cache limit of 5GB and it also supports Linux, macOS and Windows.

It might be worth checking out as an option. It's quite powerful and flexible, you can get a feel of what it can do from the workflow page. E.g. using cron to make a nightly build. You can also create a Docker container if need be.

My suggestion would be to definitely check it out, if you are still evaluating which CI platforms to consider. If you already have a setup, then it's probably not worth the hassle to switchover.

---

Since one of the stated goals here is binaries, if anyone knows how to actually build the Haskell program so it can be shared, then I would love to dive into setting it up for automatic releases on GitHub. Having toyed with editor setups recently, I'm quite jealous of how smooth rust-analyzer is to use. The editor plugins (even in vim) take care of downloading and updating the binaries for the language server.

[jneira]

It seems GitHub Actions now have a cache limit of 5GB and it also supports Linux, macOS and Windows.

I am afraid that we can easily exceed that limit if it is per repo: hie is using more than 20 Gb (30?) aprox in its 31 azure jobs: https://dev.azure.com/hvriedel/hvriedel/_build/results?buildId=288&view=results

We could start to cut off caches but azure has no cache limit other than the nodes hard disk space and you can just not worry about it

[Tehnix]

azure has no cache limit

Ah, okay. Then Azure seems like a great fit for this, considering that builds will take forever without caching. All in all, it should still be straight-forward to create GitHub releases anyways, so I don't think the particular CI that ends up being chosen will be a blocker for that.

[jneira]

After talking about this in irc:

• we will use definitely circle ci azure, like hie or ghcide
• but we should be more focused about what configurations should mark a commit as ready to be merged without manual inspection and analysis
  
  
  
  • the hard baseline will be circleci on linux
  
  
  • we can test in azure windows (and macos), but marking some steps (likely tests) as fallibles if necesary to achieve that goal
  
  

[expipiplus1]

CircleCI requires one to log in (and grant public and private RW permissions!) to view the logs. I worry that this would be an impediment to contributors.

[WorldSEnder]

No need to wonder anymore, it is for me. The fact that I have to sign up just to view the results of the builds is problematic. I just wanted to checkout the errors in #472 to see what is missing and what else I can work on but I'm not allowed to.

[bubba]

Now that we have the binary infrastructure setup on GitHub actions and working relatively well, we should give another look at running the tests on it. It would be nice to take advantage of @jneira's work of getting the ghcide tests running on Windows and have the test suite running on all platforms

[jneira]

Well, we have tests running for linux and windows + cabal in github (thanks again @bubba) and for linux + stack in circleci.
As i commented above, i would not remove circleci to dont lose stack builds, although maybe we could disable the test suite there.

Otoh i would add maocs to github actions, to cover all major os's.

Oops they already are there :wink:

So for me this one could be closed with the actual state of ci.
//cc @bubba

[jneira]

Closin, as we have already setup the ci as discussed here