Haskell-ide-engine: Default build process may crash users machine

[hasufell]

Just tried to build HIE and it spiked my memory up, bombed my swap and made my laptop unresponsive for straight 20 minutes before I could even abort the build process, endangering my files and current work.

This is one of the worst things with haskell build tools. Making assumptions about users memory in a language where the garbage collector is totally unpredictable.

Related:

• https://github.com/haskell/cabal/issues/5776
• https://github.com/commercialhaskell/stack/issues/4296

PS: I have never encountered this in any other language.

[Anrock]

Sounds like upstream issue to me. Not sure what is expected action here.

[hasufell]

Sounds like upstream issue to me. Not sure what is expected action here.

Run cabal with -j1.

Also, it's not even clear how the user is supposed to do that with this stack-shake-cabal config.

[lorenzo]

I think the best way forward for hie is to offer binaries, instead of asking users to compile it from scratch.

[hasufell]

I think the best way forward for hie is to offer binaries, instead of asking users to compile it from scratch.

That's a totally different issue. And I do not want to use binaries.

[fendor]

Cant we also try to restrict the memory?
The flag -j1 instructs shake to only use a single core for the build environment, e.g. shake rules. We can define additional flags that are passed to stack/cabal to enforce a specific amount of cores.

[lorenzo]

That's a totally different issue. And I do not want to use binaries

Cool, but I still think that’s the best way forward to shield most users from all the installation problems that’s may arise, such as this one. You, as an advanced user who does not want binaries, can still tweak the configuration.

[Anrock]

@hasufell were you building HIE via Stack? If so, wouldn't it be easier to adjust number of build threads in your global stack config?

I, personally, never had problems with builds locking up my system on any of my machines (one is octacore with 16GB and second is quadrocore with same RAM) and OSes (Lin, Win, OSX), so to me it sounds like this issues is specific to your machine.

[hasufell]

@hasufell were you building HIE via Stack? If so, wouldn't it be easier to adjust number of build threads in your global stack config?

stack install.hs cabal-hie-8.6.5

so to me it sounds like this issues is specific to your machine.

I don't think so. Other users have reported similar problems. HIE is especially difficult, because it requires you to go through shake.

[mohkale]

I agree with @lorenzo. HIE has way too many dependencies and takes far too long to build for it to be practical to build it yourself on windows, where it failing to build for a completely arbitrary reason isn't too out of the ordinary. I tried to install it 9 times yesterday over the course of 6 hours. By the end I still couldn't get it running. I gave up and intend to try again today.

Update:

well the install finally finished. it took an hour and two minutes and a whopping 3 GB of ram (at the highest, with windows taking 3gb more and me running on an 8gb machine, that's cutting it pretty close) to dump 312mbytes (including a 63mb rebuild of stack for some reason) of binary files in ~/AppData/Roaming/local/bin. I finally got it to compile by killing every other running program so the compiler could get enough memory. Aparrently a 251 exit code means the compiler/linker ran out of memory (I couldn't really find any mention of that anywhere, so here's hoping google will pick up on that keyword). the final compiled size of hie-8.6.5.exe is 165mb, I could've downloaded that in 15 minutes (worst case) and saved myself a tonne of hassle.